Test NAT using Ostinato

Recently, Ostinato customer Javor reached out to me and wanted advice on how to configure traffic stream(s) to test 2 million NAT sessions.

A NAT session is based on the 5-tuple (SrcIP, DstIP, Protocol, SrcPort, DstPort). One NAT session is created per unique 5-tuple.

His first attempt was to configure the Src/Dst IP as random but he ran into a known Ostinato bug that causes incorrect TCP/UDP checksum. I suggested he workaround the checksum problem by overriding it to 0 (0 checksum value implies no checksum), but his test required that checksums not be 0. So he tried to configure incremental Src/Dst IP alongwith incremental Src/Dst Port and ran into a different problem. In his own words,

I find Ostinato for a great product.

But now I’m fighting with Ostinato to test CGNAT performance of one router and trying to create some stream that can generate the following packets with valid checksum mandatory.

So when I create a new stream with:
SRC IP - Increment - 10.0.0.0/16 - 1000 hosts
DST IP - Increment - 16 hosts
SRC Ports - incremental - very important for this test - 60000
DST Ports - incremental - 100

In fact the stream works but not in way that I expect.

The stream just generated sequentially flows one by one like this for example:
src-ip_1, src port 30001 -> dst-ip_1, port 1001
src-ip_2, src port 30002 -> dst-ip_2, port 1002
src-ip_3, src port 30003 -> dst-ip_3, port 1003

But I don’t want this. I would like to achieve the following way of order and incrementation: src_ip1, src port 30001 -> dst-ip_1, port 1001
src_ip1, src port 30002 -> dst-ip_1, port 1002
src_ip1, src port 30003 -> dst-ip_1, port 1003
src_ip1, src port 30004 -> dst-ip_1, port 1004
src_ip1, src port 30005 -> dst-ip_1, port 1005
……

So I would like the stream to generate the packets in order to start first from src-ip and incrementally passthrough over all src-ports range ( for example range of 5000 ports ) and then to go to the next src-ip then to next src-ip and so on….

Final goal is to utilize the router device with maximum unique sessions but not only unique per IP but each IP to generate more sessions. I’m trying to utilize ~2mln sessions.

As workaround I have tried to generate a lot of streams with fixed src/dst hosts with incremental range of src ports and it works but for 2mln sessions I need to generate a lot of streams ~ 240 streams in my case which is very very difficult and a lot of work - Copy/duplicate/change src-dst host and so on.

So I wonder if there is any easier way to achieve this.

Essentially the problem is that when you have multiple variable fields configured on a stream, each variable field will increment with every packet.

What Javor wants is a way to increment one variable field every X number of packets (where X > 1) while another variable field increments with every packet.

I told him that this was currently not possible and support may be added in a future release, but the problem, for some reason, refused to leave my head. A couple of days of background processing later, I had an insight. This is possible to do now albeit with some restrictions. Restrictions that I think Javor can live with.

Here’s how.

Let’s start with the objective of 2 million NAT sessions. If we iterate over the entire SrcPort range (0 – 65535), we need 2,000,000/65536 = ~ 30.5 SrcIP; let’s round it up to 32 SrcIP.

So, we create two variable fields - one for SrcPort and another for SrcIP.

The SrcPort variable field configuration is straight forward -

• Increment SrcPort from 0 to 65535 step 1

(If you are using Ostinato v1.0 or earlier, you will not be able to enter count as 65536 due to a bug. Once you read this post in full, you will find a workaround for it)

The SrcIP is trickier. We’ll go slow. The most critical part is that the SrcIP should be incremented by 1 every 65536 packets.

Variable field mask to the rescue!

Originally, the variable field mask was implemented to allow increments in bit fields smaller than a byte without affecting adjacent bit fields. But it can do more—we can use mask to achieve what we want in this case.

From the Ostinato user guide -

The computation used for varying fields is as follows -

new = (old AND ~mask) OR ((value op n*step) AND mask)

where

• old is 8/16/32-bit value (depending on counter type) starting at offset bytes from start of protocol
• op is based on the mode - increment/decrement/random
• n varies from 0 to (count-1) incrementing by 1 every packet
• new is the value that will replace old

In a IP packet, the Src IP is 4 bytes (offsets 12 to 15) and Dst IP is 4 bytes (offsets 16 to 19). Let’s define a custom field for IPv4 as follows -

• Field Custom
• Type Counter32
• Offset 14
• Mask 0xFFFF0000
• Mode Increment
• Step 1

Let’s not bother about Value and Count for now. We’ll come to those. For now, let’s look at this visually. Given an IP frame with SrcIP as s1.s2.s3.s4 and DstIP as d1.d2.d3.d4 -

Offset      12                  16                  20
         ---+----+----+----+----+----+----+----+----+---
IP Frame    | s1 | s2 | s3 | s4 | d1 | d2 | d3 | d4 |  
         ---+----+----+----+----+----+----+----+----+---
Mask                  | FF | FF | 00 | 00 |
         ---+----+----+----+----+----+----+----+----+---
N*Step                | 00 | 00 | 00 | 01 | <== increments every packet
         ---+----+----+----+----+----+----+----+----+---

Our mask covers the last 2 bytes of SrcIP and first 2 bytes of DstIP. But the mask makes sure that the DstIP is not modified. For every packet, the N*Step value will be incremented by 1. The first 65536 packets will have this value increment from 0 to 65535 (0xFFFF) i.e. only the last 2 bytes are changed across these many packets, but because the mask blocks off change in these 2 bytes, they are not reflected in the packet. For the 65537th packet, this value will become 0x10000 (65536) which means the LSB of the last byte of SrcIP will get incremented. Subsequent packets will again change only the lower 2 bytes (which has no impact because of the mask) while the upper two bytes have the constant value 0x0001. After another 65536 iterations, the upper 2 bytes will be incremented by 1 i.e. to 0x0002 and so on.

Effectively, we are ensuring that SrcIP gets incremented every 65536 iterations!

If you work with binary and hexadecimal numbers regularly, this should make sense. If not, re-read it a couple of times and it will— trust me!

The insightful reader would have noticed that this will work only with powers of 2 (e.g. 65536 = 2^16). Yes, that’s the limitation that I talked about at the start!

Let’s come to the remaining fields.

First count. We want 32 SrcIP across 65536 SrcPort, so the count should be 32*65536 = 2097152

Finally value. This is the starting value of SrcIP. But, only the last 2 bytes of it followed by first 2 bytes of DstIP (that we don’t want to modify). Let’s say you want the starting value of SrcIP to be 10.2.24.100 (0x0A021864), we take the last 2 bytes 0x1864 as the first 2 bytes of the value and the last 2 bytes as 0x0000 i.e. 0x18640000, convert it to decimal (409206784) and enter that in the value field -

That’s it, we can now test 2 million NAT sessions using a single Ostinato stream! And depending on the NAT table size of the DUT, you can even simulate a NAT table full condition!

Not so fast, though!

There’s another piece of coincidence in this example. The SrcPort variable field count 65536 is exactly equal to 2^16—the range of increments that we mask off by configuring the last 2 bytes (16 bits) of the mask as 0.

What if we want a different range to vary for SrcPort. It would still work as long as the count is a power of 2. e.g. let’s say we want to vary across 16384 (16K) ports instead of 65536 (64K). All we have to do is change SrcIpStep such that SrcIpStep*SrcPortCount = the masked off portion (2^16 = 64K). In other words, Src IP Step = 64K/SrcPortCount = 4 for this example. This is useful when some NAT systems limit the max NAT session per host to a smaller number.

While writing this blog post, I found a bug that the count field in the Variable Fields GUI does not allow to enter 65536 for a 16 bit field such as the UDP source port. This bug is present in all versions uptil 1.0. If you are using one of these versions, instead of 32 SrcIp * 65536 Src Ports, you can use 64 SrcIp * 32768 Src ports to achieve 2 million NAT sessions—it’s still a single stream.