Ostinato throughput on a 40G NIC
Update (2021): Ostinato now supports 10G, 25G and 40Gbps line-rate rate traffic using the Turbo Transmit add-on.
A few weeks ago I attended the DPDK Summit in Bangalore, where M Jay from Intel loaned me a dual-port 40G interface to enable Ostinato to become a high-speed packet generator using DPDK. Thanks M Jay and Intel!
If you’ve been following Ostinato for a long time, you might recall I had done a prototype of Ostinato with DPDK back in 2014 which won the 6Wind DPDK Design contest. That work was done on a VM using VNICs instead of physical NICs and the lack of the latter (amongst other things) meant I never productized that code.
Before I (re)start the DPDK work on Ostinato, I wanted to get some baseline measurements with the existing libpcap based code. This blog post presents these results.
You can also look at the results for a 1G port for Linux, MacOS, Live ISO.
I configured the same single stream as in previous tests -
- Protocols: Mac, Ethernet II, IPv4, UDP, Pattern
- Source and Destination Mac addresses populated with the actual Mac addresses of the source and sink
- Source and Destination IP addresses populated with the actual IP addresses of the source and sink
- Packets/sec: 0
- Number of packets: 10 (default)
- After this stream: Goto first
One difference from the previous tests is that I’m using a single host with a dual-port NIC with the ports connected back to back. Ostinato packets sent on port 1 are received back on port 2 on the same host.
Also, I’m adding two jumbo sized packet sizes to this test - 4096 bytes and 9018 bytes.
Here are the numbers -
|Packet Size||Send Rate (Kpps)||Recv Rate (Kpps)||Send Rate (Mbps)||Recv Rate (Mbps)|
Mbps rate was calculated using the below formula to take into account the ethernet line overhead of 20 bytes (1 byte SFD + 7 byte preamble + 12 byte inter-packet gap) -
Rate (in Mbps) = (PacketSize + 20) * KppsRate * 8 / 1000
- Ostinato 0.9 - for Linux (Ubuntu 16.04, 64-bit)
- Ubuntu 16.04.3 LTS
- Processor: Xeon E5-2620v4 @ 2.10GHz (8-core)
- RAM: 32GB
- NIC: Intel XL710 40Gbps dual-port PCI 3.0, x8
- The first that jumped out was that there was a lot of fluctuation in the send rate while transmitting e.g. for 256 bytes packets, I saw rates ranging from 635Kpps to 728Kpps. This is a very wide range and I don’t have an explanation. I have used the lower end of the range seen in the above results for all packet sizes.
- The second thing that jumps out from the results table is that receive rate gets capped at ~640Kpps even if the send rate is higher. Again, I don’t have an explanation for the same.
- The next thing that is suprising is that the max rate is achieved for 4096 packets, instead of 9018.
- Although it’s a 8-core CPU, only two core reach 100% CPU during the test - this is expected since for a single port, Ostinato doesn’t use more than one core and we have both transmit and receive port on the same host. The first core with 100% cpu utilization (presumably the one handling the tx port) has 15% user and 85% system usage - indicating that the major time is spent in the kernel driver. The second core had 100% utilization in softirq - presumably the one handling the receive port
- RAM usage not measured
- One heartening observation is that Ostinato (even without DPDK) can be used as a 10G traffic generator - if you can use jumbo frames.
For more Ostinato posts, subscribe for email updates.
Leave a Comment