Ostinato RX stream stats always zero

Ostinato user Leslie reached out to me to report that Ostinato per-stream stats are not working on the RX side (always 0) with VLAN-tagged packets on Ubuntu 22.04 LTS.

Upon investigation, it turns out to be a libpcap issue.

But first, a bit of background.

To implement per-stream statistics, Ostinato tags these packets with the magic value of 0x1d10c0da as the last 4 bytes of the packet, just before the Ethernet FCS. This serves as a signature to quickly check if it’s an Ostinato generated packet. We can then read the stream GUID which is encoded just before the signature.

To increase performance, Ostinato uses a pcap filter to read into user space only packets with the Ostinato signature. Simply put, we need to use the following capture filter -

ether[len - 4:4] == 0x1d10c0da

However, there’s a problem with this scheme. Say, we send out these packets with signature. Somewhere in transit or at the destination, the packet encounters an error and a ICMP error is sent back. The ICMP error packet contains the original packet as payload and so the ICMP error packet will also have the Ostinato signature at the end!

To avoid matching these packets, we modify our capture filter as -

(ether[len - 4:4] == 0x1d10c0da) and not icmp

But that matches only untagged packets, not vlan-tagged packets; so we modify the capture filter as -

(ether[len - 4:4] == 0x1d10c0da) and not (icmp or (vlan and icmp))

And that’s where we run into the libpcap problem.

The BPF instructions generated by libpcap for the above filter are incorrect!

(Note that the actual capture filter string used by Ostinato is a bit more complicated for reasons not relevant to this blog post)

More details about the generated BPF for this problem and discussions with Wireshark and libpcap teams are available at the below links, if you’re curious -

• Wireshark Q&A
• libpcap bug report

The problem is Linux specific (likely due to BPF extensions) and is seen only for vlan tagged packets.

Ubuntu 22.04 includes libpcap 1.10.1. The problem is present even with 1.9.x and 1.8.x. libpcap 1.7.x introduced the BPF vlan check (although the code looks very different).

So what’s the solution?

While I figure out how best to handle this in Ostinato, for now the solution is to use libpcap 1.6.2. Here are instructions to build that from source -

wget https://www.tcpdump.org/release/libpcap-1.6.2.tar.gz
tar zxvf libpcap-1.6.2.tar.gz
cd libpcap-1.6.2
./configure
make

That will generate libpcap.so.1.6.2.

But Ostinato usually is linked against libpcap.so.0.8 (ldd drone | grep libpcap). So we create a softlink by that name in the same libpcap-1.6.2 folder.

ln -s libpcap.so.1.6.2 libpcap.so.0.8

Next step is to make Ostinato use this libpcap than the system one. For that we use LD_LIBRARY_PATH when starting Ostinato (or drone).

LD_LIBRARY_PATH=/path/to/libpcap-1.6.2/ ostinato

Starting Ostinato like that should fix the rx stream stats problem.

For more Ostinato related content, subscribe for email updates.