Srivats P

Enabling extra repos for RHEL on OBS

me@srivatsp.com (Srivats P) — Tue, 20 Jan 2026 00:00:00 +0000

It took me a bit of time and effort to figure this out, so thought I’d post it for other people’s benefit (and for future-forgetful-me!).

If you use Open Build Service (OBS) to build Linux RPM packages on RHEL-compatible distros like Alma Linux or Rocky Linux and your builds fail due to dependencies not being found, read on. This might help.

Starting RHEL 8, RedHat (and clones) shifted to a multi-repo layout for packages named BaseOS, CodeReady Builder (CRB), AppStream etc. Packages are now divided between these repos.

By default OBS only enables the baseos repo for these distros. To enable other repos (including the Fedora EPEL) on your OBS project, you need to add them to your OBS project’s Meta -

  <project name="home:pstavirs">
  . . .
  <repository name="AlmaLinux_9">
    <path project="AlmaLinux:9" repository="baseos"/>
    <path project="AlmaLinux:9" repository="appstream"/>
    <path project="AlmaLinux:9" repository="CRB"/>
    <path project="Fedora:EPEL:9" repository="standard"/>
    <arch>x86_64</arch>
  </repository>

If you are using a local OBS instance with interconnect enabled to the public OBS instance, just prefix the local project name of the interconnect OBS instance -

  <repository name="AlmaLinux_10">
    <path project="openSUSE.org:AlmaLinux:10" repository="baseos"/>
    <path project="openSUSE.org:AlmaLinux:10" repository="appstream"/>
    <path project="openSUSE.org:AlmaLinux:10" repository="CRB"/>
    <arch>x86_64</arch>
  </repository>

Here’s how my Interconnect is set up on my local OBS instance (note the project name openSUSE.org) -

Hope this helps you!

veth on MacOS

me@srivatsp.com (Srivats P) — Wed, 10 Dec 2025 00:00:00 +0000

Linux virtual-Ethernet (veth) interfaces are quite useful for a variety of purposes - for me, that’s to test Ostinato.

I recently discovered MacOS also supports something similar - though they are called Fake Ethernet (feth) there.

Not much information is available on these interfaces and their capabilities, but they do seem to work like basic veth pairs on Linux.

Here’s how you can create veth-like or feth interfaces on MacOS.

Run the following commands using sudo -

ifconfig feth0 create
ifconfig feth1 create
ifconfig feth0 peer feth1
ifconfig feth0 up
ifconfig feth1 up

That’s it!

If you send traffic on feth0 you will receive it on feth1 and vice-versa.

If there’s more information about these feth interfaces, do let me know in the comments.

Saving Ostinato port config

me@srivatsp.com (Srivats P) — Tue, 22 Apr 2025 00:00:00 +0000

Back in Feb, as I was telling yet another customer that the easy way to start with Ostinato Python API is not to write python script from scratch, but to save your streams as a python script from the GUI, I remembered that python code is generated only for streams, not for emulated devices or port settings.

High time I fixed that.

File Format infra

The infra is already there. I just need to do some code generation for port settings (Tx mode, port reservation, stream stats etc.) and device groups.

Easy Peasy.

As I started changing the code, I figured out that the infra has access only to the streams, not to port settings or device group config!

Of course, it’s a Save Streams As functionality. Duh!

We do export everything when you save a session, but that is across all ports of all port groups … overkill for what I wanted – an easy way to get customers to start with the Ostinato Python API.

What was needed was to implement a Save As for port configuration.

Like any lazy (read good) programmer, I created copies of streamfileformat.* files into portfilefileformat.* and started making changes.

But soon my conscience kicked in.

We already have infra code duplication between session and streams for open/save. Port open/save was adding to this duplication!

It’s Refactoring time!

I refactored StreamFileFormat and SessionFileFormat classes to use a newly created template class. Some manual testing later, I was confident this was working (and hopefully not causing a regression).

Post the refactoring, I created PortFileFormat abstract and concrete classes and the infra for saving port configuration as a native file format extendable to save as Python script (which is a different save format).

The code was done - but there was no way to invoke it without adding menu items to the GUI.

Frontend

So, I did that next. Plus, the main code to aggregate port settings, all the streams and device groups into a single PortContent structure that will be passed to the file format infra which will in turn write to file.

With all that code done, it was time to test stuff out. It was already late Friday evening, but I couldn’t resist the temptation to try it out.

💥 CRASH!

And that’s where I left it for the weekend.

Came in Monday morning and fixed the crash and things were working fine!

UX Improvement time

Time for some UX enhancements related to this feature.

First, warn the user about overwriting existing configuration when opening a port file.

Next up - add code to enable drag-drop of Ostinato port files (.oprt)

Let’s add some icons to the Port open/save.

For good measure I added some icons to the existing Open/Save of Session and Streams which till now didn’t have any icons

Next, I extracted the port related actions into a new top-level menu.

and added a prompt/warning when saving streams.

Consistent Terminology

After these changes, the phrase Port Configuration now includes streams and device groups also. So renamed a few things in the UI - the erstwhile Port Configuration is now Port Settings to avoid confusion.

Post the renaming in the software, updated the user facing documentation as well.

Save port config as Python

Finally, with the save port config infra now available, migrated the Save streams as python script to Save port config as python script by adding port settings and device groups to the code generation.

Documentation

Added a port save/open page to the user guide.

And updated the API guide to use Save port as Python option instead of Save streams as Python.

Updated internal developer documentation for future reference

And with that, it’s a wrap for this feature!

C++ Enum Support in LuaBridge2

me@srivatsp.com (Srivats P) — Wed, 19 Feb 2025 00:00:00 +0000

Recently I was doing some work to add support for Lua scripting to the Ostinato network traffic generator.

Although there are several solutions for Lua bindings for C++ out there, I decided to go with LuaBridge.

Note that there is a fork of LuaBridge called LuaBridge3, but it requires C++17 and with Ostinato I target a wide variety of platforms some of which are old and still use C++11.

So I decided to stick with the original LuaBridge.

LuaBridge however doesn’t support binding C++ enums. A little bit of googling and reading the documentation I found a solution which I will document here with examples.

First we will add support for passing C++ enums into Lua (as a number). Given the following C++ enums -

enum ProtocolIdType {
    ProtocolIdTypeLlc,
    ProtocolIdTypeEthernet,    
    ProtocolIdTypeIp,
    ProtocolIdTypeTcpUdp
};

enum CksumType {    
    CksumTypeIp,
    CksumTypeIpPseudo,
    CksumTypeTcpUdp
};

We add support for these enums by specializing the LuaBridge Stack template class for each enum type we want to support.

// Helper template for all enums we want to support
template <typename T>
struct EnumWrapper {
    static typename std::enable_if<std::is_enum<T>::value, void>::type
        push(lua_State* L, T value) {
        lua_pushnumber (L, static_cast<std::size_t>(value));
    }

    static typename std::enable_if<std::is_enum<T>::value, T>::type
        get(lua_State* L, int index) {
        return static_cast<T>(lua_tointeger(L, index));
    }
};

// Use EnumWrapper template for each enum type we want to support
namespace luabridge {
    template <>
    struct Stack<ProtocolIdType>
        : EnumWrapper<ProtocolIdType> {};

    template <>
    struct Stack<CksumType>
        : EnumWrapper <AbstractProtocol::CksumType> {};
}

Now you can pass the enums back and forth between C++ and Lua - the C++ enums will be treated as integers in Lua. So, you can call a Lua function pass in a C++ enum value.

LuaRef payloadProtocolId = getGlobal(L, "payloadProtocolId");
uint id = payloadProtocolId(ProtocolIdTypeIp);

LuaRef idType = getGlobal(L, "payloadProtocolIdType");
ProtocolIdType protocolId = idType();

Remember in Lua code these variables are integers. So any code referencing these will need to compare against the integer values.

if payloadProtocolIdType == 1 then
    -- do something
end

What if you wanted to use enums instead of integers in Lua? You can export them as Lua global variables by registering them with LuaBridge as a property under a Lua namespace (table).

// Helper template function for registering enum constants with LuaBridge
template<int T>
int enumValue() {
    return T;
}

getGlobalNamespace(L_)
    .beginNamespace("Protocol")
        // Enum ProtocolIdType
        .addProperty("ProtocolIdLlc", &enumValue<AbstractProtocol::ProtocolIdLlc>)
        .addProperty("ProtocolIdEth", &enumValue<AbstractProtocol::ProtocolIdEth>)
        .addProperty("ProtocolIdIp", &enumValue<AbstractProtocol::ProtocolIdIp>)
        .addProperty("ProtocolIdTcpUdp", &enumValue<AbstractProtocol::ProtocolIdTcpUdp>)
        // Enum CksumType
        .addProperty("CksumIp", &enumValue<AbstractProtocol::CksumIp>)
        .addProperty("CksumIpPseudo", &enumValue<AbstractProtocol::CksumIpPseudo>)
        .addProperty("CksumTcpUdp", &enumValue<AbstractProtocol::CksumTcpUdp>)
    .endNamespace();

After this registration, you can now reference the enums in Lua code as global variables.

local protocolId = Protocol.ProtocolIdIp

That’s it. Hope this has been helpful.

QtScript to QML Migration - Notes from the Trenches

me@srivatsp.com (Srivats P) — Sat, 18 Jan 2025 00:00:00 +0000

Ostinato uses the QtScript module to support custom protocol userscripts.

Qt5 had deprecated the QtScript module and replaced it with a new Javascript engine in the QML module. With Qt6, the QtScript module has been removed.

To migrate Ostinato from Qt5 to Qt6, I needed to figure out the migration changes for QScriptEngine and related classes which were part of QtScript.

A search for a migration guide from QScriptEngine to QJSEngine didn’t return any useful results - which is very unlike Qt documentation!

In the end, the changes needed turned out to be straightforward. YMMV but I’m sharing my notes here - this should get you started in the right direction.

Qmake/CMake changes

To start, I suggest you make changes to your qmake/cmake files to replace QtScript module with QML module. Make this change first so that you can trigger builds to identify compilation issues after every change.

In your qmake .pro file(s), replace

- QT += script
+ QT += qml

If using CMake, replace

- find_package(Qt5 REQUIRED COMPONENTS Core Gui Widgets Script)
+ find_package(Qt5 REQUIRED COMPONENTS Core Gui Widgets Qml)

QScriptEngine ⇒ QJSEngine

The main scripting engine class QScriptEngine has been replaced with QJSEngine. So do a find-replace for that along with the header file change -

-#include <QScriptEngine>
+#include <QJSEngine>

Note that instead of QScriptEngine::hasUncaughtException() you should use QJSEngine::evaluate() and check for isError() of the returned QJSValue.

-    engine_.evaluate(script);
-    if (engine_.hasUncaughtException())
+    QJSValue value = engine_.evaluate(script);
+    if (value.isError())

QScriptValue ⇒ QJSValue

The QScriptValue equivalent is QJSValue. But some method names differ between the two. e.g.

QScriptValue::isValid() → !QJSValue::isUndefined()
QScriptValue::isFunction() → QJSValue::isCallable()
QScriptValue::call() → QJSValue::callWithInstance() (for object method calls)
QScriptValue::toUInt32() → QJSValue::toUInt()
QScriptValue::toInt32() → QJSValue::toInt()

You can do a find-replace for the above or redefine QScriptValue to inherit from QJSValue and override the methods as shown below.

class QScriptValue : public QJSValue
{
    bool isValid() const { return !isUndefined(); }
    bool isFunction() const { return isCallable(); }
    uint toUInt32() const { return toUInt(); }
    int toInt32() const { return toInt(); }    
    // ...
}

I used this QScriptValue redefinition to start with to quickly get things compiling and working. Once compilation was successful and my tests were passing, I did find-replace for QScriptValue with QJSValue alongwith the changed methods and removed the redefinition.

These were the only changes I needed to make. You may find more changes are needed but hopefully this will get you a good way there.

Gotchas

There are a couple of gotchas though e.g. A C++ QObject’s dynamic properties are not accessible in the script which has access to the object. However, properties added to the object in the script are accessible to the C++ code.

doori

me@srivatsp.com (Srivats P) — Tue, 29 Oct 2024 00:00:00 +0000

The backstory

Watch the video first!

The other day while I was browsing my LinkedIn feed, it stuck me suddenly that the logo for dori.ai is very similar to the Doordarshan (DD) logo.

I was about to comment on this similarity on a WhatsApp group of friends and former colleagues.

Instead, I spent an hour making this video! 🤷🏽

The title doori is of course a portmanteau of Dori and Doordarshan.

A friend who works at Dori assures me that the logo designer likely never heard of DD or even encountered the DD logo anywhere.

About Dori

dori.ai is a company founded by former colleagues of mine, providing AI-enabled computer-vision solutions for manufacturing, logistics and other enterprise use-cases.

About Doordarshan

Doordarshan (DD) is India’s National Channel (aka public broadcaster) and the first ever broadcast TV channel launched in 1959 in India. My generation grew up watching the once-a-week movie broadcast on Sundays at a neighbor’s (who had a TV, very few households did) along with other kids from the street.

The montage of the DD logo with the signature tune composed by Pt. Ravi Shankar was broadcast every day when it began transmission for the day (‘twas not the days of 24-hour channels!) is iconic and instantly recognizable for everyone who grew up with it.

Disclaimer

This video mashup is a creative expression made solely for entertainment purposes.

This mashup is independently created and is not affiliated with, endorsed by, or connected to Dori.ai, Doordarshan, or any of their associated entities. All trademarks, logos, and media used remain the property of their respective owners.

PCAP import - a crash and a flash

me@srivatsp.com (Srivats P) — Tue, 15 Oct 2024 00:00:00 +0000

A few months ago an Ostinato customer reported the Ostinato GUI crashing when importing a large PCAP file (~600MB).

After a few failed attempts at reproducing the error on Linux, I realized the customer was using Windows and switched to that.

I was able to reproduce the crash with Ostinato on Windows - both with intelligent import enabled or disabled and set out to debug the same.

I started with intelligent import disabled since it would be easier.

No error of any kind. The GUI just closes and the process doesn’t exist. There was no obvious error.

To make matters worse, the crash happened at 99% and it took almost 50min to reach this point. It’s going to be a pain to debug if every repro takes an hour!

So the first order of business was to code up a hack to reduce the import time - while still being able to reproduce the crash. And that’s just what I did.

Phew! At least now I can run iterations without having to wait for an hour each time! Back to debugging. gdb just reported a SEGV and program terminated. I couldn’t even determine where in the code it was crashing.

printf-debugging wouldn’t work because we were importing millions of packets in a loop - not that I didn’t try! In fact, it took me down a tunnel where I suspected the crash was possibly inside libprotobuf and so I spent a day or two upgrading that - to no success.

Out of Memory?

I suspected we were running out of memory but there was enough free memory in the system (32GB RAM) and task manager showed a usage of ~1GB. I was really out of ideas after a day or two of debugging and decided to park it for some time and promptly got distracted with other Ostinato bugs and features.

Until 2 weeks ago when another customer reported the same issue. And so I got back to it once again - but still no luck!

Till I had the brainwave of using catch throw in gdb!

It did turn out to be a memory issue as suspected - an exception was being thrown from the C++ new memory allocator. But I still didn’t get it - there’s a lot of memory free in the system and the process is using only about 1GB as per task manager!

A real head-scratcher.

Virtual memory address space?

More googling and asking ChatGPT I got another hint - check the virtual address space of the process! A win32 binary has a 4GB virtual space of which only 2GB is available to the user space. The memory use reported by task manager was titled “Working set (memory)” which as per task manager itself means “Amount of physical memory currently in use by the process”.

So I ran procexp (process explorer) and to my good luck it reported the virtual address space usage by the process! So I ran the PCAP import again and watched the virtual address space of the process slowly increasing from the base 1GB as we imported more packets till it crashed and procexp showed a virtual space usage of ~2GB!

This also explains why I wasn’t able to repro on Linux - because my Linux dev env is 64-bit while my Windows Ostinato dev env is 32-bit!.

My Windows 10 itself is 64-bit but my Qt/Mingw chain is 32-bit due to historical reasons. There was never an incentive earlier to upgrade it to 64-bit. Setting up a new development environment with all the dependencies would take a couple of days (or more!) which is why I’ve never convinced myself to do it before now!

When I explained the problem to the customer, he was happy to switch to Linux to make progress (and he did!). So I bought myself some breathing time. And I decided to do some more experiments to better understand the problem and handle it gracefully.

Try and Recover

Unfortunately all my attempts at using a try…catch block failed to catch the exception and went straight to SEGV (segmentation fault).

I wasn’t sure what was happening. But one thing that I did see was that the exception can happen in any thread, not just the one that is doing the pcap import and consumes bulk of the memory. This is because the app and QT in other threads could be doing their own heap allocations which could trigger the same exception.

So unless I can try…catch all the threads, it’s not of any use!

Which brings us to the question - even if we wrapped try...catch around all threads - what do we gain?

To be able to identify the problem and handle it gracefully, try…catch will have to be granular, not at the very top of QThread::run() - the latter has no hope of recovery.

The only way to handle this seemed to be to monitor memory use ourselves in the thread during pcap import and avoid the crash altogether by aborting the import if we reach say 90% of available memory to the process.

Given that there is no portable cross-platform way to find out available free memory for a process, this is a fools errand for none to little gain. Which leads me to the following conclusion.

We’ll just upgrade Ostinato for Windows from 32-bit to 64-bit going forward!

Speed: From Hack to Fix

But it wasn’t all in vain.

The hack I wrote up to reduce the PCAP import time - I cleaned it up and converted it into a proper optimization to reduce the PCAP import time. A 500MB PCAP file with 450,000 packets of average size 1218 bytes is now imported in a flash!

Note this is only for raw PCAP import (i.e. when you uncheck the intelligent import option - which is by default checked when you open a PCAP file in Ostinato). With raw pcap import, packets are just read in as hex dumps into Ostinato.

I also reduced the time taken to delete existing streams (when importing a PCAP file in overwrite mode) and in the process direct delete of Ostinato streams in the GUI should also be MUCH faster now!

For intelligent PCAP import (aka via PDML) I intend to adopt a different strategy of writing a multi-core standalone converter program for the command-line that can be used by the Ostinato Python API as well.

Stay tuned for that!

Do you have any pet peeves while using Ostinato? Let me know in the comments below!

Making Ostinato's variable fields more flexible

me@srivatsp.com (Srivats P) — Tue, 10 Sep 2024 00:00:00 +0000

The Problem

Back in 2020, Javor wanted to test 2 million CGNAT scale using Ostinato but was running into a problem generating the flows he wanted.

The stream just generated sequentially flows one by one like this for example:
src-ip_1, src port 30001 -> dst-ip_1, port 1001
src-ip_2, src port 30002 -> dst-ip_2, port 1002
src-ip_3, src port 30003 -> dst-ip_3, port 1003

But I don’t want this. I would like to achieve the following way of order and incrementation:
src_ip1, src port 30001 -> dst-ip_1, port 1001
src_ip1, src port 30002 -> dst-ip_1, port 1002
src_ip1, src port 30003 -> dst-ip_1, port 1003
src_ip1, src port 30004 -> dst-ip_1, port 1004
src_ip1, src port 30005 -> dst-ip_1, port 1005
……
So I would like the stream to generate the packets in order to start first from src-ip and incrementally passthrough over all src-ports range ( for example range of 5000 ports ) and then to go to the next src-ip then to next src-ip and so on….

Irrespective of how many variable fields you have in a Ostinato stream, each variable field increments with every packet - which was not what Javor wanted.

He wanted a counter behavior similar to nested loops.

I filed issue #302 on the Ostinato issue tracker to keep track of the feature request.

The Workaround(s)

For Javor’s use case I came up with a workaround to do what he wanted using the variable field mask and wrote a blog post about it.

But the variable field mask workaround had a limitation of being able to use only those counts which are power of 2 (read the blog post for details). In addition to this limitation, the solution is not very intuitive for users to understand and use.

Last week another customer Ashok had a similar requirement to test Linux conntrack with scaled sessions. However, Ashok’s scale was lower and he was ok with creating multiple Ostinato streams - one each for a pair of source and destination UDP ports while each Ostinato stream would vary IP addresses 4000 times. 20 streams of 4000 IP address each makes for 80K flows.

The Solution

I figured it was time to dust off this issue and look into what it takes to implement it properly. I worked it out an example on paper first -

Now that I knew what needed to be done, I did a quick and dirty test with a hard-coded value -

And it worked!
Yup, that one-line change was the essential core of what was required!

Next, I needed to replace that hard coded value with a user-specified parameter.

I spent a fair amount of time trying to come up with a suitable name for the new parameter. Struggling to come up with a name I liked, I asked ChatGPT to suggest some names.

I short listed a few and sent a message to a few friends to solicit human feedback!

Based on the feedback received and my own thoughts, I decided to finalize on calling the parameter skip - I wasn’t entirely convinced of the name as it may lead to off-by-one confusion, but decided to go ahead with it.

Other code changes were just to add the skip field to the UI and the Protobuf RPC.

In the end, it turned out to be actually a very small code change that need not have been on the issue tracker for 4 years!

Note to Self

I am more motivated to implement or fix something when there’s a related trigger - in this case the new customer wanting this same feature. In the future I will take advantage of this increased motivation to at least do a prototype/scoping of the feature right away.

Wrapping up

Finally, to wrap up the feature, I added a test to the variable fields test suite and updated the Ostinato User Guide.

Is there a feature you’d like added to Ostinato? Let me know in the comments!

Ostinato Timed Transmit

me@srivatsp.com (Srivats P) — Wed, 31 Jul 2024 00:00:00 +0000

Back in May, I implemented Transmit Duration in Ostinato. This was a pre-requisite to the next feature I planned to implement - Timed Transmit. I even teased a GUI image fragment.

Unfortunately, 2 months passed before I could start working on Timed Transmit (Solopreneur, juggling multiple hats!)!

What’s Timed Transmit?

The user specifies a time duration for Transmit and the agent automatically stops the transmit after that time - simple!

If you are a techie, you’ve by now subconsciously translated the above feature definition to

Add a input field in the GUI, start a timer when starting transmit and on timeout stop the transmit.

And you’d be right - that’s what I thought too!

The implementation

It ended up taking a full week to implement - as it often happens, in my excitement I hadn’t spent the time to break down all that was needed to be done before beginning the work!

What all was needed, you ask? Here goes -

GUI mock-ups and experiments to finalize the widgets required, look and UX
Work that was done and then discarded as I found better solutions e.g. I added a new RPC first instead of changing the RPC signature for backward compatibility reasons; but when later I found a way to maintain backward compatiblity with the RPC change I discarded the new RPC
Controller-Agent RPC signature change requiring code changes on both controller and agent side
Debugging problems with QTimer - researching/reading documentation and then working around QTimer documented behaviour/limitations in multi-threaded applications
Handling various edge/error cases (the 80:20 rule!)
Researching, experimenting on ways to hide the RPC signature change so that existing scripts using the Ostinato Python API do not require changes while still supporting the newer RPC with transmit duration
Python API changes for the new RPC signature including figuring out how to deprecate the older startTransmit() RPC so that script users are forewarned about a future removal
Add automated RPC tests for both new and old (deprecated) APIs
Debugging RPC test failures because I didn’t read the debug logs carefully and went down the wrong tunnel (What a time sink!)
Python API documentation update
Save As Python Script changes in the GUI to use the newer RPC
User Guide documentation update
Feature video (see below)
This blog post! (a previous draft was so long and meandering, that I gave up in the middle of writing it and did a full rewrite).

This is as much a list for me to remind myself next time I underestimate a feature!

Timed Transmit in action

Hope, you enjoyed that!

What do you think of this new feature? Let me know in the comments below!

For more Ostinato related content, subscribe for email updates.

A Qt app with auto-resizing docks

me@srivatsp.com (Srivats P) — Tue, 18 Jun 2024 00:00:00 +0000

Most desktop apps have a main window with one child window which displays the document surrounded by toolbars, toolboxes and palettes. You know what I’m talking about.

Here’s an image from the Qt documentation.

Ostinato (a desktop app built using Qt), however, has a main window with 3 dock windows (QDockWidget) - Ports & Streams, Port Statistics and Logs, but no central widget.

When Ostinato starts, two child windows are programmatically set to equally share the vertical space. The third Logs window is added as a tab to the Port Statistics window, since it is informational and need not occupy screen estate, until required.

Now as long as you don’t change the size of the main window all is good. But if you increase (or maximize) the window, you end up with something like this -

Ugh!

All that wasted space and ugly looking to boot!

I immediately drag the two windows so that they take up the full space again, before continuing to work.

But some customers don’t. Or worse - don’t realize that they can resize the windows manually!

This wasn’t the highest priority item to address all these years and so I didn’t pay much attention to it other than cursory look for some solution, which I didn’t find.

Solution to what you ask?

To utilize all the space available for the Ostinato dock windows whenever the main window is resized.

Last week, I decided to take a look (read: I was tired after some serious debugging of a customer issue and needed something lighter to tackle!).

First try

My first thought was to hook into main window resizes using QMainWindow::resizeEvent() and programmatically resize the dock windows using QMainWindow::resizeDocks() to take up half the main window each as done at start up.

Which worked.

But, I immediately realized that before the user changes the main window size, they might have changed the layout or sizes of the dock windows. In such a situation, any change to the main window size should retain the user layout and only change the dock window sizes proportionally.

Second try

My second attempt was to try and calculate the new sizes of the dock windows as follows -

dockNewSize = (newMainWindowSize/oldMainWindowSize)*dockOldSize

Which is all fine and dandy. Within QMainWindow::resizeEvent() the input parameter gives us both the old and new size of the main window, but the dock windows have already been resized by Qt’s own logic and we don’t have access to the old dock sizes!

The only way to get access is to maintain the dock sizes across all dock and main windows resizes. That’s too much trouble for a lazy programmer like me!

That’s when I thought of a different way to approach the problem.

Third try

Don’t change the dock sizes at all. Instead, change the central widget’s size.

Of course, we didn’t have a central widget currently, so I created dummyCentral (a QWidget) and called QMainWindow::setCentralWidget(dummyCentral). Then, in resizeEvent(), call dummyCentral->resize(0, 0). Setting the central widget to a size 0 would make the dock windows automatically resize utilizing all the available space while retaining the current dock layout!

Pure Genius!

Except that it didn’t work.

(It should have, but it didn’t. Continue reading to know why!)

It was late and I decided to call it a day feeling mentally exhausted.

But as so often happens when you go to bed with a problem on your mind, you don’t sleep well and you are tossing and turning with thoughts of the problem still in your head.

“The central widget resizing strategy should work - there’s no reason why it shouldn’t!”

So I decided to take another look the next day.

Fourth time is the charm

While looking at the code for the (dummy) central widget that I had written, I realized I didn’t need to resize the dummy widget, if I set it to (0, 0) before I originally add it to the main window!

Unless, Qt handles (0, 0) size specially and doesn’t add it to the main window at all (No, it doesn’t handle it specially)!

Also, it helps if you can see something on the screen while debugging!

So, I removed the resizeEvent() method and created a QLabel as the central widget and set its text to XANADU.

Don’t ask me why - it was the first word that came to mind; something to do with Mandrake, the magician I presume!

When I ran the code, no XANADU on the screen!

Head-scratcher! But something to dig further!

Looking at the dummy widget creation code and the code around it, I noticed that I was calling setCentralWiget() before setupUi() (the main window uses a Qt .ui file).

That didn’t seem right, so I moved the call to setCentralWidget() after setupUi() and addDockWidget() calls.

Voila!

XANADU magically appears inside the main window!

Maximizing the main window resizes the docks to utilize all the space around XANADU!

Here’s my final cleaned-up code that I committed to the repo.

    QWidget *cw = new QWidget(this);
    cw->setFixedSize(0, 10); // fixed => both min and max size
    cw->setStyleSheet("background: red");

    ...

    // Set central widget after adding docks
    setCentralWidget(cw);

And here’s the result of adding those 4 lines of code - docks use all the space available!

The careful reader would have noted and may be wondering, why I set the dummy widget height to 10 instead of 0?

With a size of (0, 0) for the central widget, the left/right dock areas don’t seem to be available, so using a width of 0 ensures that the widget is not visible and a height of 10px ensures that there is some space between top and bottom docks that user can insert a dock in the left/right areas (to create a middle row, so to speak).

Even with the 10px, I find that it’s still quite fiddly dragging a dock to the left/right dock areas for the main window to recognize and visually indicate its readiness to accept the drop there! Increasing the height of the main window seems to help.

The red background is not really needed, but I left it in for debug purposes - I just need to change the width to a non-zero value and the red widget should show up and stand out on screen.

Here’s a screenshot with three ~~ducks~~ docks in a ~~row~~ column -

Ok, that pun was a stretch, I know!

But talking of stretches, the three row dock windows are not behaving like I’d like them to with respect to vertical stretch (logs window seems to hog the space, while I would like the stats window to get the space). The usual QWidget::sizePolicy::verticalStretch doesn’t work because these three windows are not in a normal QLayout, but in three separate QDockWidget laid out by the QMainWindow’s dock-aware layout manager.

I probably need to play around with sizeHint(), sizePolicy() or the minimum(), maximum() size constraints for one or more of those dock windows to get the vertical space utilization I’d like - but that’s a bunch of experiments for another day (and another blog post)!

For more Ostinato related content, subscribe for email updates.

Ostinato Show Transmit Duration

me@srivatsp.com (Srivats P) — Tue, 28 May 2024 00:00:00 +0000

The Ostinato latency and jitter feature sends a timing tagged (TTAG) packet roughly every 5 seconds. While implementing the feature, I often had to keep track of the transmit duration to make sure that enough number of timing packets were sent out for the test case I was trying to verify.

Manually tracking time was doable, but not ideal - so I added Show running Transmit Duration as a future feature to work on.

The latency and jitter shipped as part of Ostinato v1.3.0. Last week I got around to implementing the show transmit duration feature for the next release. It shows not just the running transmit duration, but also the last transmit duration as well.

Here’s a short video preview of this feature -

Hope, you enjoyed that!

What do you think of this new feature? Let me know in the comments below!

Next up, timed transmit with auto-stop -

For more Ostinato related content, subscribe for email updates.

Marketing to Developers

me@srivatsp.com (Srivats P) — Mon, 08 Apr 2024 00:00:00 +0000

As a product creator with an engineering background, I’m always trying to learn more about good marketing.

So when I saw this tweet from Kelsey Hightower, I got spelunking into the many many replies to it - from developers themselves!

Although my product Ostinato is not targeted at developers but network engineers - the latter audience I believe is similar enough for the replies to be useful to me.

I summarized and categorized the replies so that I can refer to in the future without having to re-read the thread again.

You can also download the above summary as a handy PDF.

Generate RARP (Reverse ARP) packets

me@srivatsp.com (Srivats P) — Wed, 28 Feb 2024 00:00:00 +0000

Recently, Ostinato customer Barry sent me this email -

I want to generate RARP request packets. I created an ARP packet in a stream. Saved the stream as a PCAP file. Hex edited the PCAP file to change the type to 0x0835. Imported and tried to send. It is not working.
Do you have a suggestion?

RARP (Reverse ARP) is a very old protocol that is not used these days and so Ostinato does not support it (Barry was working with some old equipment that still used RARP).

I looked up the RARP frame format and found that it is exactly the same as ARP except for two fields.

Here’s the RARP RFC 903 extract -

   RARP uses the same packet format that is used by ARP, namely:

      ar$hrd (hardware address space) -  16 bits
      ar$pro (protocol address space) -  16 bits
      ar$hln (hardware address length) - 8 bits
      ar$pln (protocol address length) - 8 bits
      ar$op  (opcode) - 16 bits
      ar$sha (source hardware address) - n bytes,
                                       where n is from the ar$hln field.
      ar$spa (source protocol address) - m bytes,
                                       where m is from the ar$pln field.
      ar$tha (target hardware address) - n bytes
      ar$tpa (target protocol address) - m bytes

   ar$hrd, ar$pro, ar$hln and ar$pln are the same as in regular ARP
   (see [1]).

The differences are -

The EtherType is 0x8035 instead of 0x0806
The Op codes are 3 & 4 (instead of the 1 & 2 ARP op codes)
- 3 - Reverse ARP request
- 4 - Reverse ARP reply

This was easily doable with Ostinato by using packet field overrides!

We start with creating a normal ARP packet in Ostinato -

To override the EtherType is straight-forward -

The ARP op code input box seems to offer only op code 1 and 2.

But actually you can just type in any op code value (yeah, I realize this is not very obvious).

In this case we need 3 (RARP Request) -

💡 TIP: You can override ANY packet field even if the protocol doesn’t seem to offer a way to do that!

That’s it!

We can now successfully send a valid RARP packet using Ostinato that can be verified using Wireshark -

Ostinato 1.4 RARP support

I have now added RARP support to Ostinato and this will be available starting v1.4.

All you need is to select the appropriate RARP op code and the EtherType will be automatically updated to RARP -

For more Ostinato related content, subscribe for email updates.

Ostinato goes Turbo at 100Gbps

me@srivatsp.com (Srivats P) — Tue, 23 Jan 2024 00:00:00 +0000

Happy New Year!

In true geek fashion, I spent the Christmas holidays tinkering - this time on the Ostinato network traffic generator’s high-speed add-on - Turbo Transmit. And I’m happy to share that Ostinato Turbo can now generate 100Gbps!

What was the trick you ask?

Horizontal scaling - using multiple cores.

The scaling problem

The base Ostinato software can use a max of only one core per port to transmit packets. Trying to transmit packets from multiple cores to a single ethernet port runs into contention at the kernel, driver and NIC thus not delivering the desired performance increase. There wasn’t much Ostinato could do about this, but it’s something that needed to be solved at the lower layers.

NIC manufacturers know about this problem and when designing high-speed NICs - 10Gbps or higher, they introduced multiple TX and RX queues (or sometimes combined TX+RX queues) for each port e.g. here’s Intel 40G XL710 -

$ ethtool -l enp4s0f0np0
Channel parameters for enp4s0f0np0:
Pre-set maximums:
RX:		0
TX:		0
Other:		0
Combined:	16
Current hardware settings:
RX:		0
TX:		0
Other:		0
Combined:	2

Once the hardware supported multiple queues, the Linux kernel and drivers were updated to utilize that.

Voila! No contention - as long as we use one CPU core per NIC queue!

We can now use multiple CPU cores independently.

Ostinato Turbo

The Ostinato Turbo add-on was architected to utilize multiple queues per port (and therefore multiple cores) from day one, but till now used only one core and one queue because as it turned out that was enough - thanks to the magic of AF_XDP fast path which delivered a whopping 20 million packets/sec with just a single core (for reference 10Gbps at 64 bytes is 14.8 Mpps). The catch - the NIC driver must support AF_XDP zero-copy which Intel and a few other NICs do.

In November, a customer asked me if Ostinato could do 100Gbps and I told him Turbo could do 100Gbps at 600 byte packet sizes (or higher) on the Intel E810 NIC. How, you ask? Because 100Gbps at 600 bytes comes to 20 Mpps - our single core/queue limit!

The customer was ok with this and went ahead. Here’s a screenshot from their test -

At around the same time, another customer reached out and asked about 10Gbps on AWS/Azure. My tests on those platforms revealed that although both supported AF_XDP, but not zero-copy - the max single core performance was only about 1 million packets/sec. For 10Gbps that works out to a minimum packet size of 1230 bytes to achieve 10Gbps which was not acceptable to the customer.

Going multi-core

It was time to extend Turbo to use multiple queues and cores - something I had designed-in but not implemented yet.

The first challenge was how to modify the highly-optimized and somewhat complex packet Tx code to utilize multiple cores - this took only a few days of experimenting.

The second challenge was to sync the Tx across cores after each stream - implementing a thread barrier solved that problem.

The last big challenge was to ensure per-stream stats and latency/jitter work across cores - a challenge again because of the highly-optimized and complex code for the same.

Before Christmas eve I had all the core functionality working and the productivity continued post Christmas too – till unexpectedly, I had to take a few days off.

Getting back into the groove proved to be a bit of a challenge - having to update the turbo-tune bash script didn’t help (No, I don’t enjoy bash scripting - even with ChatGPT help!). I was distracted and procrastinating.

Once I got through that slump and I thought I’m almost done, I hit a pesky race condition for some cases. You know how much fun those are!

All in all, I got everything done in about 4 weeks - which I must admit I’m quite proud of and a testament to architecting Turbo right to make this multi-core work easy.

With these changes you need just 8 CPU cores to generate 100Gbps traffic at 64 bytes on a AF_XDP zero-copy supported NIC such as the Intel E810!

I posted almost-daily updates on a build-in-public Twitter thread all through those weeks - as you can see from the screenshots above. This is something I’ve been doing off late for all big and small Ostinato features and if you are interested in Ostinato development, you should follow me on Twitter.

10Gbps on the public cloud

The multi-core Turbo enables not just 100Gbps on physical NICs but also 10Gbps on AWS and Azure cloud VMs (Google Cloud should also work, but I haven’t got around to test on it yet).

Here are my results on a Azure cloud VM -

An important point to note about the cloud VMs is that all of them impose a aggregate bandwidth limit across all interfaces of the VM instance which is typically lower than the interface link speed. This bandwidth limit varies according to instance type, so you have to choose a VM/instance type accordingly!

Turbo System Requirements

With this newly acquired ability to scale TX across multiple CPU cores - the system requirements to run Turbo are much simpler, cheaper and accessible now - all you need is a NIC (or vNIC) that supports multiple queues and a multi-core CPU!

Adopt Ostinato Turbo today and save your team and your company a gazillion dollars in test equipment!

Questions? Let me know in the comments!

If you found this interesting, help me spread the word by sharing this article using the share buttons below!

For more Ostinato related content, subscribe for email updates.

Ostinato show reserved ports broken in 1.3.0

me@srivatsp.com (Srivats P) — Tue, 14 Nov 2023 00:00:00 +0000

Close on the heels of releasing Ostinato v1.3.0, I discovered a regression - the Show Reserved Ports Only feature is broken. When this option is selected in v1.3.0 - the port list is shown correctly, but the port statistics is empty.

On investigation, it looks like this regression was introduced as part of the Port Stats UI improvement changes that went into 1.3.0. It was some minor code refactoring that I did, but apparently failed to verify. The fix has now been committed.

Workaround

Since the fix will only be available in 1.4.0 which is several months away, Ostinato 1.3.0 users can use the Ports Stats Config View as a workaround till then.

Click on the icon shown above to open the below dialog -

The left-side list is the list of ports that is NOT shown and the right-side list is the list of ports that is shown in the port stats window. You can move the ports from one list to the other by selecting one or more ports in either list and using the left/right arrow icons.

Yes, it’s a few more clicks unfortunately, so here are some tips -

You can shift-click and ctrl-click to select multiple ports in either list
You can double-click a port on either list to move it to the other list
You can drag the ports in the shown (i.e. right-side) list to change the order in which they appear

e.g. using the above, if you have reserved only a few ports, shift-select all shown ports (right-side list) and press ← (left arrow icon) and then double-click on the reserved ports in the left-side list to add them to the shown (right-side) list.

Found any other 1.3.0 regression? Let me know in the comments!

For more Ostinato related content, subscribe for email updates.

Latency and Jitter test

me@srivatsp.com (Srivats P) — Wed, 19 Jul 2023 00:00:00 +0000

Like most products, the number of new features requested for Ostinato is long. The effort to implement these also vary widely. Earlier this year I decided to implement one big feature for the next Ostinato version and I zeroed in on latency and jitter measurement.

The reason why this is big is because this requires timestamping packets which is an infra piece not yet supported in Ostinato. Also, doing this means changing the very core of the highly optimized packet transmission code in Ostinato which has not any major changes for almost 8-10 years (for good reason!) and any changes here need to be done carefully.

I estimated about a month to get this done for the base code after which I would have to do the same for Turbo which is a different codebase.

From a requirements point of view, I decided to implement this as a per-stream metric similar to packet loss, traffic rates and other existing per-stream stats. No additional configuration is needed beyond the basic stream stats configuration.

Basic questions

Before starting design and development, I needed to answer some fundamental questions.

Q. As a software traffic generator, would software (kernel/user level) timestamps be acceptable?

ping/iperf timing figures seem to be acceptable for a lot of users which are also based on software timestamps, so I made an assumption this will be the case for Ostinato too.

Q. Should TX timestamps be stored within the packet or only locally at the TX agent?

I decided to go with the latter.

Q. Should all TX packets be timestamped?

No, that’s likely to be too much additional cost.

Q. In that case, how often should TX packets be timestamped?

RFC 2544 suggests 30 sec while the commercial hardware traffic generators likely do it for every packet. 30 seconds seemed too long for me and 1 second too short (again from a cost point of view), so I arbitrarily chose a 5 seconds interval

Design and challenges

To identify timestamped packets, I added a TimingTag (TTAG) to the packet signature used for stream stats. The TTAG is essentially a unique identifier inserted on-the-fly in a packet every 5 seconds to correlate TX and RX timestamps for the same packet.

For these TTAG packets the TCP/UDP checksum would be incorrect unless we fix that as well on-the-fly. I spent a lot of time experimenting and finding ways to do this with least cost at Tx time only to discover much later that my approach would fail for interleaved streams. So, for the time being, I decided to live with incorrect checksums for TTAG packets - these are sent out only once every 5 seconds.

However I made sure sure that these packets do get counted at the RX end even if they have an incorrect TCP/UDP checksum.

The other challenge was that in case of multiple streams - in sequential or interlaved transmit mode - the one TTAG packet every 5 seconds should be for each TX stream. The stream rates, duration and packet counts are all user configured. After several misteps and a few different prototypes, I landed on something that worked for both modes and was performant.

On the RX side, we have to capture and parse each packet to identify if it is a TTAG packet and if so, record the RX timestamps. Fortunately, we already capture all stream stats signature packets and parse them. I just extended the code to parse the TTAG also and record the Tx timestamp.

Now that we have both TX and RX timestamps, we can diff the two and calculate the latency on a per-stream basis - we keeping doing this over the entire duration of the transmit to get the average latency.

And finally the protobuf API additions and GUI code to include the latency value from agent to controller as part of stream stats and to display the same in the stream stats window.

Next I had to do everything for the high-speed Turbo code base which is completely different than the base code. Moreover it has two completely different TX algos - one for low-speed high accuracy and another for high-speed high-throughput and changes were required for both.

All the while making these changes, I had to stop and refactor existing code multiple times to fit in the new code. And at other times rewrite the new code to be more optimal so as not to affect performance of Tx rates. Of course, I introduced a few bugs while making these changes - hopefully, I’ve caught and fixed ‘em all!

With latency available, calculating jitter was straight-forward.

In the end, it took more than 2.5K lines of code spread over 45 files and almost 4 months. So much for my estimate of 1 month!

As you may have guessed from the above screenshot, this feature was built in public with regular updates over the 4 months posted to a twitter thread.

Follow me on Twitter for more such build in public features in the future.

Per-stream latency and jitter will be available in the next Ostinato version - v1.3.

Ostinato Port Stats UI improvements

me@srivatsp.com (Srivats P) — Mon, 13 Mar 2023 00:00:00 +0000

Recently Ostinato user Ezra gave me some feedback about the Ostinato Port Stats UI -

I find it very unreadable when for every parameter there is Rx and bellow Tx.

I think it would be much clearer if all TX (byte, packets, whatever) will be on top and then all Rx (packets, byte, etc..) will be below.

I would order it as bellow. With the current order it is very difficult to see in a glance the parameter I am interested. I have to read all and even worse, the lines don’t start with Receive or Transmit so I must read the entire text to find what I am looking for. Please start every line with Rx or Tx and if possible, please collect them as I recommended._

Tx rate Byte/Sec
Tx rate Bit/Sec
Tx rate Frames/Sec
Tx Frames (total)
—– please leave a space here ——
Rx rate Byte/Sec
Rx rate Bit/Sec
Rx rate Frames/Sec
Rx Frames (Total)
—- please leave a space here —–
Rx Errors
Rx Drop Errors
Rx Fifo Errors

For reference, here’s the current Port Stats UI which Ezra is referring to.

One of the main reasons users look at the port stats is to compare the Tx and Rx stats on 2 or more ports, so having them close to each other, I feel is better than grouping all Tx together followed by all Rx.

So, I started by fixing the order, so that send and receive for all stats are always consistent (it wasn’t already for some inexplicable reason) - send followed by receive. I also removed the Byte Send/Receive Rate stats as networking users deal in bitrates rather than byte-rates.

Next, I extended the alternate background color of the stats to the row headers as well for more of a visual cue.

A nice side-effect of doing the above that emerges is that the send rows all have a gray background, while the receive rows (except the drop/error ones) all have a white background.

This alternate header row colors doesn’t work on Windows because, by default, Qt will use a platform native look and Windows doesn’t allow alternate background color of header rows. A workaround is to emulate the Windows style by starting Ostinato as ostinato -style windows

I then visually grouped the related stats using a horizontal line.

Extending the horizontal line to the row headers would be nice, but was too complicated to implement, so I let that slide.

Finally, I moved on to Ezra’s suggestion to rename the stats starting withTx/Rx.

Using the Tx/Rx terminology instead of Send/Receive made the pair names the same length and differ only in one character - in my opinion this makes it difficult to quickly distinguish between them.

So I reverted the changes and retained the existing names, just rephrasing them so that they all start with Send or Receive.

What do you think of these changes? Let me know in the comments below!

For more Ostinato related content, subscribe for email updates.

Ostinato RX stream stats always zero

me@srivatsp.com (Srivats P) — Mon, 27 Feb 2023 00:00:00 +0000

Ostinato user Leslie reached out to me to report that Ostinato per-stream stats are not working on the RX side (always 0) with VLAN-tagged packets on Ubuntu 22.04 LTS.

Upon investigation, it turns out to be a libpcap issue.

But first, a bit of background.

To implement per-stream statistics, Ostinato tags these packets with the magic value of 0x1d10c0da as the last 4 bytes of the packet, just before the Ethernet FCS. This serves as a signature to quickly check if it’s an Ostinato generated packet. We can then read the stream GUID which is encoded just before the signature.

To increase performance, Ostinato uses a pcap filter to read into user space only packets with the Ostinato signature. Simply put, we need to use the following capture filter -

ether[len - 4:4] == 0x1d10c0da

However, there’s a problem with this scheme. Say, we send out these packets with signature. Somewhere in transit or at the destination, the packet encounters an error and a ICMP error is sent back. The ICMP error packet contains the original packet as payload and so the ICMP error packet will also have the Ostinato signature at the end!

To avoid matching these packets, we modify our capture filter as -

(ether[len - 4:4] == 0x1d10c0da) and not icmp

But that matches only untagged packets, not vlan-tagged packets; so we modify the capture filter as -

(ether[len - 4:4] == 0x1d10c0da) and not (icmp or (vlan and icmp))

And that’s where we run into the libpcap problem.

The BPF instructions generated by libpcap for the above filter are incorrect!

(Note that the actual capture filter string used by Ostinato is a bit more complicated for reasons not relevant to this blog post)

More details about the generated BPF for this problem and discussions with Wireshark and libpcap teams are available at the below links, if you’re curious -

The problem is Linux specific (likely due to BPF extensions) and is seen only for vlan tagged packets.

Ubuntu 22.04 includes libpcap 1.10.1. The problem is present even with 1.9.x and 1.8.x. libpcap 1.7.x introduced the BPF vlan check (although the code looks very different).

So what’s the solution?

While I figure out how best to handle this in Ostinato, for now the solution is to use libpcap 1.6.2. Here are instructions to build that from source -

wget https://www.tcpdump.org/release/libpcap-1.6.2.tar.gz
tar zxvf libpcap-1.6.2.tar.gz
cd libpcap-1.6.2
./configure
make

That will generate libpcap.so.1.6.2.

But Ostinato usually is linked against libpcap.so.0.8 (ldd drone | grep libpcap). So we create a softlink by that name in the same libpcap-1.6.2 folder.

ln -s libpcap.so.1.6.2 libpcap.so.0.8

Next step is to make Ostinato use this libpcap than the system one. For that we use LD_LIBRARY_PATH when starting Ostinato (or drone).

LD_LIBRARY_PATH=/path/to/libpcap-1.6.2/ ostinato

Starting Ostinato like that should fix the rx stream stats problem.

For more Ostinato related content, subscribe for email updates.

Ostinato in a container

me@srivatsp.com (Srivats P) — Tue, 10 Jan 2023 00:00:00 +0000

For a long time now, I’ve been wanting to package Ostinato as a container. But never got around to it.

Until now.

As 2022 came to a close, I took the plunge, installed Docker, read a few tutorials and created my first container, packaging the Ostinato agent Drone.

Packaging the agent (Drone) and the Ostinato GUI together took a few days as I needed to figure out how best to package a X windows GUI app inside a container — I decided to take the VNC route as it’s something Ostinato users (e.g. those using it with GNS3/EVE-NG/CML) are familar with.

Ostinato being a Qt GUI app, my first attempt was to use the Qt built-in VNC server (just pass the arg -platform vnc to any Qt GUI app). While that worked, the dialogs and windows were not resizable — probably because I hadn’t included a Window Manager in the image. I would also need to package in Wireshark to view capture files, so a minimal desktop or window manager was anyway called for. I went ahead with FLWM — the same one that the Ostinato VM uses. Now with multiple apps and a GUI desktop environment, I needed a full blown VNC server instead of the Qt built-in one. After a bit (read too much!) of research, I settled on x11vnc.

After much reading of docs and some experimentation, I now had 2 Ostinato docker containers -

$ docker images
REPOSITORY                    TAG        IMAGE ID       CREATED       SIZE
ostinato/ostinato             v1.2.0-1   11ab5ac21cbe   3 days ago    751MB
ostinato/agent                v1.2.0-1   c1be11ba26fd   3 days ago    204MB
<none>                        <none>     c096bf3cb0da   3 days ago    206MB
<none>                        <none>     2a0175a90aba   5 days ago    754MB
<none>                        <none>     edc5c451a16f   6 days ago    751MB
<none>                        <none>     908a68fb0900   6 days ago    754MB
ubuntu                        22.04      6b7dfa7e8fdb   4 weeks ago   77.8MB
ghcr.io/nokia/srlinux         latest     db41df9a3ad0   5 weeks ago   2.8GB

Next step was to try the Ostinato container with Containerlab which was fairly straightforward (except for some SR Linux config snippet that the containerlab discord members kindly helped me with) -

Ostinato for Containerlab is now available on the Ostinato website.

For more bespoke environments, the generic containers for the latest Ostinato release (v1.2.0) are now available to all Ostinato Super Bundle subscribers.

How will you use the Ostinato container? Let me know in the comments below.

For more Ostinato related content, subscribe for email updates.

Distinguish field names for combo protocols

me@srivatsp.com (Srivats P) — Tue, 20 Dec 2022 00:00:00 +0000

A few days ago, Ostinato user João reached out on Twitter and requested for a way to distinguish between SVLAN and CVLAN fields for a VlanStack protocol when trying to configure the vlan field to vary across packets.

So I made some code changes for “combo” protocols such as VlanStack (SVLAN+VLAN), IPv4over4, IPv4over6 etc.

For cases such as IP 4over4, the prefix Outer/Inner is used instead.

These changes will also reflect in the Packet View and Find & Replace as well.

These changes will be part of the next Ostinato v1.3.

If something in Ostinato is troubling you or if you are missing a feature in Ostinato that would make you more effective, reach out via Twitter or the Forum and I’ll try to implement it.

For more Ostinato related content, subscribe for email updates.

Ostinato 1.2.0 released

me@srivatsp.com (Srivats P) — Mon, 29 Aug 2022 00:00:00 +0000

It’s been more than 2 years since the last Ostinato release - v1.1.

I’m happy to announce the release of Ostinato 1.2.0 with many new features.

Here are some of them -

🧱 Send IP/GRE packets

📈 New per-stream stats - duration, avg tx/rx pkt/bit rates

🔀 IMIX traffic made easy

Details

🔎 Bulk edit packets using Find & Replace

Details

💯 Port traffic rate configuration as load %

Details

🕑 Export PCAP with nanosec resolution

Details

🎨 GUI Themes - both light and dark!

Details

➕ a bunch of bugfixes!

You can check out the full changelog.

Questions? Suggestions? Feedback?

Let me know in the comments below!

Want to be notified about new Ostinato features as they are developed? Follow me on twitter where I tweet about Ostinato developement.

For more Ostinato related content, subscribe for email updates.

Using Ostinato with mininet

me@srivatsp.com (Srivats P) — Tue, 22 Feb 2022 00:00:00 +0000

I’d heard about mininet off and on, but had never really tried it myself.

Mininet is a Linux based network emulator which creates a network of virtual hosts, switches, controllers, and links all on a single Linux host. It does this using containers (namespaces essentially) and connects them using veth pairs.

An Ostinato user was trying to use Ostinato with mininet and was not seeing flows on the OVS switch and so reached out to me. I used this opportunity to try mininet out.

I installed the mininet VM as recommended. The VM doesn’t have a desktop environment installed and the recommendation is to use X11 forwarding to run GUI apps like Wireshark. Ostinato is also a GUI app, so will also require X11 forwarding.

But, first a couple of notes about X11 forwarding for those using mininet VM on a Windows host.

Both the native Windows ssh client and WSL ssh client were not able to do X11 forwarding - not sure if they don’t support it or I was doing something incorrect. I was able to use good ol’ Putty with X11 forwarding successfully.

Both Wireshark and Ostinato are Qt based GUI apps. However Qt5 and Xming don’t play nicely (some error about XInput not being supported). I replaced Xming with VcXsrv and was able to use that successfully.

Back to mininet.

Mininet has several predefined topologies with option to create custom ones. I went with the default minimal topo -

mininet@mininet-vm:~$ sudo -E mn
*** Creating network
*** Adding controller
*** Adding hosts:
h1 h2
*** Adding switches:
s1
*** Adding links:
(h1, s1) (h2, s1)
*** Configuring hosts
h1 h2
*** Starting controller
c0
*** Starting 1 switches
s1 ...
*** Starting CLI:

mininet> net
h1 h1-eth0:s1-eth1
h2 h2-eth0:s1-eth2
s1 lo:  s1-eth1:h1-eth0 s1-eth2:h2-eth0
c0

mininet>

Visually, this is how the above topo looks -

The h1-eth0 and h2-eth0 interfaces are inside the h1 and h2 containers respectively while s1-eth1 and s2-eth2 are in the default/root namespace and ifconfig on the mininet prompt and the VM prompt confirm the same.

mininet> h1 ifconfig
h1-eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.0.0.1  netmask 255.0.0.0  broadcast 10.255.255.255
        ether 92:69:d5:2c:1b:d0  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

mininet> h2 ifconfig
h2-eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.0.0.2  netmask 255.0.0.0  broadcast 10.255.255.255
        ether 16:0d:d7:bc:be:a8  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

mininet>

mininet@mininet-vm:~$ ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.1.35  netmask 255.255.255.0  broadcast 192.168.1.255
        ether 08:00:27:3c:f7:01  txqueuelen 1000  (Ethernet)
        RX packets 1573653  bytes 99181783 (99.1 MB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 2759420  bytes 3961121460 (3.9 GB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 413122  bytes 3819486904 (3.8 GB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 413122  bytes 3819486904 (3.8 GB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

s1-eth1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        ether da:2b:00:64:0e:fa  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

s1-eth2: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        ether 1e:4d:c5:e6:4d:ae  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

To generate traffic into the switch, we need to run Ostinato inside the host containers, NOT in the root namespace. To do that, start an xterm on h1 and then start Ostinato from inside that xterm

mininet> xterm h1
mininet>

If you don’t see the xterm window, make sure you started mininet by passing -E to sudo like this: sudo -E mn

From the h1 xterm window (not the main mininet VM), start Ostinato -

root@mininet-vm:~# ostinato &
root@mininet-vm:~#

If you don’t see any ports in Ostinato, see this Ostinato FAQ.

Make sure that Ostinato lists the h1-eth0 port.

From here on, create Ostinato streams, devices and the usual Ostinato stuff.

To examine flows on the openflow switch, use dpctl dump-flows on the mininet prompt -

mininet> dpctl dump-flows
*** s1 ------------------------------------------------------------------------
 cookie=0x0, duration=202.156s, table=0, n_packets=9929, n_bytes=595740, idle_timeout=60, priority=65535,udp,in_port="s1-eth1",vlan_tci=0x0000,dl_src=26:e5:29:01:96:5b,dl_dst=d6:41:7c:21:0b:02,nw_src=10.0.0.1,nw_dst=10.0.0.2,nw_tos=0,tp_src=0,tp_dst=0 actions=output:"s1-eth2"
 cookie=0x0, duration=49.649s, table=0, n_packets=2434, n_bytes=146040, idle_timeout=60, priority=65535,tcp,in_port="s1-eth1",vlan_tci=0x0000,dl_src=26:e5:29:01:96:5b,dl_dst=d6:41:7c:21:0b:02,nw_src=10.0.0.1,nw_dst=10.0.0.2,nw_tos=0,tp_src=0,tp_dst=0 actions=output:"s1-eth2"
 cookie=0x0, duration=49.636s, table=0, n_packets=2434, n_bytes=131436, idle_timeout=60, priority=65535,tcp,in_port="s1-eth2",vlan_tci=0x0000,dl_src=d6:41:7c:21:0b:02,dl_dst=26:e5:29:01:96:5b,nw_src=10.0.0.2,nw_dst=10.0.0.1,nw_tos=0,tp_src=0,tp_dst=0 actions=output:"s1-eth1"
 cookie=0x0, duration=0.652s, table=0, n_packets=0, n_bytes=0, idle_timeout=60, priority=65535,icmp,in_port="s1-eth2",vlan_tci=0x0000,dl_src=d6:41:7c:21:0b:02,dl_dst=26:e5:29:01:96:5b,nw_src=10.0.0.2,nw_dst=10.0.0.1,nw_tos=0,icmp_type=3,icmp_code=3 actions=output:"s1-eth1"
 cookie=0x0, duration=90.647s, table=0, n_packets=2, n_bytes=84, idle_timeout=60, priority=65535,arp,in_port="s1-eth1",vlan_tci=0x0000,dl_src=26:e5:29:01:96:5b,dl_dst=d6:41:7c:21:0b:02,arp_spa=10.0.0.1,arp_tpa=10.0.0.2,arp_op=2 actions=output:"s1-eth2"
 cookie=0x0, duration=44.568s, table=0, n_packets=1, n_bytes=42, idle_timeout=60, priority=65535,arp,in_port="s1-eth2",vlan_tci=0x0000,dl_src=d6:41:7c:21:0b:02,dl_dst=26:e5:29:01:96:5b,arp_spa=10.0.0.2,arp_tpa=10.0.0.1,arp_op=1 actions=output:"s1-eth1"

mininet> dpctl dump-flows tcp
*** s1 ------------------------------------------------------------------------
 cookie=0x0, duration=55.213s, table=0, n_packets=2706, n_bytes=162360, idle_timeout=60, priority=65535,tcp,in_port="s1-eth1",vlan_tci=0x0000,dl_src=26:e5:29:01:96:5b,dl_dst=d6:41:7c:21:0b:02,nw_src=10.0.0.1,nw_dst=10.0.0.2,nw_tos=0,tp_src=0,tp_dst=0 actions=output:"s1-eth2"
 cookie=0x0, duration=55.200s, table=0, n_packets=2706, n_bytes=146124, idle_timeout=60, priority=65535,tcp,in_port="s1-eth2",vlan_tci=0x0000,dl_src=d6:41:7c:21:0b:02,dl_dst=26:e5:29:01:96:5b,nw_src=10.0.0.2,nw_dst=10.0.0.1,nw_tos=0,tp_src=0,tp_dst=0 actions=output:"s1-eth1"

mininet> dpctl dump-flows udp
*** s1 ------------------------------------------------------------------------
 cookie=0x0, duration=210.107s, table=0, n_packets=10323, n_bytes=619380, idle_timeout=60, priority=65535,udp,in_port="s1-eth1",vlan_tci=0x0000,dl_src=26:e5:29:01:96:5b,dl_dst=d6:41:7c:21:0b:02,nw_src=10.0.0.1,nw_dst=10.0.0.2,nw_tos=0,tp_src=0,tp_dst=0 actions=output:"s1-eth2"

mininet>

The full Ostinato arsenal of features is now available at your disposal to generate the traffic flows that you need to verify your SDN controller and openflow switch.

For more Ostinato related content, subscribe for email updates.

Ostinato Themes

me@srivatsp.com (Srivats P) — Tue, 08 Feb 2022 00:00:00 +0000

Amongst mostly positive Ostinato reviews on Capterra, Ostinato user Tamás mentions that the Ostinato UI looks “old school”. And I agree.

Ostinato uses Qt as the GUI toolkit. It uses Qt widgets and was built for Desktops before QtQuick for mobile first GUIs became available. Being the one-man-band of Ostinato and having the wisdom gleaned from being a programmer for 30+ years, I know that redoing the UI is not prudent (much as I might want to start from a clean slate!).

However, I knew that Qt offered ways to change the look and feel of the UI widgets using Qt Style Sheets – CSS brought to desktop widgets. Now, while I know CSS concepts and can tweak a few things when pushed, I’m not really a front-end guy. Moreover, from experience I know that if I start on that road, I will end up spending days and weeks pixel tweaking!

So, I looked for ready made QSS themes. What comes up as the first result is QSS Stock. While it has several themes, they didn’t feel very professional to my eyes. Next, I looked for open source projects and found two that I liked.

The first - qt-material aims to style Qt widgets as per the Material Design guidelines. The second was QDarkStyle.

I made the necessary tweaks required to integrate their respective QSS files along with the icons with Ostinato. Here’s a video with the end results -

There are some things that I’m not happy with and would like to change, but for the next version Ostinato 1.2, I plan to release it as it is and tag it as EXPERIMENTAL.

What do you think? Share your thoughts and feedback in the comments below.

For more Ostinato related content, subscribe for email updates.

Import and export PCAP files with nanosecond timestamps

me@srivatsp.com (Srivats P) — Tue, 25 Jan 2022 00:00:00 +0000

Ostinato has supported importing and exporting PCAP files since 2011. In 2020, I added support to import non-PCAP format (e.g. pcapng) capture files by converting them to PCAP files internally.

Ostinato customer Orhan recently reached out to me to investigate why an imported PCAP file when exported back to PCAP after editing the packets in Ostinato had incorrect timestamps.

After debugging, I found that this was due to the input pcap file having nanosecond precision timestamps while Ostinato only supports microsecond precision timestamps (the original PCAP format) — in some cases.

While a nanosecond PCAP was imported correctly in intelligent mode and would replay with the same nanosecond accuracy to the extent possible, saving a PCAP always used microsecond timestamps.

A nanosecond PCAP file imported in raw (aka non-intelligent) mode lost its nanosecond precision and fell back to using microsecond timestamps.

Starting v1.2, Ostinato saves PCAP files with nanosecond precision timestamps. Also, importing a nanosecond pcap in raw mode is now natively supported without losing the timestamp precision.

Note: Packet captures triggered from Ostinato, however, still use microsecond precision - this will be changed in a future version. A workaround, for now, is to directly use tcpdump (--time-stamp-precision=nano) or tshark (--time-stamp-type <type>) for capturing packets. Note that the NIC, OS and libpcap version (>= 1.0) should support nanosecond timestamps - many of them don’t.

For more Ostinato related content, subscribe for email updates.

Edit PCAP using Ostinato

me@srivatsp.com (Srivats P) — Mon, 20 Dec 2021 00:00:00 +0000

A popular use case for Ostinato is to import and replay PCAP (packet capture) files - and that’s one reason why Ostinato is Wireshark in Reverse!

Sometimes you need to edit the packet contents before you replay e.g. changing IP or MAC addresses, TCP/UDP port numbers etc. You can do that with Ostinato after you import the PCAP file.

However, you need to edit each imported packet individually (each PCAP packet is imported as a Ostinato stream). This is doable if you have only a few packets to edit, but quickly gets painful and downright impossible if you have hundreds or thousands of packets that you need to rewrite.

Here’s where tools like bittwiste and tcprewrite can help rewrite all packets as per your requirement. But they have their own limitations.

Starting Ostinato v1.2, you can do this bulk rewrite of protocol fields directly within Ostinato using the new Find & Replace Feature. And you can rewrite ALL fields of ALL protocols natively supported by Ostinato.

The first thing to ensure though is that the Recalculate Checksums option is selected when importing the PCAP file. Without this, when you replay edited packets, you will end up with packets being dropped due to incorrect checksums.

Once your packets are imported as Ostinato streams, select Find & Replace from the streams context-menu to open the Find & Replace Dialog -

The above image should be pretty much self-evident. You select the protocol and field that you want to change, enter the find and replace values and you are good to go. You can use match/replace masks for partial rewrites. See the Ostinato find & replace documentation for further details.

💎 Find & Replace works not only for PCAP editing but also for Ostinato streams created from scratch

Once you’ve rewritten the protocol fields, you can replay and transmit the edited packets or save them as a PCAP file.

Here’s a video showing how Find & Replace works -

For more Ostinato related content, subscribe for email updates.

Send IGMP packets

me@srivatsp.com (Srivats P) — Tue, 05 Oct 2021 00:00:00 +0000

I have always written quick ‘n dirty scripts and small utilities for myself, all through my career and even earlier in college. As a matter of fact, Ostinato is one such tool that just kept growing bigger and bigger 😺!

But before there was Ostinato, there was a IGMP packet generator that I wrote to verify IGMP Snooping on networking devices. I recently discovered the source code for this tool in my backups and am now releasing it as a free tool.

Presenting sendigmp

For more Ostinato related content, subscribe for email updates.

Generate IMIX traffic with Ostinato

me@srivatsp.com (Srivats P) — Tue, 14 Sep 2021 00:00:00 +0000

Ostinato can craft packets and generate network traffic for various protocols at various packet sizes.

But first things first.

What is IMIX traffic?

IMIX or Internet Mix is a traffic profile representing typical Internet traffic. It’s a mix of specific packet sizes and number of packets of that size.

Whether the traffic profile truly mirrors the Internet traffic is debatable, still having a standard IMIX profile is a useful benchmark. There’s no official standard for IMIX - each vendor seems to have their own variant. For this post, we will go with Simple IMIX as described in Wikipedia.

Simple IMIX

Eth Packet Size	No. of Packets	% Packet Distribution
64	7	58.33%
594	4	33.33%
1518	1	8.33%

The Wikipedia page specifies IP packet size. We derive the Ethernet packet size as follows

ethernetPacketSize = max(ipPacketSize+18, 64)
  18 = size of (dmac + smac + ethtype + fcs)
  64 = minimum ethernet packet size

That’s an average packet size of ~362 bytes.

Create IMIX streams in Ostinato

With Ostinato 1.2.0 or later

In Ostinato 1.2.0 or later, simply select IMIX as the frame length mode,

and change the number of packets to at least 12 (7+4+1).

That’s it. Seriously!

This method uses the Simple IMIX distribution as specified above. If you want to use a different distribution, use the below method.

With Ostinato 1.1 or earlier

We need to create 3 streams in Ostinato, one for each row in the Simple IMIX table above and send packets in a loop.

If you prefer video, watch the IMIX demo from my talk at Sharkfest 2020.

Here are the detailed steps -

Create the first stream of frame length 64 and number of packets as 7
Configure the protocols and protocol fields for the stream as required
Duplicate the above stream for a count of 2
Rename the new streams as IMIX stream 2 and 3
Edit the second stream to set frame length 594 and number of packets as 4
Edit the third stream to set frame length 1518 and number of packets as 1
Set the third stream to loop back to first stream
Click on Apply followed by Start Transmit

Use interleaved streams to mix up the IMIX packets

If you look at the Wireshark capture, you’ll notice that streams are sent in sequence, i.e. 7 packets of size 64, followed by 4 packets of size 594 and finally 1 packet of size 1518, before it repeats that same pattern

If you are wondering why Wireshark shows 4 bytes less for all packet sizes, that’s because Ostinato configuration of packet size is including FCS, while Wireshark shows packet sizes without FCS (the NIC strips it off).

Although this sequence should not cause a problem, if you want to mix up the sizes, you can configure the port to use Interleaved streams instead of Sequential streams

In interleaved mode, all streams are transmitted simultaneously, so you will get a mix of the configured packet sizes, but sent in the same 7:4:1 ratio.

Note that, after changing the port mode to interleaved, you have to edit the streams so that the Packets/sec rate is in the 7:4:1 ratio. A quick way to do this is to set each stream pps rate to 7, 4 and 1 respectively and then change the Avg pps rate from 12 to whatever value you want - any subsequent change to the Avg pps rate will retain the 7:4:1 rate ratio across the streams.

Verify IMIX using Wireshark

To verify, capture the packets in Wireshark, go to Statistics | Packet Lengths and look at the Percent column -

Verify that the Percent values match the % Packet distribution in the Simple IMIX table above.

Gotchas

Make sure your DUT/SUT supports packet size of 1518 - in my experiments with virtual topologies, VirtualBox would drop packets > 1504 bytes.

Other IMIX profiles

If you want to try other IMIX profiles, see the Ostinato IMIX Guide.

For more Ostinato related content, subscribe for email updates.

Ostinato at FOSDEM 2021

me@srivatsp.com (Srivats P) — Thu, 09 Sep 2021 00:00:00 +0000

Earlier this year I gave a talk at FOSDEM 2021 on how to add more protocols to the Ostinato traffic generator. FOSDEM is one of Europe’s largest volunteer-driven open source and free software conferences.

The video and slides are posted on the Ostinato website.

How to generate Geneve encapsulated packets

me@srivatsp.com (Srivats P) — Tue, 31 Aug 2021 00:00:00 +0000

A decade or so ago, virtualization took the world by storm and that introduced multiple L2 tunneling encapsulations such as VxLAN, NVGRE and STT.

In 2020, IETF also standardized GENEVE (Generic Network Virtualization Encapsulation). See RFC 8926 for details. Before you dismiss this as yet another L2 tunnel encapsulation, know that AWS already uses Geneve!

Similary to VxLAN, Geneve is carried as a UDP payload and encapsulates an inner Ethernet or IP packet.

+=====+====+=====+========+================+=================+
| Eth | IP | UDP | GENEVE | (Inner) Eth/IP | (Inner) payload |
+=====+====+=====+========+================+=================+

Here’s the Geneve packet format.

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|Ver|  Opt Len  |O|C|    Rsvd.  |          Protocol Type        |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|        Virtual Network Identifier (VNI)       |    Reserved   |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                                                               |
~                    Variable-Length Options                    ~
|                                                               |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

To generate Geneve packets using Ostinato, we need to write a protocol builder userscript. Since the frame format is similar to VxLAN, I used the VxLAN userscript and made modifications required for Geneve, all of which was fairly straightforward.

Here’s the userscript you need to use for Geneve -

protocol.name = "Geneve"

protocol.protocolFrameSize = function() {
  return 8;
}

protocol.protocolFrameValue = function(index) {
    var ver = 0;	    // 0 to 3
    var opt_len = 0;	    // 0 - 63
    var control_pkt = 0;    // 0 or 1
    var critical_option = 0;// 0 or 1
    var vni = 0x749127;	    // example VNI value
    var proto = protocol.payloadProtocolId(Protocol.ProtocolIdEth);

    if (proto == 0)
        proto = 0x6558      // Assume ethernet aka Transparent Bridging

    var pfv = new Array(8);

    pfv[0] = (ver << 6 | opt_len) & 0xFF;
    pfv[1] = (control_pkt << 7 | critical_option << 6) & 0xFF;
    pfv[2] = (proto >> 8) & 0xFF;
    pfv[3] = proto & 0xFF;

    pfv[4] = (vni >> 16) & 0xFF;
    pfv[5] = (vni >> 8) & 0xFF;
    pfv[6] = vni & 0xFF;
    pfv[7] = 0;

    return pfv;
}

protocol.protocolId = function(id_type) {
    if (id_type == Protocol.ProtocolIdTcpUdp)
        return 6081;
    return 0;
}

Let’s verify the script by capturing our Geneve packet in Wireshark (click to enlarge) -

Some limitations -

The above userscript doesn’t generate Geneve options
For the outer UDP, override the checksum field to 0 - this is to avoid an Ostinato TCP/UDP checksum bug when you have 2 (or more) IP headers in a packet.

If you need details on how to create all the protocols headers (outer, inner) for a Geneve packet, see Crafting VxLAN packets using Ostinato - it’s exactly the same, just replace the VxLAN userscript with the Geneve userscript above.

Or just download this sample Ostinato stream and open it in Ostinato directly.

For more Ostinato related content, subscribe for email updates.

Ostinato at SharkFest 2020

me@srivatsp.com (Srivats P) — Tue, 24 Aug 2021 00:00:00 +0000

It’s been almost a year, but I just realized that I hadn’t posted about my Ostinato talk at Sharkfest 2020 here on my blog.

As a long time Ethereal/Wireshark user and Gerald Combs fan, I’ve always wanted to attend Sharkfest, but being based in India I never had the opportunity. But COVID-19 turned all conferences virtual including the 2020 edition of SharkFest - thus, I was able to not just attend, but also present last year.

The video and slides are already posted on the Ostinato website.

How to open PCAP files in Ostinato on EVE-NG, GNS3 and CML

me@srivatsp.com (Srivats P) — Wed, 18 Aug 2021 00:00:00 +0000

This post is written for EVE-NG. But the procedure is exactly same for GNS3 and CML as well.

Ostinato (standard) for EVE-NG runs the Ostinato GUI controller and Drone agent within a VM accessed via VNC. This makes it very easy to use for almost all workflows.

However, it introduces an extra step for the following Ostinato workflows -

Open PCAP files for edit/replay
Save Ostinato streams
Save Ostinato session

The above workflows access the Ostinato VM’s local filesystem. So you need to transfer files between the host and the VM.

Here are the detailed steps for transferring files.

One time setup

Stop the Ostinato instance, if it is running
Connect eth0/mgmt interface of Ostinato to Management/Cloud0 network
Start the Ostinato instance
Open a terminal within the Ostinato VNC instance and run ifconfig eth0 to make sure it gets an IP address via DHCP

Transfer files

Check which user you are logged in as in the VM by running whoami. If the user is ostinato, you are good to proceed to the transfer step, else if the user is tc, it doesn’t have a password assigned and ssh/scp don’t play nice with blank passwords. Therefore -

Assign a password to user tc by using the passwd command inside the terminal in the VNC.

Now you can ssh/scp to the DHCP assigned eth0 IP address using the username tc and the password that you assigned. If you are ostinato, use ostinato as the password as well.

To transfer from host to VM, run on the host -

scp <file> <user>@<ostinato-eth0-ip>:~

You can use the above steps to scp the pcap file to the Ostinato appliance and open it from the Ostinato GUI.

To transfer from VM to host, run on the host -

scp <user>@<ostinato-eth0-ip>:<full-path-to-file> <destination-path-on-host>

You can use this to save the streams/session from the Ostinato GUI and scp it to the host filesystem.

Persistence

Note that the Ostinato VM is designed to run out of ramfs, so any changes you make inside the VM (such as setting a password or creating files) won’t survive an appliance reboot.

There is however one exception.

Any files in /home/ostinato/ (or /home/tc) can be made persistent by doing a clean shutdown of the appliance from inside the VM -

Select Exit from the menu at the bottom of the window
Select Backup in the Backup options in the TC Exit Options window and click OK

If you do a clean exit as shown above, files in your home directory will be saved and restored across a reboot. Stopping the Ostinato node from the EVE-NG lab is not a clean exit and therefore your files won’t be pesistent.

For more Ostinato related content, subscribe for email updates.

Test SDWAN using application traffic

me@srivatsp.com (Srivats P) — Tue, 10 Aug 2021 00:00:00 +0000

Did you know that you could generate application traffic using Ostinato to test your SDWAN and application aware routing deployments?

In this post, I’m going to show you how.

Note that the application traffic generated by Ostinato will be stateless, synthetic traffic for the purpose of allowing your SDWAN edge nodes to detect the application and trigger the right policy — so you can verify your policies.

Ostinato is not stateful, does not support TCP connections and cannot generate fake traffic to websites to increase ad impressions etc.

Let’s generate Facebook traffic -

Create a new stream
Change packet size to 1500
Select protocols - Mac | Untagged | Ethernet | IPv4 | TCP | Text | Pattern
Go to Protocol Data tab, click on Internet Protocol ver 4 (IPv4) section and set source and destination IP addresses
Go to Transmission Control Protocol (TCP) section, override the source port and enter any value between 1024 and 65535 (this value MUST be unique for each application flow)
Go to the Text Protocol section, change line-ending to CRLF (as required by the HTTP standard) and enter GET / HTTP/1.1<Enter>Host: facebook.com<Enter><Enter><Enter>. Note the multiple enters at the end to insert blank lines - those are REQUIRED - HTTP requests end with two CRLF (0D 0A 0D 0A) - you can verify this in the Packet view tab
Go to stream control and set the rates etc.

To verify if the traffic is generated correctly, I used ntopng which uses nDPI - an open-source deep packet inspection library

Want to generate YouTube traffic instead? Just change Host: facebook.com to Host: youtube.com or any other application as required in the Text Protocol configuration. Remember if you are generating traffic for multiple applications, you MUST give each application a different TCP source port value.

Not working for you? Double-check the following -

Make sure source/destination mac are resolved; otherwise set mode to Fixed and set them manually
Make sure the HTTP text ends with at least two CRLF (0D 0A 0D 0A)
If sending traffic for multiple applications/protocols, make sure you change the source TCP port numbers so that each flow is unique

I’ve used ntopng/nDPI here for application detection. This procedure should work with other application detection libraries and devices as well. If it doesn’t, let me know and we can try to figure it out together.

Staircase style traffic with Ostinato

me@srivatsp.com (Srivats P) — Tue, 03 Aug 2021 00:00:00 +0000

A few weeks ago, Ostinato user Peter asked me -

I basically want to send traffic at different traffic % load for a specific interval of time. 10 seconds 10% 1G, 10 seconds 20% 1G, through 10 seconds 90% 1G and then back down to 10% and repeat. Is there a way to do that through the [Ostinato] GUI?

I sent him instructions to do so and figured it might be useful for others too, so posting here as well -

Note that although Peter didn’t need to generate 100% 1G traffic, it is easy to extend the below instructions to do that.

Let’s first visualize this staircase style traffic requirement -

Note that Ostinato stream configuration takes input as Number of Packets instead of Duration in seconds, and Packets/sec instead of % line rate, so we need to do some conversion first. Here are the formulae for the same -

PacketsPerSec = TargetBpsRate/((PktSize+20)*8)
NumberOfPackets = PacketsPerSec * DurationInSec

20 bytes is the per-packet ethernet line rate overhead (preamble, start of frame and inter packet gap)

I’ll assume for now that packet size doesn’t matter and go with 1500 byte packets (more on this later).

Using the above formulae, we calculate Packet Rate and Number of Packets for each of the 9 streams -

See the table on Google Sheets

Next steps -

Create stream #1 in the GUI and configure the protocols and fields as desired.
Configure packet size as 1500
Configure the Packet Rate and Number of Packets from the table above corresponding to stream #1
Duplicate the stream - enter count as 8 (for a total of 9 streams)
Edit the remaining 8 streams and change the packet rate and number of packets as per the above table
Ensure the first 8 streams are set to Goto Next and the last one to Goto First
Finally, hit “Apply” followed by “Start Transmit”

The reason I picked 1500 as the packet size as opposed to 64 (default packet size for Ostinato streams) is that Ostinato cannot do 1Gbps for 64 byte packets without the Turbo add-on.

To get line rate for small packet sizes like 64 bytes, you can use the Ostinato Turbo add-on which provides wire speed 10/25/40Gbps

If you have any questions, post in the comments below.

A deeper understanding of the OSI 7-layer model

me@srivatsp.com (Srivats P) — Tue, 08 Dec 2020 00:00:00 +0000

In the previous post, I talked to Graham Cassells - a Cisco Networking Academy teacher in Australia, who is using Ostinato in his classes. I also talked to his students at Lake Tuggeranong College about their course, the tools and the topics that interest them.

This is a bright set of kids! They’re interested in IT, Networking, cyber-security, and they’re keen to understand how hacking and penetration testing works.

The students generally use Packet tracer, Wireshark and Ostinato on both Windows 10 and Kali Linux for network related studies. For security practice, they use a cyber range - a Lego city with hackable trains and planes.

Cisco networking academy students at LTC

One of their recent study exercises was to run a normal ping, capture and study the packet in Wireshark, then use Ostinato to replicate and spoof the same packet to trick or fool a victim. They loved this exercise.

Using Ostinato helped the students visualize encapsulation and the OSI model layers in practice for a deeper understanding of networks and packets. They weren’t just learning the theory any more, they were creating these packets and running it through the system to see how it fared.

Christopher from the class said, “Ostinato was great for understanding more on how a packet is structured and sent across a network”. Connor, another student, said the exercise provided a practical understanding of Mac and IP addresses and enhanced their classroom theory training. Since Ostinato works great with other tools like Wireshark - so they’d definitely recommend it to their friends and other students.

Several students found it fascinating that when used for pen-testing, Ostinato-created packets couldn’t be told apart from normal packets, if done right. The digital fingerprint of the Ostinato created packet was identical to the one they were spoofing.

I asked them about what they found lacking with Ostinato and students basically wanted to use it better - they asked for more tutorials and guides to fully utilitize all the capabilities of the tool (More tutorials, is something I hear very often and will try to provide!). There was also a specific feedback about the Ostinato stream configuration dialog not showing a scroll bar and so they didn’t realize there were more options they could select (Will fix that in the next version!).

Thank you students for talking to me!

Ostinato improves networking students' understanding and skill level: Graham Cassells

me@srivatsp.com (Srivats P) — Wed, 18 Nov 2020 00:00:00 +0000

Graham Cassells is a CCNA and teacher of networking and cyber security at high schools and TAFE (vocational training for adults) for the last 20 years. He currently teaches the Cisco Networking Academy program at Canberra’s Lake Tuggeranong College in Australia.

He has been using Ostinato in his classes and was gracious enough to answer my questions via email. I asked him about the Cisco Networking Academy, how he uses Ostinato and his annoyances and wishlist for Ostinato.

Cisco Networking Academy program

What is the Cisco Networking Academy program?

The Cisco Networking Academy is an online program run around the world by Cisco for skills towards attaining the CCNA.

How is the program structured in terms of duration and coursework?

There are lots of courses under the academy. There are 3 networking units that comprise of around 10-15 modules - online study resources, short interactive quizzes and assessments, around 4 hours of study per week over 16 weeks. It takes around 2 years of part time study to complete the program.

How many students do you typically have in a batch?

We run classes of around 20 each year in our senior high school (college) years 11 & 12 for 16-18 year olds – for free. Our industry contacts are very envious of our young students who get the opportunity to study Cisco at school.

Which courses in the program do you teach?

I teach the Routing and Switching course - it is integrated in our ACT department of Education IT courses along with cyber security topics to enhance the practical side of the theory.

What networking technologies does the course include?

The OSI encapsulation model, IPv4 & IPv6 subnetting, setting up switches, routers, building simple networks, setting up security including firewalls such as ACLs, and using fault finding techniques.

What equipment, tools and software do you use?

We have Cisco 2960 switches and Cisco 1941 routers, wireless routers, PCs, Raspberry Pis and testing equipment. Our software includes Packet Tracer, Wireshark, Putty and Ostinato under various platforms such as Linux and Windows.

Graham’s networking class at LTC

Ostinato’s role in networking education

How did you discover Ostinato?

We looked for packet injection and creation tools and it was listed everywhere as a good tool to start with.

Were there any similar tools you also evaluated?

We looked at Hping and Scapy.

Why did you select Ostinato over other tools?

The time needed to learn the coding didn’t work well and Ostinato’s GUI and price suited the college well.

What do you use Ostinato for and how?

We set up a practical assessment task (the students work in pairs) - make a simple client to client network and Ping and Wireshark the result. Then, create a packet that spoofs the sender and compare the results to get a deeper understanding of encapsulation and digital forensic analysis.

The objective is understanding more than how packet data looks, it's about how it works at the levels in the OSI model.

As a teacher, what do you like most about Ostinato?

The quick and simple way to create injection packets using the GUI, and the way it links directly with Wireshark for analysis.

How has Ostinato impacted learning outcomes?

The students enjoy the application and it consolidates the abstract theories that need understanding in the application of networking technologies analysis and use.

I would definitely recommend Ostinato to all teachers who teach networking - it brings the theory out and allows the practical application that improves students’ understanding and skill level.

Ostinato feedback

What in Ostinato irritates you the most and you’d like fixed?

None really. If you follow the basic use videos on Youtube it is easy to set up and use. The most annoying thing is not knowing how to make packets for more services to test - it takes time, I guess.

What one thing, if added to Ostinato, would make it more useful for you?

More teaching resources to show more ways it can be used - maybe tutorials, Youtube vidoes of simulating applications.

What do your students think about Ostinato?

They have commented that they liked it, and the energy they showed in class was higher than normal as they worked to create a forensic packet comparison.

Anything else you would like to say?

Please keep developing it, and make some tutorial videos to show more complex use, e.g. how to make a simple web page payload, email, DHCP, DNS, FTP packets would be great and would cover a lot in terms of our lack of understanding in these technologies.

Next up - What Graham’s students think about Ostinato.

A bigger nudge to check the logs

me@srivatsp.com (Srivats P) — Tue, 02 Jun 2020 00:00:00 +0000

In Ostinato version 1.0, I added a Logs window to the UI so that errors/warnings could be notified to the user. Since most often the logs window is hidden behind a tab, I added a blinking icon to the tab whenever there was an error or warning.

But, there at the bottom there - that’s easy to miss.

With the upcoming Ostinato 1.1, you won’t -

It may seem like a small thing. But small things matter. They are not inconsequential. Not to someone who misses that tiny blinking icon and wastes time wondering why something is not working as expected.

This is one of the last bits that was planned for Ostinato v1.1 - I will now move on to doing pre-release testing followed by making the release and building the packages.

Stay tuned!

Test NAT using Ostinato

me@srivatsp.com (Srivats P) — Tue, 05 May 2020 00:00:00 +0000

Recently, Ostinato customer Javor reached out to me and wanted advice on how to configure traffic stream(s) to test 2 million NAT sessions.

A NAT session is based on the 5-tuple (SrcIP, DstIP, Protocol, SrcPort, DstPort). One NAT session is created per unique 5-tuple.

His first attempt was to configure the Src/Dst IP as random but he ran into a known Ostinato bug that causes incorrect TCP/UDP checksum. I suggested he workaround the checksum problem by overriding it to 0 (0 checksum value implies no checksum), but his test required that checksums not be 0. So he tried to configure incremental Src/Dst IP alongwith incremental Src/Dst Port and ran into a different problem. In his own words,

I find Ostinato for a great product.

But now I’m fighting with Ostinato to test CGNAT performance of one router and trying to create some stream that can generate the following packets with valid checksum mandatory.

So when I create a new stream with:
SRC IP - Increment - 10.0.0.0/16 - 1000 hosts
DST IP - Increment - 16 hosts
SRC Ports - incremental - very important for this test - 60000
DST Ports - incremental - 100

In fact the stream works but not in way that I expect.

The stream just generated sequentially flows one by one like this for example:
src-ip_1, src port 30001 -> dst-ip_1, port 1001
src-ip_2, src port 30002 -> dst-ip_2, port 1002
src-ip_3, src port 30003 -> dst-ip_3, port 1003

But I don’t want this. I would like to achieve the following way of order and incrementation: src_ip1, src port 30001 -> dst-ip_1, port 1001
src_ip1, src port 30002 -> dst-ip_1, port 1002
src_ip1, src port 30003 -> dst-ip_1, port 1003
src_ip1, src port 30004 -> dst-ip_1, port 1004
src_ip1, src port 30005 -> dst-ip_1, port 1005
……

So I would like the stream to generate the packets in order to start first from src-ip and incrementally passthrough over all src-ports range ( for example range of 5000 ports ) and then to go to the next src-ip then to next src-ip and so on….

Final goal is to utilize the router device with maximum unique sessions but not only unique per IP but each IP to generate more sessions. I’m trying to utilize ~2mln sessions.

As workaround I have tried to generate a lot of streams with fixed src/dst hosts with incremental range of src ports and it works but for 2mln sessions I need to generate a lot of streams ~ 240 streams in my case which is very very difficult and a lot of work - Copy/duplicate/change src-dst host and so on.

So I wonder if there is any easier way to achieve this.

Essentially the problem is that when you have multiple variable fields configured on a stream, each variable field will increment with every packet.

What Javor wants is a way to increment one variable field every X number of packets (where X > 1) while another variable field increments with every packet.

I told him that this was currently not possible and support may be added in a future release, but the problem, for some reason, refused to leave my head. A couple of days of background processing later, I had an insight. This is possible to do now albeit with some restrictions. Restrictions that I think Javor can live with.

Here’s how.

Let’s start with the objective of 2 million NAT sessions. If we iterate over the entire SrcPort range (0 – 65535), we need 2,000,000/65536 = ~ 30.5 SrcIP; let’s round it up to 32 SrcIP.

So, we create two variable fields - one for SrcPort and another for SrcIP.

The SrcPort variable field configuration is straight forward -

Increment SrcPort from 0 to 65535 step 1

(If you are using Ostinato v1.0 or earlier, you will not be able to enter count as 65536 due to a bug. Once you read this post in full, you will find a workaround for it)

The SrcIP is trickier. We’ll go slow. The most critical part is that the SrcIP should be incremented by 1 every 65536 packets.

Variable field mask to the rescue!

Originally, the variable field mask was implemented to allow increments in bit fields smaller than a byte without affecting adjacent bit fields. But it can do more—we can use mask to achieve what we want in this case.

From the Ostinato user guide -

The computation used for varying fields is as follows -
new = (old AND ~mask) OR ((value op n*step) AND mask)
where

old is 8/16/32-bit value (depending on counter type) starting at offset bytes from start of protocol

op is based on the mode - increment/decrement/random

n varies from 0 to (count-1) incrementing by 1 every packet

new is the value that will replace old

In a IP packet, the Src IP is 4 bytes (offsets 12 to 15) and Dst IP is 4 bytes (offsets 16 to 19). Let’s define a custom field for IPv4 as follows -

Field Custom
Type Counter32
Offset 14
Mask 0xFFFF0000
Mode Increment
Step 1

Let’s not bother about Value and Count for now. We’ll come to those. For now, let’s look at this visually. Given an IP frame with SrcIP as s1.s2.s3.s4 and DstIP as d1.d2.d3.d4 -

Offset      12                  16                  20
         ---+----+----+----+----+----+----+----+----+---
IP Frame    | s1 | s2 | s3 | s4 | d1 | d2 | d3 | d4 |  
         ---+----+----+----+----+----+----+----+----+---
Mask                  | FF | FF | 00 | 00 |
         ---+----+----+----+----+----+----+----+----+---
N*Step                | 00 | 00 | 00 | 01 | <== increments every packet
         ---+----+----+----+----+----+----+----+----+---

Our mask covers the last 2 bytes of SrcIP and first 2 bytes of DstIP. But the mask makes sure that the DstIP is not modified. For every packet, the N*Step value will be incremented by 1. The first 65536 packets will have this value increment from 0 to 65535 (0xFFFF) i.e. only the last 2 bytes are changed across these many packets, but because the mask blocks off change in these 2 bytes, they are not reflected in the packet. For the 65537th packet, this value will become 0x10000 (65536) which means the LSB of the last byte of SrcIP will get incremented. Subsequent packets will again change only the lower 2 bytes (which has no impact because of the mask) while the upper two bytes have the constant value 0x0001. After another 65536 iterations, the upper 2 bytes will be incremented by 1 i.e. to 0x0002 and so on.

Effectively, we are ensuring that SrcIP gets incremented every 65536 iterations!

If you work with binary and hexadecimal numbers regularly, this should make sense. If not, re-read it a couple of times and it will— trust me!

The insightful reader would have noticed that this will work only with powers of 2 (e.g. 65536 = 2^16). Yes, that’s the limitation that I talked about at the start!

Let’s come to the remaining fields.

First count. We want 32 SrcIP across 65536 SrcPort, so the count should be 32*65536 = 2097152

Finally value. This is the starting value of SrcIP. But, only the last 2 bytes of it followed by first 2 bytes of DstIP (that we don’t want to modify). Let’s say you want the starting value of SrcIP to be 10.2.24.100 (0x0A021864), we take the last 2 bytes 0x1864 as the first 2 bytes of the value and the last 2 bytes as 0x0000 i.e. 0x18640000, convert it to decimal (409206784) and enter that in the value field -

That’s it, we can now test 2 million NAT sessions using a single Ostinato stream! And depending on the NAT table size of the DUT, you can even simulate a NAT table full condition!

Not so fast, though!

There’s another piece of coincidence in this example. The SrcPort variable field count 65536 is exactly equal to 2^16—the range of increments that we mask off by configuring the last 2 bytes (16 bits) of the mask as 0.

What if we want a different range to vary for SrcPort. It would still work as long as the count is a power of 2. e.g. let’s say we want to vary across 16384 (16K) ports instead of 65536 (64K). All we have to do is change SrcIpStep such that SrcIpStep*SrcPortCount = the masked off portion (2^16 = 64K). In other words, Src IP Step = 64K/SrcPortCount = 4 for this example. This is useful when some NAT systems limit the max NAT session per host to a smaller number.

While writing this blog post, I found a bug that the count field in the Variable Fields GUI does not allow to enter 65536 for a 16 bit field such as the UDP source port. This bug is present in all versions uptil 1.0. If you are using one of these versions, instead of 32 SrcIp * 65536 Src Ports, you can use 64 SrcIp * 32768 Src ports to achieve 2 million NAT sessions—it’s still a single stream.

Qt model view and cut copy paste

me@srivatsp.com (Srivats P) — Tue, 07 Apr 2020 00:00:00 +0000

I’ve spent the last few weeks implementing cut-copy-paste in Ostinato.

While Qt (the cross-platform library that Ostinato uses for the UI) has great documentation, how to implement cut-copy-paste was not so obvious. Especially with Qt Model-Views which is used extensively by Ostinato. I hope this post will help you, if you are looking to do something similar.

The first thing to know is that the implementation of drag and drop and clipboard actions (cut, copy, paste) are very similar. You should first read the Qt documentation for Drag and Drop and Using Drag and Drop with Item Views before proceeding further. Also read about QClipboard.

Go ahead, I’ll wait.

Need some convincing? The reason why that documentation is important to read is because it describes all the concepts we need and also mentions clipboard actions briefly without going into details. The clipboard action implementation details are what we will get into here.

Model support for cut, copy, paste

This should be fairly clear to you by now (you did read the drag and drop documentation linked above didn’t you?). Here’s a short recap.

Note: Most of Ostinato’s views (that support copy-paste) are QTableView, so that’s what we’ll talk about here. The concepts apply to QListView and QTreeView also but we won’t discuss those.

The QAbstractItemModel methods relevant for copy are -

QStringList mimeTypes() const;
QMimeData* mimeData(const QModelIndexList &indexes) const;

To paste the data into the model, the QAbstractItemModel methods used are -

bool canDropMimeData(const QMimeData *data, Qt::DropAction action,
        int row, int column, const QModelIndex &parent) const;
bool dropMimeData(const QMimeData *data, Qt::DropAction action,
        int row, int column, const QModelIndex &parent);

Since cut is just a copy plus delete, to support the latter the model needs to implement -

bool removeRows(int row, int count, const QModelIndex &parent = QModelIndex())

Ostinato uses views with a selection behaviour of QAbstractItemView::SelectRows, so a delete means deleting a row, rather than clearing the contents of a single cell.

Models which don’t implement removeRows() won’t support a cut.

The QAbstractItemModel default implementation for the above methods uses a application specific mime-type of application/x-qabstractitemmodeldatalist which uses QDataStream to encode row, column and data for each selected cell. This mime data can be copied to the clipboard and pasted back using dropMimeData() which decodes it back into row, column and data to update the model.

This should work like a charm as long as you export/import all the data from your model via data() and setData()

In case of Ostinato, the Stream/DeviceGroup objects have a large number of fields in a complex hierarchy that cannot be displayed in a simple table, so the model only exposes some of them.

This meant that I had to re-implement the above functions in my models with my own custom mime-type and encoding/decoding. I will not get into the details of that here as this is really application specific.

Create the cut, copy, paste actions

The Qt class for menu items (application menu or context menu) is QAction. We want a single set of these actions that can be used both in the application menu bar under the top level Edit menu and also in the right-click context menu for each top level window/widget in our application.

I created a singleton ClipboardHelper class, added the actions as private members and exposed them via a public actions() method -

class ClipboardHelper : public QObject
{
    Q_OBJECT
public:
    ClipboardHelper(QObject *parent=nullptr);

    QList<QAction*> actions();

private slots:
    void actionTriggered();

private:
    QAction *actionCut_{nullptr};
    QAction *actionCopy_{nullptr};
    QAction *actionPaste_{nullptr};
};

ClipboardHelper::ClipboardHelper(QObject *parent)
    : QObject(parent)
{
    actionCut_ = new QAction(tr("&Cut"), this);
    actionCut_->setObjectName(QStringLiteral("actionCut"));
    actionCut_->setIcon(QIcon(QString::fromUtf8(":/icons/cut.png")));

    actionCopy_ = new QAction(tr("Cop&y"), this);
    actionCopy_->setObjectName(QStringLiteral("actionCopy"));
    actionCopy_->setIcon(QIcon(QString::fromUtf8(":/icons/copy.png")));

    actionPaste_ = new QAction(tr("&Paste"), this);
    actionPaste_->setObjectName(QStringLiteral("actionPaste"));
    actionPaste_->setIcon(QIcon(QString::fromUtf8(":/icons/paste.png")));

    connect(actionCut_, SIGNAL(triggered()), SLOT(actionTriggered()));
    connect(actionCopy_, SIGNAL(triggered()), SLOT(actionTriggered()));
    connect(actionPaste_, SIGNAL(triggered()), SLOT(actionTriggered()));
}

QList<QAction*> ClipboardHelper::actions()
{
    QList<QAction*> actionList({actionCut_, actionCopy_, actionPaste_});
    return actionList;
}

Note that I have used a single slot for all 3 actions - this is because their implemenation is very similar, so we use QObject::sender() to identify which action triggered the slot. The key point of executing the action is that we find the widget that has the current focus, check if that widget has a slot corresponding to the action and if so, we invoke that slot. This is all done using QMetaObject magic -

void ClipboardHelper::actionTriggered()
{
    QWidget *focusWidget = qApp->focusWidget();

    if  (!focusWidget)
        return;

    // single slot to handle cut/copy/paste - find which action was triggered
    QString action = sender()->objectName()
                        .remove("action").append("()").toLower();
    if (focusWidget->metaObject()->indexOfSlot(qPrintable(action)) < 0) {
        // slot not found in focus widget corresponding to action
        return;
    }

    action.remove("()");
    QMetaObject::invokeMethod(focusWidget, qPrintable(action),
            Qt::DirectConnection);
}

We now add these actions to the main window menubar’s Edit menu -

ClipboardHelper  *clipboardHelper;

MainWindow::MainWindow(QWidget *parent) 
    : QMainWindow (parent)
{
    clipboardHelper = new ClipboardHelper(this);
    ... 
    menuEdit->addActions(clipboardHelper->actions());
    ...
}

We also add these actions to the context menu of each window/widget as required -

streamList->addActions(clipboardHelper->actions());
deviceGroupList->addActions(clipboardHelper->actions());

We now have our actions available in the UI, which when clicked trigger the ClipboardHelper::actionTriggered() slot which in turn invokes the corresponding slot in the item view widget which will call into the model to execute the action.

What is our item view class? In Ostinato’s case it is QTableView. But if you look at the Qt documentation for this class or it’s ancestor QAbstractItemView class, you’ll see that they have no cut(), copy(), paste() slots!

We will create a new subclass by inheriting QTableView and add these slots. All the windows/widgets that used QTableView will now use XTableView instead -

class XTableView : public QTableView
{
    Q_OBJECT
public:
    XTableView(QWidget *parent) : QTableView(parent) {}
    virtual ~XTableView() {}

public slots:
    void cut()
    void copy()
    void paste()
}

To implement the copy slot, we call the model’s mimeData() with the current selection. But before we do that we sort the selection because views allow arbitrary selection using Ctrl/Shift modifiers. The data returned from the model is handed over to the global clipboard object -

void XTableView::copy()
{
    QModelIndexList selected = selectionModel()->selectedIndexes();
    if (selected.isEmpty())
        return;
    std::sort(selected.begin(), selected.end());

    QMimeData *mimeData = model()->mimeData(selected);
    qApp->clipboard()->setMimeData(mimeData);
}

To paste, we retrieve the item from the clipboard, verify if the model can accept the mime type by calling canDropMimeData() before calling dropMimeData().

void XTableView::paste()
{
    const QMimeData *mimeData = qApp->clipboard()->mimeData();

    if (!mimeData || mimeData->formats().isEmpty())
        return;

    // If no selection, insert at the end
    int row, column;
    if (selectionModel()->hasSelection()) {
        row = selectionModel()->selection().first().top();
        column = selectionModel()->selection().first().left();
    } else {
        row = model()->rowCount();
        column = model()->columnCount();
    }

    if (model()->canDropMimeData(mimeData, Qt::CopyAction,
                row, column, QModelIndex()))
        model()->dropMimeData(mimeData, Qt::CopyAction,
                row, column, QModelIndex());
}

Cut is just copy followed by removeRows -

void XTableView::cut()
{
    copy();
    foreach(QItemSelectionRange range, selectionModel()->selection())
        model()->removeRows(range.top(), range.height());
}

That puts in place the missing piece in the chain from the UI trigger to the model methods being called, completing our cut-copy-paste implementation.

Err, maybe not.

Enabling/Disabling the actions

There is still the little matter of enabling/disabling the menu actions based on certain conditions.

Each action should be enabled only when -

Cut: focus widget has a selection, a cut slot and model allows delete
Copy: focus widget has a selection and a copy slot
Paste: focus widget has a paste slot and can accept the clipboard item

Since these actions belong to ClipboardHelper, we extend that class to track when the focus widget changes by connecting it to a slot -

connect(qApp, SIGNAL(focusChanged(QWidget*, QWidget*)),
        SLOT(updateCutCopyStatus(QWidget*, QWidget*)));

In the slot, we can check if the focus widget has a current selection and cut/copy slot and enable/disable the actions accordingly. We also need to take care of the case where the selection state changes without the focus widget being changed. We do that by tracking the selection change of the focus widget in another slot -

void ClipboardHelper::updateCutCopyStatus(QWidget *old, QWidget *now)
{
    const XTableView *view = dynamic_cast<XTableView*>(old);
    if (view) {
        disconnect(view->selectionModel(),
                   SIGNAL(selectionChanged(const QItemSelection&,
                                           const QItemSelection&)),
                   this,
                   SLOT(focusWidgetSelectionChanged(const QItemSelection&,
                                                    const QItemSelection&)));
    }

    if  (!now) {
        // No focus widget to copy from
        actionCut_->setEnabled(false);
        actionCopy_->setEnabled(false);
        return;
    }

    const QMetaObject *meta = now->metaObject();
    if (meta->indexOfSlot("copy()") < 0) {
        // Focus Widget doesn't have a copy slot
        actionCut_->setEnabled(false);
        actionCopy_->setEnabled(false);
        return;
    }

    view = dynamic_cast<XTableView*>(now);
    if (view) {
        connect(view->selectionModel(),
                SIGNAL(selectionChanged(const QItemSelection&,
                                        const QItemSelection&)),
                SLOT(focusWidgetSelectionChanged(const QItemSelection&,
                                                 const QItemSelection&)));
        if (!view->hasSelection()) {
            // view doesn't have anything selected to copy
            actionCut_->setEnabled(false);
            actionCopy_->setEnabled(false);
            return;
        }
        actionCut_->setEnabled(view->canCut());
    }

    // focus widget has a selection and copy slot: copy possible
    actionCopy_->setEnabled(true);
}

void ClipboardHelper::focusWidgetSelectionChanged(
        const QItemSelection &selected, const QItemSelection &/*deselected*/)
{
    // Selection changed in the XTableView that has focus
    const XTableView *view = dynamic_cast<XTableView*>(qApp->focusWidget());
    actionCopy_->setEnabled(!selected.indexes().isEmpty());
    actionCut_->setEnabled(!selected.indexes().isEmpty()
                                && view && view->canCut());
}

Updating the paste action status is much more straight forward -

void ClipboardHelper::updatePasteStatus()
{
    QWidget *focusWidget = qApp->focusWidget();
    if  (!focusWidget) {
        // No focus widget to paste into
        actionPaste_->setEnabled(false);
        return;
    }

    const QMimeData *item = QGuiApplication::clipboard()->mimeData();
    if  (!item || item->formats().isEmpty()) {
        // Nothing on clipboard to paste
        actionPaste_->setEnabled(false);
        return;
    }

    const QMetaObject *meta = focusWidget->metaObject();
    if (meta->indexOfSlot("paste()") < 0) {
        // Focus Widget doesn't have a paste slot
        actionPaste_->setEnabled(false);
        return;
    }

    const XTableView *view = dynamic_cast<XTableView*>(focusWidget);
    if (view && !view->canPaste(item)) {
        // Focus widget view cannot accept this item
        actionPaste_->setEnabled(false);
        return;
    }

    // Focus widget can accept this item: paste possible",
    actionPaste_->setEnabled(true);
}

While implementing the above, we added an expectation that XTableView will provide a few more methods like canCut() etc. which we now add and implement -

bool XTableView::hasSelection() const
{
    return !selectionModel()->selectedIndexes().isEmpty();
}

bool XTableView::canCut() const
{
    // This is a heuristic
    return (model && model->supportedDropActions() != Qt::IgnoreAction)
}

bool XTableView::canPaste(const QMimeData *data) const
{
    return model()->canDropMimeData(data, Qt::CopyAction,
            0, 0, QModelIndex());
}

XTableView::canCut() should ideally check if removeRows() is implemented by the model. I couldn’t find a clean portable way to check that. So, instead I implemented supportedDropActions() to return Qt::IgnoreAction in all the models that are read-only and don’t implement removeRows(); the default QAbstractItemModel::supportedDropActions() returns Qt::CopyAction.

The advantage of creating XTableView with cut/copy/paste slots is that several native Qt widgets like QLineEdit, QTextEdit etc. also have similar named slots. So our code should work with those widgets too. That was the plan. Practically, they don’t have hasSelection/canCut/canPaste methods so they won’t work because the actions won’t be enabled.

All is not lost, though! What the native widgets do support is the cut/copy/paste actions if the short cut key combinations (Ctrl/Cmd-x/c/v) are received by the widget.

Shortcut keys

If you look at the code where we created the actions in the ClipboardHelper constructor, you will notice that I did not associate any keyboard shortcut with the actions. That’s deliberate.

Normally, when you press any key or key combination, the current focus widget receives a key press event. Most native Qt widgets like line edits and text edits etc. will do a cut, copy, paste in response to that event without any additional code required. However, if you associate keyboard shortcuts for menu actions, the keypress events are captured by the menu and are not sent to the widget.

In other words by not associating shortcut keys to the menu actions, we allow them to be received by the focus widget which in turn does the right thing. So, even if the menu items for cut/copy/paste may be disabled, the functionality will still work if shortcut keys are used!

However, QTableView doesn’t process those key combinations. Which means the shortcut keys don’t work with model-view windows. We fix that by implementing XTableView::keyPressEvent() -

virtual void XTableView::keyPressEvent(QKeyEvent *event)
{
    if (event->matches(QKeySequence::Cut)) {
        cut();
    } else if (event->matches(QKeySequence::Copy)) {
        copy();
    } else if (event->matches(QKeySequence::Paste)) {
        paste();
    } else
        QTableView::keyPressEvent(event);
}

I hope you found this useful! Let me know in the comments below.

Cut Copy Paste with Ostinato - video

me@srivatsp.com (Srivats P) — Wed, 25 Mar 2020 00:00:00 +0000

In the last few weeks, I added cut, copy, paste support in Ostinato. It’s available for streams, device groups, statistics and logs.

Menu actions are available in the Application Edit Menu and in the context menu for some of the windows. Irrespective of whether the menu item is available or not, you can always hit shortcut keys for your platform (e.g. ctrl-c, ctrl-v for Windows and cmd-c, cmd-v for MacOs).

The feature will be part of the upcoming Ostinato 1.1.

What’s your number one pain point with Ostinato that you’d like me to address? Let me know in the comments below.

Be More

me@srivatsp.com (Srivats P) — Tue, 14 Jan 2020 00:00:00 +0000

Based off a conversation with a friend -

Ostinato Device Emulation - Video tutorial

me@srivatsp.com (Srivats P) — Thu, 17 Oct 2019 00:00:00 +0000

There have been indications (and some direct feedback) that there isn’t enough documentation about Ostinato and it’s various features. While there is the Ostinato User Guide, I believe people are looking for a tutorial or HOW TO kind of documentation. So here goes …

When a question about Ostinato device groups was posted recently on the forum, I decided that it was time to make a tutorial about the feature. And as I tend to do, I went all ambitious and perfectionist about it and decided to do a video tutorial. I knew video would be a lot of work and time consuming, but it turned out to be even more than I thought. Although I’m still inclined to make tweaks to the video, it is time to curb that instinct and hit publish!

So, here’s the result of the last 10 days of work on this tutorial. I hope it’s helpful.

Video Tutorial

(Watch full-screen for a better view)

Text Recap

Although the video goes into much more details (and I highly recommend you watch it), here’s a quick recap -

For the test topology on the left, we want to test if the Network under test (NUT) is routing traffic correctly between two subnets.

The two edge routers PE1 and PE2 are connected to two ports on the Ostinato traffic generator.

On Ostinato, select a port connected to one of the edge routers and follow these steps -

Create a device group specifying
- Device count
- IP stack - IPv4/IPv6/Dual/None
- IP address (range)
- Gateway IP
Create a stream
- Ensure Src/Dst mac are set to “Resolve”
- Populate Src/Dst IP - src IP must match one of the device IP

Repeat the above steps on the second Ostinato port.

Ostinato will automatically do a ARP and fill in the source and destination mac address appropriately.

You can also look at the Ostinato user-guide page for Device Emulation.

If you have any questions or feedback, hit the comments below. Also, let me know if you’d like to see more Ostinato tutorials and on which topics.

Brown Boy (Blaft)

me@srivatsp.com (Srivats P) — Tue, 17 Sep 2019 00:00:00 +0000

My friend P S Nissim self-published a Kindle short - Brown Boy back in 2017.

Earlier this year it was picked up and republished by Blaft - an indie publishing house based in Chennai.

Working with Nissim and the publisher, I redesigned the cover art (click the image to enlarge)

Cover Design: Srivats P

Here’s the old Brown Boy cover for comparison.

Crafting VxLAN packets using Ostinato

me@srivatsp.com (Srivats P) — Thu, 29 Aug 2019 00:00:00 +0000

This post was originally published in Sep 2011 at the Inficone, but that site is now off the Internet; so reposting it here.

The buzzwords these days are “virtualization” and “cloud”. At the recently concluded VMworld 2011, VMware and Cisco announced the VxLAN technology which was developed to address scalability problems in today’s data centers.

Virtualized data centers today using traditional VLANs have scaling problems due to the 12-bit VLAN space which translates to a maximum of 4094 usable VLANs. Another requirement for multi-tenant networks is to isolate tenant traffic from one another. A VLAN could be used to provide this isolation on a L2 network but is again limited by the the 12-bit space. A L3 solution to this problem would force tenants to always use IP which would exclude tenants using non IP or other L2 protocols for inter-VM communication. Another problem is the mac address explosion at the Top of Rack switch which now needs to learn mac addresses per VM rather than per physical server.

VxLAN (short for Virtual eXtensible Local Area Network) attempts to solve these problems by creating virtual overlay L2 networks for inter-VM communication on top of the physical L2/L3 networks. It does this by doing a MAC-in-IP tunnelling and introducing a 24-bit Virtual Network Identifier (VNI) space.

For the full technical details on what problems VxLAN is attempting to solve and how, see the IETF draft (Update: Now RFC 7348).

Let’s look at the proposed frame format -

Outer Ethernet Header (including an optional VLAN tag)
Outer IP Header
Outer UDP Header
VxLAN Header
Inner Ethernet Header (including an optional VLAN tag)
Original Ethernet Payload (excluding the original Ethernet FCS)
FCS for Outer Ethernet Frame

Ostinato supports all the above protocols except the new VxLAN Header. We can simulate the VxLAN header very easily using Ostinato’s Userscript protocol.

Before we write the VxLAN userscript let’s look at the VxLAN frame format

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|R|R|R|R|I|R|R|R|            Reserved                           |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                VXLAN Network Identifier (VNI) |   Reserved    |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

It’s a simple 8 byte header with 2 fields - Flags (8-bit) and VNI (24-bit) while the remaining fields are all reserved and MUST to be set to 0. The flag field has only one defined bit - I which indicates that the header contains a valid VNI tag while the remaining bits MUST be set to 0.

Based on this information, here’s the VxLAN userscript -

protocol.name = "VxLAN"
protocol.protocolFrameSize = function() {
  return 8;
}
protocol.protocolFrameValue = function(index) {
    var flags  = 0x08; // set I bit, reset all R bits
    var vni  = 0x749127; // example VNI value

    var pfv = new Array(8);

    pfv[0] = flags;
    pfv[1] = 0;
    pfv[2] = 0;
    pfv[3] = 0;

    pfv[4] = (vni >> 16) & 0xFF;
    pfv[5] = (vni >> 8) & 0xFF;
    pfv[6] = vni & 0xFF;
    pfv[7] = 0;

    return pfv;
}
protocol.protocolId = function() {
    return 4789; // IANA assigned
}

We first define the protocol’s name and its size. We then define the contents of the protocol as an array which we populate with the flag field set to 0x08 (I bit set, R bits reset) and an example VNI value of 0x749127. Since VxLAN is a payload for UDP, we define the port number that UDP should use in the protocolId() function. That’s all that is required to define the VxLAN protocol.

To configure the full tunnelled frame format in Ostinato, create a new stream, edit it - go to “Advanced Protocol Selection” and add the following protocols in the given order -

MAC
Eth II
IPv4
UDP
UserScript
MAC
Eth II
IPv4
DATA

Go to protocol data tab, select UDP and override the UDP checksum field to set it as 0 (recommended by the VxLAN draft).

Next goto Userscript, copy paste the above script and click on “Compile” - you should see the compilation result as “Success”. Configure any other protocol fields that you may want and then click OK.

Click on Apply and hit play/transmit to send your VxLAN Tunnelled frames.

The following snapshot shows how the final packet would look like including the just created VxLan.

To make things even more simpler, you can open this stream file containing a VxLAN stream in Ostinato.

You might find these useful as well -

Analyzing VxLAN packets using Wireshark

me@srivatsp.com (Srivats P) — Tue, 13 Aug 2019 00:00:00 +0000

This post was originally published in Nov 2011 at Love My Tool (rebranded as Network Data Pedia recently), but is no longer available there; so reposting it here. Since the original post, Wireshark has introduced a built-in VxLAN dissector - but this post is still instructive for how to create a Lua dissector

As virtualization and cloud computing platforms are getting more and more deployed, the L2 network that the VMs are deployed in is exploding in size and VM migration from one host to another becoming problematic. To solve this problem VMware and Cisco have introduced a new protocol - VxLAN.

To stay on topic, we will not concern ourselves about the protocol details but only about the protocol frame format. For more information on how the protocol works and what it does see Denton Gentry’s 3-part VxLAN article or go straight to the VxLAN IETF Draft

Being a new protocol, Wireshark doesn’t yet have a built-in dissector for the VxLAN protocol. Anticipating such scenarios, Wireshark exports a Lua interface for creating quick-and-dirty dissectors. We will use this interface to write a Wireshark Lua dissector for VxLAN.

The VxLAN frame is a Ethernet-in-UDP encapsulated frame i.e. Ethernet-IP-UDP-VxLAN-Ethernet-IP. If you open vxlan.pcap in Wireshark, you will see that Wireshark stops dissecting as soon as it encounters the VxLAN header and shows all remaining bytes after UDP as just data bytes. The reason why Wireshark stops there is it does not understand how to decode the VxLAN header. That is what we will fix with our Lua script

Before we look at the Lua code itself, let’s look at the VxLAN header format -

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|R|R|R|R|I|R|R|R|            Reserved                           |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                VXLAN Network Identifier (VNI) |   Reserved    |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

It’s a simple 8 byte header - a 8-bit flags field starting at offset 0 with only one bit currently defined while the remaining 7 are reserved, and a 24-bit VNI Tag starting at offset 4. All the remaining bytes in the header are reserved.

Without further ado, here’s the VxLAN Lua dissector code -

do
    local p_vxlan = Proto("vxlan","Virtual eXtended LAN");

    local f_flags = ProtoField.uint8("vxlan.flags", "Flags", base.HEX)
    local f_flag_i = ProtoField.bool("vxlan.flags.i", "I Flag", 8,
             {"Valid VNI Tag present", "Valid VNI Tag NOT present"}, 0x08)
    local f_rsvd1 = ProtoField.uint24("vxlan.rsvd1", "Reserved", base.HEX)
    local f_vni = ProtoField.uint24("vxlan.vni", "VNI", base.HEX)
    local f_rsvd2 = ProtoField.uint8("vxlan.rsvd2", "Reserved", base.HEX)

    p_vxlan.fields = {f_flags, f_flag_i, f_rsvd1, f_vni, f_rsvd2}

    function p_vxlan.dissector(buf, pinfo, root)

        local t = root:add(p_vxlan, buf(0,8))

        local f = t:add(f_flags, buf(0,1))
        f:add(f_flag_i, buf(0,1))

        t:add(f_rsvd1, buf(1,3))
        t:add(f_vni, buf(4,3))
        t:add(f_rsvd2, buf(7,1))

        t:append_text(", VNI: 0x" .. string.format("%x", buf(4, 3):uint()))

        local eth_dis = Dissector.get("eth")
        eth_dis:call(buf(8):tvb(), pinfo, root)
    end

    local udp_encap_table = DissectorTable.get("udp.port")
    udp_encap_table:add(9029, p_vxlan)
end

Let’s examine the code closely.

We start by putting the entire dissector in a do-end code block to restrict the scope of the local variables we use.

We call Proto() to create our protocol p_vxlan. Then we create one ProtoField for each field in the header. Notice that we use the appropriate ProtoField.<type> for each field based on the VxLAN header format above. Once all fields are defined, we set p_vxlan.fields to a table containing all the fields.

Next we define our p_vxlan dissector function. This function takes 3 input parameters - the packet buffer that we need to dissect, packet information containing meta data like length of the frame, captured time etc. and the root of the dissected tree to which we will add the dissected information.

The first thing that we add to the root of the tree is the p_vxlan protocol specifying that it starts at offset 0 and spans 8 bytes.

Next we add each field, in the order in which they are defined in the header, as subitems of our protocol. Note that we add the I Flag field as a subitem of the Flag field instead of adding it as a subitem of our protocol.

After adding all the fields, we append our Protocol Name with the most important field of the protocol - VNI so that the value is visible even without expanding the protocol tree.

Before we end the function, recall that VxLAN is an encapsulated packet format and there’s another packet starting with the Ethernet MAC addresses following the VxLAN header. So we call the ethernet dissector and pass it the buffer from offset 8 (after the VxLAN header).

The final step in our script is to register the p_vxlan protocol as a UDP encapsulated dissector. In this script, we are using 9029 as an example UDP port number since IANA hasn’t assigned an official port number yet.

That’s all there is to writing our VxLAN Lua dissector.

Now we need to make Wireshark use this dissector. Open Wireshark, go to Help|About|Folders and look for Personal Plugins and Global Plugins - create a file named vxlan.lua in either of those two locations and then restart Wireshark. Go to Help|About|Plugins and verify vxlan.lua is listed.

Now open vxlan.pcap in Wireshark to see vxlan.lua in action!

For a Lua primer and language reference see Programming in Lua . For Wireshark specific Lua information, see Lua support in Wireshark

To create your own VxLAN packets, see Crafting VxLAN packets using Ostinato.

Ostinato throughput on a 40G NIC

me@srivatsp.com (Srivats P) — Tue, 02 Apr 2019 00:00:00 +0000

Update (2021): Ostinato now supports 10G, 25G and 40Gbps line-rate rate traffic using the Turbo Transmit add-on.

A few weeks ago I attended the DPDK Summit in Bangalore, where M Jay from Intel loaned me a dual-port 40G interface to enable Ostinato to become a high-speed packet generator using DPDK. Thanks M Jay and Intel!

If you’ve been following Ostinato for a long time, you might recall I had done a prototype of Ostinato with DPDK back in 2014 which won the 6Wind DPDK Design contest. That work was done on a VM using VNICs instead of physical NICs and the lack of the latter (amongst other things) meant I never productized that code.

Before I (re)start the DPDK work on Ostinato, I wanted to get some baseline measurements with the existing libpcap based code. This blog post presents these results.

You can also look at the results for a 1G port for Linux, MacOS, Live ISO.

I configured the same single stream as in previous tests -

Protocols: Mac, Ethernet II, IPv4, UDP, Pattern
Source and Destination Mac addresses populated with the actual Mac addresses of the source and sink
Source and Destination IP addresses populated with the actual IP addresses of the source and sink
Packets/sec: 0
Number of packets: 10 (default)
After this stream: Goto first

One difference from the previous tests is that I’m using a single host with a dual-port NIC with the ports connected back to back. Ostinato packets sent on port 1 are received back on port 2 on the same host.

Also, I’m adding two jumbo sized packet sizes to this test - 4096 bytes and 9018 bytes.

Performance figures

Here are the numbers -

Packet Size	Send Rate (Kpps)	Recv Rate (Kpps)	Send Rate (Mbps)	Recv Rate (Mbps)
64	755	630	507	423
128	753	640	892	758
256	738	635	1,630	1,402
512	732	636	3,115	2,707
1024	725	642	6,055	5,362
1518	725	639	8,920	7,862
4096	574	566	18,901	18,637
9018	225	225	16,268	16,268

Mbps rate was calculated using the below formula to take into account the ethernet line overhead of 20 bytes (1 byte SFD + 7 byte preamble + 12 byte inter-packet gap) -

Rate (in Mbps) = (PacketSize + 20) * KppsRate * 8 / 1000

Specs

Software

Ostinato 0.9 - for Linux (Ubuntu 16.04, 64-bit)
Ubuntu 16.04.3 LTS

Hardware

Processor: Xeon E5-2620v4 @ 2.10GHz (8-core)
RAM: 32GB
NIC: Intel XL710 40Gbps dual-port PCI 3.0, x8

Observations

The first that jumped out was that there was a lot of fluctuation in the send rate while transmitting e.g. for 256 bytes packets, I saw rates ranging from 635Kpps to 728Kpps. This is a very wide range and I don’t have an explanation. I have used the lower end of the range seen in the above results for all packet sizes.
The second thing that jumps out from the results table is that receive rate gets capped at ~640Kpps even if the send rate is higher. Again, I don’t have an explanation for the same.
The next thing that is suprising is that the max rate is achieved for 4096 packets, instead of 9018.
Although it’s a 8-core CPU, only two core reach 100% CPU during the test - this is expected since for a single port, Ostinato doesn’t use more than one core and we have both transmit and receive port on the same host. The first core with 100% cpu utilization (presumably the one handling the tx port) has 15% user and 85% system usage - indicating that the major time is spent in the kernel driver. The second core had 100% utilization in softirq - presumably the one handling the receive port
RAM usage not measured
One heartening observation is that Ostinato (even without DPDK) can be used as a 10G traffic generator - if you can use jumbo frames.

For more Ostinato posts, subscribe for email updates.

How to change the default boot option in Ostinato virtual appliance

me@srivatsp.com (Srivats P) — Sun, 24 Mar 2019 00:00:00 +0000

The Ostinato virtual appliance has two boot options -

Boot Ostinato (Agent + Controller)
Boot Drone (Agent only)

Boot Ostinato is the default and therefore is selected and if you don’t change it within 30s, the boot proceeds with the same.

For some use cases you might want the default option to be Boot Drone e.g. for automation use cases using the Python API where your automation framework boots appliances or if you will always use the GUI controller remotely or if you just want the appliance to boot faster (Agent only boots a shade faster than Agent + Controller)

The Ostinato virtual appliance uses the EXTLINUX bootloader from the SYSLINUX project

To change the default option to Boot Drone instead, you need to change the DEFAULT and ONTIMEOUT statements in the extlinux config file.

Here are the detailed steps -

Boot the Ostinato Virtual Appliance
Select Boot Drone (Agent only) from the boot menu

After the appliance boots and you get a shell prompt (press Enter a couple of times if you don’t see the prompt), run the following commands -

cd /mnt/sda1/boot/extlinux
sudo sed -i "s/DEFAULT ostinato/DEFAULT drone/" extlinux.conf
sudo sed -i "s/ONTIMEOUT ostinato/ONTIMEOUT drone/" extlinux.conf
sudo reboot

After the reboot (and for all subsequent boots), Boot Drone (Agent only) will be set as the default boot option.

Note that after booting into agent only mode, if you decide you want to the start the Ostinato GUI controller, you don’t need to reboot, just run startx.

If you want to change other aspects of the boot menu (e.g. the timeout value), lookup the EXTLINUX boot menu documentation - Ostinato uses the Simple Menu System - and change extlinux.conf accordingly.

For more Ostinato posts, subscribe for email updates.

Hell's Kitchen

me@srivatsp.com (Srivats P) — Tue, 16 Oct 2018 00:00:00 +0000

No, this is not about the cookery reality TV show.

Back in 1996, for our graduate project I and Mudit wrote an arcade-style side-scroller game that we called Hell’s Kitchen

I recently discovered the files in a backup archive and had the urge to play it once again. Although, I had the .exe file - this game was built to run on MS DOS. And as we know those old DOS games don’t work on recent Windows anymore.

So, as any self-respecting nerd would do, I created a FreeDOS VM using VirtualBox and transferred the games from my Windows 10 host to the DOS VM using a temporary ISO.

I still wasn’t sure if it would work ‘coz the game used Mode X and a bunch of other low level hardware and VGA reprogramming stuff like color cycling, video memory page flipping, a new keyboard ISR etc.

It worked!

Sure, some things are not right - the color cycling is not right, the sound effects don’t work, the end credits are too fast, but the essential game play all works pretty well!

And some of the stuff is cringeworthy - especially the intro and end credits - but I view that as a sign of aesthetic growth.

A shout out to Sudarshan Purohit who did the game design and Justin Fernandes (RIP) who drew the sprites for the main character - Masala Das and the other assorted powerups and lives.

Another shout out to Michael Abrash who introduced and inspired the whole endeavour via his book Power Graphics Programming that I picked up on a whim at the Darya Ganj Sunday flea market.

Update (Jan 2023): Now you can play Hell’s Kitchen directly in the browser thanks to the magic of WebAssembly, FreeDOS and js-dos.

Play Hell’s Kitchen

If you are still reading this and want to play it locally, here’s a OVA file of the VM that you could download and import into your favourite hypervisor.

Download Hell’s Kitchen (5MB OVA file)

Ostinato API - Wait for stream transmission to finish

me@srivatsp.com (Srivats P) — Tue, 26 Jun 2018 00:00:00 +0000

The Ostinato API tutorial configures a single stream to send 5 packets then sleeps for 7 seconds - assuming transmission would have finished by then - before fetching the tx/rx stats for verification.

This is fine for a tutorial, but for production scenarios you will want to wait till the transmission is finished instead of waiting (sleeping) for a specified amount of time.

Here’s how you can do that -

# wait for transmit to finish
while True:
    try:
        time.sleep(5)
        tx_stats = drone.getStats(tx_port)
        if tx_stats.port_stats[0].state.is_transmit_on == False:
            break
    except KeyboardInterrupt:
        log.info('transmit interrupted by user')
        break

Interested in more Ostinato tips and tricks? Subscribe to receive email updates!

How to run Ostinato in a ssh-only environment

me@srivatsp.com (Srivats P) — Tue, 19 Jun 2018 00:00:00 +0000

Ostinato has a controller-agent architecture that uses a TCP connection (on port 7878 by default) for communication between them for command and control.

One popular deployment of this architecture is where the agent runs on a headless server and connects to the DUT via direct ethernet links. Both the DUT and server are co-located in a lab. The GUI controller on the other hand runs on users’ laptop or desktop which is on the corporate network and different from the lab network. For the agent and controller to be able to talk to each other they need to be able to talk TCP over port 7878 (or another user specified port) between these two networks.

Some environments may not allow any communication between these two networks except ssh (tcp port 22) which leads to a problem in running Ostinato.

To enable Ostinato agent-controller communication in this restricted environment, one can use ssh tunnelling.

Assuming you are running the agent (drone) on host bar and the Ostinato controller GUI on host foo, run the following command on foo -

foo$ ssh -L 7878:localhost:7878 bar

The above command will create a local server on foo listening on port 7878 and any TCP traffic to this port will be transparently tunnelled to bar:7878 over ssh. The tunnelled traffic will be encrypted as well which should be enough to bypass most firewalls.

Now on foo, when you open the Ostinato controller GUI, the 127.0.0.1 local portgroup will try to contact the local server running on foo:7878 which in turn will contact the agent running on bar:7878.

If you want to learn more about ssh tunnelling, SSH Tunnelling explained is a good read.

Interested in more Ostinato tips and tricks? Subscribe to receive email updates!

Mid Air (cover)

me@srivatsp.com (Srivats P) — Tue, 29 May 2018 00:00:00 +0000

A cover of Paul Buchanan’s Mid Air

Piano and vocals: Srivats P.

Recorded on phone.

How to generate MPLS packets using Ostinato User Script

me@srivatsp.com (Srivats P) — Wed, 25 Apr 2018 00:00:00 +0000

Ostinato provides two ways to generate packets for protocols that are not available out of the box.

In the last post, we saw how to create a MPLS packet in Ostinato using hexdump. In this post we will see how to do the same using a user script instead.

Although this post talks about MPLS, the same method is applicable for any protocol - MPLS is used just as an example here

First we start with the MPLS packet format. Let’s assume we need to generate a IP/UDP packet encapsulated inside a MPLS tunnel. So the packet would consist of the following protocol headers in this order -

+----------+------+----+-----+------+
| Ethernet | MPLS | IP | UDP | Data |
+----------+------+----+-----+------+

All the protocols in the above frame except MPLS are already supported in Ostinato.

To create this packet in Ostinato, we create a stream and select the already available protocols - Ethernet II, IPv4, UDP, Pattern and then switch to the advanced protocol selection mode -

Select the :{Script}[EXPERIMENTAL] protocol in the Available Protocols list on the left, IPv4 in the Selected Protocols list on the right and click on the right arrow to insert userscript in between Eth II and IPv4 -

You can use the up/down arrows to change the order of the selected protocols at any time

Now that we have the user script header at the correct place in the packet, let’s switch to the Protocol Data tab and click on UserScript -

We need to write a script here to generate the MPLS header. To do that, let’s review the MPLS header format.

The MPLS header consists of one or more MPLS tags comprising a tag stack -

+-------+-------+-------+-----+
| Tag 1 | Tag 2 | Tag 3 | ... |
+-------+-------+-------+-----+

Each tag is 32-bit wide and is composed of the following fields -

+------------+-----+-----+-----+
| MPLS Label | Exp |  S  | TTL |
|    (20)    | (3) | (1) | (8) |
+------------+-----+-----+-----+

The numbers in () represents the bit-width of each field. The S bit is set to 1 for the last (innermost) tag and is 0 for all the other tags

Let’s create a single tag MPLS header with the following values -

Label = 12701
EXP = 5
S = 1
TTL = 64

We start the script with defining the protocol name -

protocol.name = 'MPLS';

followed by the size of the header -

protocol.protocolFrameSize = function() { return 4; }

Note: protocolFrameSize must be a function returning an integer, not a simple integer

and finally a function that returns an array of integers where the size of the array is the number of bytes in the header and each element of the array is the byte at that offset (in other words, it’s a byte array!) -

protocol.protocolFrameValue = function()
{
    var fv = Array(4);
    // TODO: fill fv with header bytes

    return fv;
}

We first create an object with the values that we want in the MPLS fields and then use some bit arithmetic to convert it to a 32-bit integer value using the header format encoding that we saw earlier -

protocol.protocolFrameValue = function()
{
    var tag = { label: 12701, exp: 5, s: 1, ttl: 64 };
    var tagVal = tag.label << 12 | tag.exp << 9 | tag.s << 8 | tag.ttl;

    var fv = Array(4);
    // TODO: convert tagVal to header bytes and fill fv 

    return fv;
}

We complete the function by extracting each byte out of the tagVal integer and populating fv with it -

protocol.protocolFrameValue = function()
{
    var tag = { label: 12701, exp: 5, s: 1, ttl: 64 };
    var tagVal = tag.label << 12 | tag.exp << 9 | tag.s << 8 | tag.ttl;

    var fv = Array(4);
    fv[0] = (tagVal >> 24) & 0xff;
    fv[1] = (tagVal >> 16) & 0xff;
    fv[2] = (tagVal >> 8) & 0xff;
    fv[3] = tagVal & 0xff;

    return fv;
}

We need one last thing - tell our preceding protocol Eth II (of which we are a payload) what our ethertype is -

protocol.protocolId = function() { return 0x8847; }

Putting all that together, here’s the full script -

protocol.name = 'MPLS';
protocol.protocolFrameSize = function() { return 4; }
protocol.protocolFrameValue = function()
{
    var tag = { label: 12701, exp: 5, s: 1, ttl: 64 };
    var tagVal = tag.label << 12 | tag.exp << 9 | tag.s << 8 | tag.ttl;

    var fv = Array(4);
    fv[0] = (tagVal >> 24) & 0xff;
    fv[1] = (tagVal >> 16) & 0xff;
    fv[2] = (tagVal >> 8) & 0xff;
    fv[3] = tagVal & 0xff;

    return fv;
}
protocol.protocolId = function() { return 0x8847; }

Copy paste the above script into the user script editor, and hit Compile (make sure the compilation results in Success) -

To preview the result of our script, go to the Packet View tab and click on the MPLS header in the top pane - the MPLS header bytes will be highlighted in the bottom pane -

The generated 32-bit value is 0x0319DB40 which is the same value that we calculated manually in the previous post

To confirm that that the MPLS header is indeed correct, we apply our changes, transmit the stream, capture the transmitted packet and view it in Wireshark -

Voila, Wireshark decodes that perfectly!

What if you need more than one tag?

Simple -

Change protocolFrameSize to return n*4 where n is the number of tags that we want
Convert the single tag object to a list of tag objects
Size the fv array to n*4
Do the tagVal and fv calculation in a loop

Here’s an example for 3 tags -

protocol.name = 'MPLS';
protocol.protocolFrameSize = function() { return 3*4; }
protocol.protocolFrameValue = function()
{
    var tags = [
        { label: 12701, exp: 5, s: 0, ttl: 64 },
        { label: 972, exp: 1, s: 0, ttl: 128 },
        { label: 19, exp: 3, s: 1, ttl: 255 }
    ];

    var fv = Array(tags.length*4);
    for (i = 0; i < tags.length; i++) {
        var tag = tags[i];
	var tagVal = tag.label << 12 | tag.exp << 9 | tag.s << 8 | tag.ttl;

	fv[i*4+0] = (tagVal >> 24) & 0xff;
	fv[i*4+1] = (tagVal >> 16) & 0xff;
	fv[i*4+2] = (tagVal >> 8) & 0xff;
	fv[i*4+3] = tagVal & 0xff;
    }

    return fv;
}
protocol.protocolId = function() { return 0x8847; }

Here’s the 3-tag stack mpls frame in Wireshark -

Ostinato userscript has much more capabilities than we saw in this post. See the UserScript Reference Documentation for details.

Interested in more Ostinato tips and tricks? Subscribe to receive email updates!

Lichner - Lily

me@srivatsp.com (Srivats P) — Mon, 16 Apr 2018 00:00:00 +0000

A rendition of Heinrich Lichner’s Opus 160 No. 6 - Lily

Sorry, your browser doesn't support HTML5 audio

Composer: Heinrich Lichner
Piano: Srivats P.

Recorded on phone.

How to generate MPLS packets using Ostinato HexDump

me@srivatsp.com (Srivats P) — Wed, 11 Apr 2018 00:00:00 +0000

Ostinato includes and supports a lot of the commonly used protocols to generate packets. However, from time to time, one may come across a protocol that is not available in Ostinato.

Fear not, Ostinato provides a way to generate packets for protocols that are not available out of the box. Actually it provides two ways, not one.

In this post we will see how to create a MPLS packet by using the first of the two ways - hexdump.

Although this post talks about MPLS, the same method is applicable for any protocol, MPLS is used just as an example here

Let’s look at the MPLS frame format. Let’s assume we need to generate a IP/UDP packet encapsulated inside a MPLS tunnel. So the frame format would consist of the following protocol headers in this order -

+----------+------+----+-----+
| Ethernet | MPLS | IP | UDP |
+----------+------+----+-----+

All the protocols in the above frame except MPLS are already supported in Ostinato.

To create this packet in Ostinato, we create a stream and select the protocols - Ethernet II, IPv4, UDP, Hex Dump and then switch to the advanced protocol selection mode -

Select the HexDump protocol in the Selected Protocols list and use the up/down arrows to move it between Eth II and IPv4

Now that we have the HexDump header at the correct place in the packet, let’s switch to the Protocol Data tab and click on HexDump -

We need to enter the MPLS header bytes here in hex format. To calculate these, we need to know the MPLS header format.

The MPLS header consists of one or more MPLS tags comprising a tag stack -

+-------+-------+-------+-----+
| Tag 1 | Tag 2 | Tag 3 | ... |
+-------+-------+-------+-----+

Each tag is 32-bit wide and is composed of the following fields -

+------------+-----+-----+-----+
| MPLS Label | Exp |  S  | TTL |
|    (20)    | (3) | (1) | (8) |
+------------+-----+-----+-----+

The numbers in () represents the bit-width of each field. The S bit is set to 1 for the last (innermost) tag and is 0 for all the other tags

Let’s create a single tag MPLS header with the following values -

Label = 12701
EXP = 5
S = 1
TTL = 64

First, let’s convert each of those values to hexadecimal -

Label = 12701 (0x319D)
EXP = 5 (0x5)
S = 1 (0x1)
TTL = 64 (0x40)

Looking at the tag encoding above and some bit arithmetic, we can now calculate the 32-bit hex encoded tag for these values -

0x0319DB40

So, we enter these bytes in the hexdump editor, and uncheck the padding -

We need to do one last thing - go to the Ethernet II tab, tick the checkbox and enter the MPLS ethertype 8847 -

To verify if the MPLS header is correct, we apply our changes, transmit the stream, capture the transmitted packet and view it in Wireshark -

Voila, Wireshark decodes that perfectly!

What if you need more than one tag? Just calculate the 4-byte (32-bit) hex value for each tag (make sure S=0 in all tags except the last one) and enter the same in the hexdump.

Here’s an example -

Tag 1:
Label = 12701 (0x319D)
EXP = 5 (0x5)
S = 0 (0x0)
TTL = 64 (0x40)

Tag 2:
Label = 972 (0x3CC)
EXP = 1 (0x1)
S = 0 (0x0)
TTL = 128 (0x80)

Tag 3:
Label = 19 (0x13)
EXP = 3 (0x3)
S = 1 (0x1)
TTL = 255 (0xff)

Hex Encoded value -

0319DA40 003CC280 000137FF

Here’s the 3-tag stack mpls frame in Wireshark -

In the next post, we’ll see how to use a Ostinato userscript to do the same thing so that we don’t need to do all this manual hex calculation which is prone to errors.

Interested in more Ostinato tips and tricks? Subscribe to receive email updates!

12 ways to mindful living

me@srivatsp.com (Srivats P) — Tue, 13 Mar 2018 00:00:00 +0000

Download A4 size printable poster PDF

Source: Zen Habits
Design H/T: Paul Zii’s 29 ways to stay creative (Paul’s original tumblr is dead)

How to change Ostinato's look and style

me@srivatsp.com (Srivats P) — Tue, 06 Mar 2018 00:00:00 +0000

Ostinato uses Qt for cross-platform UI functionality and ships with a few styles built in that mimic the look and feel of various platforms. You can use these styles without having to change or recompile the code.

Use the command line param -style <style> when invoking ostinato. The available styles are -

motif
cde
windows
cleanlooks
gtk
mac
plastique
windowsxp
windowsvista

The styles in italics are fully supported only on their native OS due to their dependency on OS specific modules and libraries

Here’s a sample gallery of a few styles for Ostinato running on Windows 10 (click on the thumbnails for full image and description) -

Ostinato style samples (on Windows 10)

While it’s mostly just fun playing with the different looks, it might also be useful if you are using Ostinato on a OS that you normally don’t use.

For example, if you are a Windows user and the Ubuntu Unity GUI throws you off, you could get back your familiar look and feel for Ostinato (and any other Qt4 app for that matter) by using -style windows

Here’s a sample gallery of a few styles for Ostinato running on Ubuntu 16.04 (click on the thumbnails for full image and description) -

Ostinato style samples (on Ubuntu 16.04)

That purple CDE window brings back memories of my first job where we had Sun Solaris workstations running CDE!

Interested in more Ostinato tips and tricks? Subscribe to receive email updates!

Ostinato for Linux Maximum Transmit Rate

me@srivatsp.com (Srivats P) — Mon, 26 Feb 2018 00:00:00 +0000

Update (2021): Ostinato now supports 10G, 25G and 40Gbps line-rate rate traffic using the Turbo Transmit add-on.

I’ve been intermittently posting results for the maximum transmit rate performance test with Ostinato 0.9. See results for Live ISO, MacOS.

Now we do the same test on Linux - specifically Ubuntu 16.04.3 LTS. I expected the results to be same as the Live ISO since it is also Linux based, but I was in for a surprise!.

I configured the same single stream as earlier -

Protocols: Mac, Ethernet II, IPv4, UDP, Pattern
Source and Destination Mac addresses populated with the actual Mac addresses of the source and sink
Source and Destination IP addresses populated with the actual IP addresses of the source and sink
Packets/sec: 0
Number of packets: 10 (default)
After this stream: Goto first

Performance figures

Here are the numbers -

Packet Size	Max Rate (Kpps)	Max Rate (Mbps)
64	920	618
128	807	955
256	442	976
512	231	983
1024	118	986
1518	80	984

Mbps rate was calculated using the below formula to take into account the ethernet line overhead of 20 bytes (1 byte SFD + 7 byte preamble + 12 byte inter-packet gap) -

Rate (in Mbps) = (PacketSize + 20) * kppsRate * 8 / 1000

Specs

Software

Ostinato 0.9 - for Linux (Ubuntu 16.04, 64-bit)
Ubuntu 16.04.3 LTS

Hardware

Processor: Xeon E5-2620v4 @ 2.10GHz (8-core)
RAM: 32GB
NIC: Intel I218-V 1Gbps

Observations

Although it’s a 8-core CPU, only one core reaches 100% CPU during the test (same as Live ISO) - as expected since for a single port, Ostinato doesn’t use more than one core
RAM usage - 26MB (value of RES from top); this is lower than the 98MB used in the LiveISO
Max line rate (well - quite close to it) is reached for 256 byte packets and larger, same as the Live ISO and MacOS.
For smaller packet sizes, Ubuntu performs better (64 - 618Mbps, 128 - 955Mbps) than the Linux ISO (64 - 403Mbps, 128 - 702Mbps) - in fact, the 128 byte rate is quite close to line rate
Speculating on what could cause this performance difference on the two different Linux distros -
- The application code is exactly the same for both Ubuntu and LiveISO
- In the packet transmit code path, there are no Qt APIs
- The non application packet transmit code path is essentially composed of pcap_sendpacket() and the underlying system call(s) - confirmed by perf
- Version comparison of these components -

Component	Ubuntu 16.04.3	Live ISO (TinyCore 8.2)
Kernel	4.13.0-32-generic (64-bit)	4.8.17-tinycore (32-bit)
glibc	2.23	2.50.3
libpcap	1.7.4-2	1.4.0

For more Ostinato posts, subscribe for email updates.

10 reasons why contributing to open source is better than an internship

me@srivatsp.com (Srivats P) — Wed, 07 Feb 2018 00:00:00 +0000

College students often look for internships. Here in India, these are typically for 5-6 months (one semester - required for the degree) or 2 months (if done during the summer)

While some organisations have limited slots available for the former - a lot of students still struggle to get one. The latter is actually quite difficult to find - mostly because there’s not much someone can get done in 2 months without costing the organisation a lot (in $ and time).

I believe it is better for students to instead use the time to contribute to an open-source project, than go hunting for internship opportunities.

All the skills that you will learn while working on an open-source project are directly transferrable to an organisation and will make you effective and productive much faster than your peers. And you know what they say about first impressions - right? So why not jump-start your career by working on and contributing to an open-source project.

Here’s how it will help -

1. Choice of project

At most organisations, you won’t be given an option to choose the project. With open-source, you have a HUGE variety of projects to choose from. So why not choose a project/technology/language that you are interested in? It could even be a tool or software that you already use!

2. Version Control

Version control in a professional setup - an organisation or an open source project - is critical and I doubt there’s any organisation that doesn’t have it in place. For open-source projects, it is the core part of the collaborative process. But college students typically don’t use or require it if all they do is short, throw-away programming assignments. Thankfully, this is beginning to change with the advent and popularity of GitHub and students are using this while in college itself.

3. Navigate a large code base

Sir, where can I find main() in the code?

A fresher once asked this question to a colleague of mine. He wanted to read the code starting from main(). The question was honest and well-intended. But, as we know, it’s not possible to do this with most code bases that you get to work on in organisations because of the sheer size of the code base.

You have to necessarily use tools like ctags, cscope, opengrok or IDEs to navigate around.

A lot of open source projects’ code will also be reasonably big in size (at least compared to the short programs students tend to write in college), so it’s an opportunity to learn these tools and skills that will be relevant when you start your career.

4. Learn to find and isolate areas of code

Related to the previous, to troubleshoot and debug the first step is always to localize the area of code that you need to look at. And this is a skill that comes with practice - how to quickly identify the potential area of code to look at from amongst the ocean of code.

A must skill in an open-source project or in an organisation.

5. Read code ‘coz there’s no documentation

Most organisations hopefully will have some documentation - spec, design etc. However, even if they do, code changes fast and documents almost never keep up. So you have to read the code as the single source of truth. How does a feature work? Read the code and figure out!

Guess what? Open-source projects are no different - they have even less documentation than organisations. Get all the practice you can!

6. Community Interaction

When you try and contribute to open source, you may get feedback from the community. While most projects will try and be nice to you (a lot of projects even have a code of conduct that requires this), you may still face brickbats or curt replies. That’s practice to develop a hard shell and not take it personally.

A valuable skill in an organisation and in life.

7. Write better - emails and docs

While progressive organisations have probably moved on to Slack et.al., email is still prevalent and relevant. You need to learn how to write emails that communicate effectively but succintly. How to write a good subject line. Etiquettes regarding to: and cc:. Format the body of the email correctly e.g. using mono-width font for code/log snippets. End with a call to action - indicating if something is expected from the recipients.

Most interaction on open-source projects happens over the mailing list or in bug reports. So you get to practice your writing skills.

As any writer will tell you, the only way to be a better writer is to write more. And that’s true for emails too! And as you will discover - being a good communicator is sometimes even more important than your programming chops.

8. Showcase your work publicly

When an internship ends what you have usually is a short section in your resume describing the project. But access to any docs, code etc. cannot be shared because it is proprietary. So any future recruiter has only that short blurb in the Resume to look at and make a judgement on.

With open-source projects, all your work is in the open for anyone to view allowing them to make better and more informed judgements on your actual work. So if you did a good job and are proud of the work, showcase your work by including a link to the project and your contribution.

In the US including your GitHub username in your Resume is almost required these days especially for entry level positions.

9. Gratitude from the project maintainer

Open-soure project maintainers are a harried lot, buried under ever-piling mountains of bug reports and feature requests.

They will be thankful for any contributions that you make. Just be mindful of the fact that you may fix a bug or develop a feature for the project, but in all likelyhood they helped you by answering your (multiple) queries, reviewing your code etc. - all of them things that take non-trivial amounts of time - the one commodity that maintainers have in limited supply.

The most important thing you can do to justify their time investment in you is #10

10. Continue to work on project

Once the internship is over, you no longer have access to the code or organisation. Not so with open-source projects. You can continue to work on the same project - new bugs or new features and become a regular contributor to the project.

Being a regular contributor is not just a big boost to your credibility (and Resume), but it also means you get into the habit of Ship early, ship often - a trait that organisations love (but hinder with their process overhead)

Convinced?

Good!

Where do I start, you ask?

Start with finding your first open source project or bug to work on or open-source friday

Ostinato for MacOS Maximum Transmit Rate

me@srivatsp.com (Srivats P) — Thu, 01 Feb 2018 00:00:00 +0000

Update (2021): Ostinato now supports 10G, 25G and 40Gbps line-rate rate traffic using the Turbo Transmit add-on.

In my last post, I presented results for a maximum transmit rate performance test on the Ostinato 0.9 Live ISO. I intended to repeat that for all platforms, but life got in the way.

I got some time today, so I did a test on MacOS with the same Ostinato 0.9 version as the Live ISO.

I configured a single stream, again same as earlier -

Protocols: Mac, Ethernet II, IPv4, UDP, Pattern
Source and Destination Mac addresses populated with the actual Mac addresses of the source and sink
Source and Destination IP addresses populated with the actual IP addresses of the source and sink
Packets/sec: 0
Number of packets: 10 (default)
After this stream: Goto first

Performance figures

Here are the numbers -

Packet Size	Max Rate (Kpps)	Max Rate (Mbps)
64	755	507
128	695	823
256	448	989
512	233	992
1024	119	994
1518	81	997

Mbps rate was calculated using the below formula to take into account the ethernet line overhead of 20 bytes (1 byte SFD + 7 byte preamble + 12 byte inter-packet gap) -

Rate (in Mbps) = (PacketSize + 20) * kppsRate * 8 / 1000

Specs

Software

Ostinato 0.9 - for MacOS
macOS Sierra Version 10.12.6

Hardware

MacBook Pro
Processor: Intel Core i7 @ 2.5GHz (4-core), Hyperthreading on
RAM: 16GB
Thunderbold to Gigabit Ethernet Adaptor

Observations

Max line rate (well - quite close to it) is reached for 256 byte packets and larger, same as the Live ISO
For smaller packet sizes, the MacOS version performs better (64 - 507Mbps, 128 - 823Mbps) than the Linux ISO (64 - 403Mbps, 128 - 702Mbps)
Although the underlying hardware and OS is different, I think the primary reason for the better performance is because of the OS - with Linux, only one core reaches 100% CPU during the test, but with MacOS (which is based on BSD) two processes show usage >= 100% - drone and kernel_task.
Looking at per core CPU usage/history in activity monitor, 4 (out of 8 vCPUs) show usage between 50 to 60%
I can’t seem to figure out how to see per thread CPU usage or the associated core for a thread to dissect futher - (if you know how to get this info, please share in the comments below)

Ostinato Live ISO Maximum Transmit Rate

me@srivatsp.com (Srivats P) — Fri, 29 Dec 2017 00:00:00 +0000

Update (2021): Ostinato now supports 10G, 25G and 40Gbps line-rate rate traffic using the Turbo Transmit add-on.

As part of the Ostinato 0.9 release, I also packaged Ostinato as a Virtual appliance and a Live ISO. I wanted to check the maximum packet generation rate performance of these form factors, so I used Rufus to burn the Live ISO to a USB pendrive and booted off the USB to collect the numbers.

I configured a single stream as given below -

Protocols: Mac, Ethernet II, IPv4, UDP, Pattern
Source and Destination Mac addresses populated with the actual Mac addresses of the source and sink
Source and Destination IP addresses populated with the actual IP addresses of the source and sink
Packets/sec: 0
Number of packets: 10 (default)
After this stream: Goto first

Performance figures

Here are the numbers -

Packet Size	Max Rate (Kpps)	Max Rate (Mbps)
64	600	403
128	593	702
256	442	976
512	231	983
1024	118	986
1518	80	984

Mbps rate was calculated using the below formula to take into account the ethernet line overhead of 20 bytes (1 byte SFD + 7 byte preamble + 12 byte inter-packet gap) -

Rate (in Mbps) = (PacketSize + 20) * kppsRate * 8 / 1000

Specs

Software

Ostinato 0.9 Live ISO

Hardware

Processor: Xeon E5-2620v4 @ 2.10GHz (8-core)
RAM: 32GB
NIC: Intel I218-V 1Gbps

Observations

Although it’s a 8-core CPU, only one core reaches 100% CPU during the test - this is expected since for a single port, Ostinato doesn’t use more than one core
RAM usage - 98MB
Max traffic rate for 64 byte packets is only 403Mbps
Max line rate (well - quite close to it) is reached for 256 byte packets and larger, but it never reaches 1000 Mbps even for larger packet size (i.e. lower Kpps rate) - I wonder why. It can’t be related to Ostinato itself, but something system related (if you have any thoughts on this, please share in the comments below)

Señor Santa Claus (cover)

me@srivatsp.com (Srivats P) — Mon, 25 Dec 2017 00:00:00 +0000

A cover of Jim Reeves’ Señor Santa Claus

Merry Christmas and Happy Holidays!

Sorry, your browser doesn't support HTML5 audio

Piano and vocals: Srivats P.

Recorded on phone.

Brown Boy

me@srivatsp.com (Srivats P) — Wed, 08 Nov 2017 00:00:00 +0000

A good friend of mine recently published a Kindle short - Brown Boy.

Hitesh lost his son in these Himalayan mountains years ago. They tried to tell him it was a wild animal that took the child on a stormy night. But Hitesh remembers the scratching sound outside his window that night. He remembers the door that yawned open by itself. Most of all, he remembers the story of the mysterious creature of legend said to haunt these dark jungles. The one everyone is afraid to talk about …

I designed the cover for the book based on a photograph by Jody Davis

Cover Photo: Jody Davis
Cover Design: Srivats P

Hey There Delilah (cover)

me@srivatsp.com (Srivats P) — Sat, 14 Oct 2017 00:00:00 +0000

A cover of Plain White T’s Hey There Delilah

Sorry, your browser doesn't support HTML5 audio

Piano and vocals: Srivats P.

Recorded on phone.

Ostinato first run UX improvements

me@srivatsp.com (Srivats P) — Sun, 01 Oct 2017 00:00:00 +0000

About a month ago, Ostinato user cube_030 provided the following suggestions as part of the Ostinato quick feedback survey

UI needs to be cleaned up some things should be automatic or more clear. One example is after loading a stream it needs to be applied. This was not clear at first.
Also some buttons are available even if they do nothing. i.e. when no interface is selected you can still hit play.
However this product has helped a lot and made testing a lab work much easier.

Around the same time I subscribed to Patrick McKenzie (patio11) and the first email/video in the series talked about improving the first run experience for users.

I’ve always known that the Ostinato UI is not intuitive for first timers - especially those who haven’t used commercial traffic generators. The split architecture and the notion of streams etc. are unfamiliar to them. But for the longest time I’ve been reluctant to simplify the interface hoping instead that a quickstart will be able to bridge the gap.

Watching Patrick’s video (available only to subscribers) convinced me that this was a missed opportunity. They will most likely open the UI, not know what to do, click a few things here and there, things won’t work and they will close the app in frustration regretting their decision to try it out.

So I decided to do something about it. Patrick recommends the first run of the app to start with a virtual tour of the interface and how to perform common tasks. I couldn’t figure out how to do this in Ostinato which is a Qt widgets based desktop app (modal dialogs with customized placement maybe?) - I decided not to spend too much time figuring this out, but see if there was some other way I could achieve the same objective.

I started with the first screen that users see when they run the app and put myself in the shoes of a new user -

A new user’s reaction to that is likely to be -

What am I supposed to do here? How do I generate a packet?

So, I wrote some helpful text utilizing the empty top right area. This area is used only when you select a port from the port list. However, when the app opens, nothing is selected in the port list, so it remains blank. A good space for basic instructions -

That’s better.

The challenge with writing instructional text is - you want it to be succint, not just because space is constrained and visually you don’t want it to be crowded, but because longer the text, less likely the user will read it. Writing the copy for this (and for subsequent textual hints) took me several iterations, tweaks and was quite time consuming, but I told myself - good copy takes time.

What would be even better was if the port group auto expanded to show all the ports instead of the user having to double-click the port group to reveal it (I remember wanting to do this when I first wrote this code but that didn’t happen for some reason - maybe ‘coz I was new to Qt and/or I had other important stuff to implement, so I probably didn’t spend much time on it). Anyway, I figured it out this time -

One problem I ran into was that once a port group or a port is selected in that list causing this welcome screen to be replaced, there’s no way to see it again. There will always be a current/selected item in that list. A bit of research and I found that this is a QTreeView bug/behaviour - so I wrote XTreeView to fix that.

But how would a user know that clicking on a blank area in the port list would bring back the welcome screen? Well, I don’t have an answer for that - I can just hope that if he clicks around, he will discover it :-)

Another thing I wanted to tackle here was a common problem that users run into when they first run Ostinato - the port group is empty. I have a detailed entry in the FAQ on why this happens and how to fix this - but a first time user doesn’t know that!

So, I added some text to point to that.

The instructions on this welcome page asks the user to select the port and hopefully that’s what he would do, but what if selects a port group?

We can do better than that. Let’s reinforce the instructions by repeating them -

Now, let’s select a port.

We run into a problem here. We no longer have a blank space but a widget here, so how do we write some instructions? I tried adding instructional text at the bottom of the stream list widget, but didn’t like that it took up valuable screen estate and reduced the size of the stream list widget. I tried adding the text next to the Apply button, which solved the screen estate problem, but wasn’t entirely satisfied with the results -

What if we could reuse the blank area of the stream list widget itself? We need the instructions only to add the first stream and hopefully user would remember those to add more streams.

So I wrote XTableView which displays some helpful text if the view has no items to be displayed. I utilized the area next to the Apply button to hint at when clicking Apply was required.

So the user right-clicks and selects New Stream from the context menu to create his first stream. Great!

What’s the next step? He needs to edit the just created stream because add just creates a default stream, not with the attributes that the user wants.

So what does the user have to do to edit the stream? Double click on the gear icon in the stream or right click the stream to access the context menu. Not very intuitive from the above screen. The user has had a glimpse of the context menu when adding a new stream, so he might have seen that it included an edit stream action also. Not something we can rely on.

Hold on. Do we really need to have a 2 step new+edit action? What if we collapse it into a single action? i.e. new stream would directly open the stream configuration dialog (used to edit the stream)?

A good idea, but we need to also handle these cases -

Include stream name and stream enable/disable in the dialog - the only way to edit these stream fields currently was directly from the stream list, not from the dialog
Adding multiple new streams requires the ability for the dialog to support editing and traversing multiple streams including the ability to identify which stream he is editing
Adding one or more streams and then pressing Cancel in the dialog should not incorrectly mark the port as dirty (more on this later)

Making these changes in the code required some refactoring and rework, but at last it was done -

Next step? Click Apply to push changes to the agent. A frequent stumbling block for new users who don’t realize that without doing that no packets will go out.

Ideally, we should auto apply any changes, but ‘coz applying changes is a sequence of synchronous RPCs, they sometimes take time and the UI is disabled for that duration. Which is a irritant if you have to create/edit multiple streams. So I had taken the original design decision to not do a auto apply and leave it to the user to decide when to apply.

There was also a matter of being able to detect all the changes made by the user to mark the port as dirty - not a straight forward task.

As with any hard task I had a lot of resistance to tackle this. But somehow this time I managed to overcome that and got down to the task of detecting all the changes that make a port dirty. In the end, it wasn’t as difficult as I thought it would be (note to self: everything is not as difficult as it appears to be).

Once I was able to detect when a port was dirty and needed to be applied, I replaced the always on generic text I had added next to the Apply button to be intelligent. The text would appear only when a change made the port dirty. I also changed the color of port in the port list to red to signal to the user that something requires his attention.

To show that these are related, I used a bit of red in the apply text to match the port name.

Ok, user hits apply. Next step?

He needs to go over to the Port Statistics window for the next task - stream transmit. So once Apply is clicked, we change the Apply text to direct him there -

Trigerring stream transmit in the port statistics window is a multi-step process -

Select the port(s)
Click on the Start Transmit toolbar button

As cube_030 pointed out in his suggestions, ‘coz the toolbar buttons are always enabled, it is not clear that user first needs to select a port.

Also, because the buttons have only icons, no text, you don’t know which is the transmit button (they do have tooltips which appear when you hover over the button, but that’s not intuitive).

I tried adding text to all the buttons but they took up too much screen estate, so I added category labels instead.

For some reason that I don’t recall now, to select a port in this window, I require that the user select entire port column by clicking on the column header - selecting a few cells of the port wouldn’t suffice. The user guide even has a helpful image to warn about this. But a first time user doesn’t know that.

So, I changed the code to automatically select the entire column when you click anywhere in that column, not just the header. I also changed the code so that buttons are enabled only if one or more ports are selected -

The user can now click on the Transmit play button to start transmitting packets.

Here’s an animated walkthrough the new UX -

To be consistent, I also added similar hints as the stream list to the device list as well, although more needs to be done there. I’ve left that for a future release.

With these UX improvements, users will hopefully have a better first (and subsequent) run experience.

For good measure, I added/improved some error messages for frequent stumbling blocks that users face - that’s the next post though!

All these UX improvements will be part of 0.9 release which I’m currently working towards.

No qDebug output with Qt 4.8.7 and MinGW

me@srivatsp.com (Srivats P) — Sat, 02 Sep 2017 00:00:00 +0000

When setting up my Qt based development environment for Ostinato, on my new Win 10 based desktop, I ran into this strange problem where the qDebug output wouldn’t be available/visible on DbgView. After several frustrating days of messing around with various debuggers and alternatives to DbgView, it appears that the problem was in Qt itself.

TL;DR Qt installation had setup win32-g++-4.6 as my QMAKESPEC but C:\Qt\4.8.7\mkspecs\features\win32\windows.prf does not add QT_NEEDS_QMAIN for this makespec, but does so for win32-g++. Adding the below line to windows.prf and regenerating the Makefiles using qmake fixed the problem -

win32-g++-4.6:DEFINES += QT_NEEDS_QMAIN

Now back to the longer story of how I found the problem.

I knew that Qt uses the Win32 API OutputDebugString() for qDebug output for GUI applications. Looking at the code for qt_message_output() (the eventual function called by qDebug), I found it strange that the API was used only for WIN_CE. For WIN32 it uses fprintf()

    if (handler) {
        (*handler)(msgType, buf);
    } else {
#if defined(Q_CC_MWERKS) && defined(Q_OS_MACX)
        mac_default_handler(buf);
#elif defined(QT_USE_SLOG2)
        slog2_default_handler(msgType, buf);
#elif defined(Q_OS_WINCE)
        QString fstr = QString::fromLatin1(buf);
        fstr += QLatin1Char('\n');
        OutputDebugString(reinterpret_cast<const wchar_t> (fstr.utf16()));
#elif defined(Q_OS_SYMBIAN)
        // RDebug::Print has a cap of 256 characters so break it up
        char format[] = "[Qt Message] %S";
        const int maxBlockSize = 256 - sizeof(format);
        const TPtrC8 ptr(reinterpret_cast<const tuint8>(buf));
        for (int i = 0; i  (fstr.utf16()));
    }

As an experiment, I installed a custom message handler -

void msgOut(QtMsgType /*type*/, const char* msg)
{
    QString fstr = QString::fromLatin1(msg);
    fstr += QLatin1Char('\n');
    OutputDebugString(reinterpret_cast<const wchar_t> (fstr.utf16()));
}

int main(int argc, char* argv[])
{
    ...
    qInstallMsgHandler(msgOut);
    ...
}

That worked. But I still believed that it should work out of the box without having to install a custom handler. When looking at the qInstallMessageHandler() code, I noticed this interesting check -

#if defined(Q_OS_WIN) && defined(QT_BUILD_CORE_LIB)
    if (!handler && usingWinMain)
        handler = qWinMsgHandler;
#endif

Hmmm, so when you uninstall a custom handler, instead of falling back to using qt_message_output(), Qt installed a special handler qWinMsgHandler() instead.

So, as an experiment, I installed qWinMsgHandler instead of my custom handler -

qInstallMsgHandler(qWinMsgHandler);

That worked. Which meant, if I cleared the handler instead of setting one, it should install qWinMsgHandler internally -

qInstallMsgHandler(0);

No dice. Huh? What? Why?

Tracing that function using gdb revealed that the usingWinMain variable was false, so qWinMsgHandler doesn’t get installed.

So where is usingWinMain set to true?

In qWinMain (corelib/kernel/qcoreapplication_win.cpp).

Now, that’s qWinMain, not WinMain - the usual entry point for Windows GUI applications. A helpful comment on top of that function pointed WinMain() to be in qtmain_win.cpp

And it’s there. The WinMain function and a call to qWinMain.

So, why isn’t usingWinMain set to true?

I asked gdb where is WinMain in the exe -

(gdb) info func WinMain
All functions matching regular expression "WinMain":

File kernel\qcoreapplication_win.cpp:
void qWinMain(HINSTANCE__*, HINSTANCE__*, char*, int, int&, QVector<char>&);

Non-debugging symbols:
0x004014c0  WinMainCRTStartup
(gdb)

That’s strange - there is no WinMain?!

Let’s look at the backtrace in gdb -

(gdb) set backtrace past-main on
(gdb) bt
#0  main (argc=1, argv=0x1f3e50) at main.cpp:64
#1  0x004013de in __tmainCRTStartup ()
#2  0x76fe8744 in KERNEL32!BaseThreadInitThunk ()
   from C:\WINDOWS\SysWOW64\kernel32.dll
#3  0x7743582d in ntdll!RtlGetAppContainerNamedObjectPath ()
   from C:\WINDOWS\SYSTEM32\ntdll.dll
#4  0x774357fd in ntdll!RtlGetAppContainerNamedObjectPath ()
   from C:\WINDOWS\SYSTEM32\ntdll.dll
#5  0x00000000 in ?? ()
(gdb)

No WinMain again. Googling around and reading some helpful stuff about main, wmain and WinMain and WinMain provided by Qt, it appears that For MinGW, WinMain() will never be called if main() exists (if you find “official” documentation on this, please send me a pointer)!

Qt works around this problem by doing this piece of preprocessing trickery in gui\kernel\qwindowdefs.h -

#if defined(QT_NEEDS_QMAIN)
#define main qMain
#endif

When QT_NEEDS_QMAIN is defined, the user provided main function is renamed to qMain forcing MinGW to use WinMain as the entry point instead of main.

Does the qmake generated Makefile include QT_NEEDS_QMAIN?

DEFINES       = -DUNICODE -DQT_DLL -DQT_SCRIPT_LIB -DQT_XML_LIB -DQT_GUI_LIB -DQT_NETWORK_LIB -DQT_CORE_LIB -DQT_HAVE_MMX -DQT_HAVE_3DNOW -DQT_HAVE_SSE -DQT_HAVE_MMXEXT -DQT_HAVE_SSE2 -DQT_THREAD_SUPPORT

No.

Did the older (Qt 4.3.3) qmake generated Makefile include QT_NEEDS_QMAIN?

DEFINES       = -DUNICODE -DQT_LARGEFILE_SUPPORT -DQT_DLL -DQT_SCRIPT_LIB -DQT_XML_LIB -DQT_GUI_LIB -DQT_NETWORK_LIB -DQT_CORE_LIB -DQT_THREAD_SUPPORT -DQT_NEEDS_QMAIN

Yes.

A quick experiment to add QT_NEEDS_QMAIN manually to the Makefile and re-building confirmed that it fixes the problem.

Buy hey wait, we’re not done yet. I’m not manually going to add this everytime I create a new git clone or do a make distclean. Why the hell isn’t qmake adding it to the generated Makefile?

Now reading through C++ code to find out what is happening is one thing. Reading through qmake’s C++ code to parse a .pro file and generate a Makefile ain’t so straightforward that looking at the code will directly tell you anything without spending significant time trying to understand that code -

So, change tactics and use grep inside Qt sources to find all instances of QT_NEEDS_QMAIN

mkspecs/features/win32/windows.prf:    win32-g++:DEFINES += QT_NEEDS_QMAIN
mkspecs/features/win32/windows.prf:    win32-borland:DEFINES += QT_NEEDS_QMAIN

Looking at windows.prf, it appears that if QMAKESPEC is win32-g++ or win32-borland, qmake will add QT_NEEDS_QMAIN to the generated Makefile. What Qt installer setup the makespec for me was win32-g++-4.6.

Adding the below line and regenerating the Makefiles using qmake did indeed add QT_NEEDS_QMAIN to the generated Makefile -

win32-g++-4.6:DEFINES += QT_NEEDS_QMAIN

So that missing line is the cause of our problem.

Or is it?

Now Qt makespec names are essentially a combination of platform and compiler separated by a -. Assuming 4.6 was the compiler version, I wasn’t sure if the scope win32-g++ should evaluate to true for win32-g++-4.6 also - after all win32-g++-4.6 is a specialization of win32-g++ as evidenced by the former’s qmake.conf #including the latter.

Running qmake -d and scanning the output for logs related to windows.prf processing confirmed that it does not - the win32-g++ scope match fails when QMAKESPEC is win32-g++-4.6

I’m not sure if that scope match failure is intended and correct or a qmake bug. I spent some time trying to read through qmake source code to answer that question but gave up after about an hour - it was beyond 2:00 AM and I had already spent all evening and all night on this investigation.

Aside: Although Ostinato is my personal project unrelated to my day job, I write several of these kind of “investigation reports” at the day job when debugging problems. I do it primarily to document the analysis for future use by myself or my colleagues. Reading through such a report offers the impression that the investigation was a straightforward logical line where one step leads to another till we find root cause, but this is never ever the case. There are so many twists, turns, tangents and threads of investigation that you follow during the investigation that lead nowhere and are wasted effort. But this is never reflected in the report.

As a matter of fact, I have long felt that “debugging” or “triaging” a technical problem is very similar to “crime” investigation - well, at least as depicted in books, tv shows and movies! That’s another post though!

Join Runner's High you do ...

me@srivatsp.com (Srivats P) — Tue, 24 Jan 2017 00:00:00 +0000

Hello World!

me@srivatsp.com (Srivats P) — Wed, 04 Jan 2017 00:00:00 +0000

There have been times when I’ve wanted to write something but didn’t ‘coz I didn’t have a “place” (aka blog) to write at. I registered this vanity domain almost an year ago with this express purpose but never managed to get it up and running for various reasons - topic for another post maybe!

This whole writing business is essentially an experiment on my part. I don’t have a fixed format or theme or topic or anything decided at all. It’s just a place for when the muse strikes. I do expect to write about Ostinato since that is such a big part of my life, but hopefully there’ll be more. And in the spirit of the experiment that it is, the writing will likely be all over the place as I embark on this ride.

Hello World!