FlowTest

FlowTest is a complex testbed for testing NetFlow/IPFIX monitoring probes. It includes a traffic generator and player that is capable of simulating traffic of realistic 1/10/100 Gbps networks, a library with various evaluation models to compare the output of the probe under test with expected flow records, and an orchestration tool that provides test automation and controls the entire environment.

Key features:

• Tools for extracting network traffic profiles of real (high-speed) networks.
• Generator of realistic network traffic (in the form of a PCAP file) based on network traffic profiles.
• High-speed PCAP player with a support for various outputs and replay speeds of 1/10/100 Gbps and more.
• Python library for comparing network flows (i.e. expected vs. received IPFIX/NetFlow records).
• Orchestration tool for automated execution of various testing scenarios and handling network devices which participate in the test.
• Tests verifying the output correctness and performance of the monitoring probes in real operation.
• Automatic and easy test environment setup using Ansible playbooks.
• Dataset of manually annotated PCAPs of various network protocols with network flow descriptions.
• Dataset of 1/10/100 Gbps network profiles from which realistic traffic can be generated.

❤️ If you use the tools or datasets resulting from this project, be sure to let us know. These references help us to get support for further development and new amazing projects. Use, for example, FlowTest project discussion forum.

The project, datasets and tools were developed in cooperation between CESNET and Progress Flowmon with a grant from the Ministry of the Interior of the Czech Republic (see Acknowledgement section).

Tools

The project provides a set of various tools. They can typically be used individually (e.g., to generate realistic network traffic) or as a whole to create a testbed for testing network monitoring probes. The steps to build or install them are shown in the section below.

• ft-profiler (Python) - Create network profiles from existing NetFlow/IPFIX records.
• ft-profile-sampler (C++) - Reduce size of high-speed network profiles without skewing profile characteristics.
• ft-profile-trimmer (Python) - Trim network profiles of excessive records.
• ft-generator (C++) - Generate complex network traffic based on network profiles.
• ft-replay (C++) - Replay PCAPs at very high speed via various outputs (AF_PACKET, AF_XDP, etc.)
• ft-analyzer (Python) - A library for comparing flow records.
• ft-orchestration (Python) - Orchestration tool for automated test execution.

Testbed

A malfunctioning NetFlow/IPFIX probe can, for example, generate partial or even incorrect flow records that can be misleading when analyzing security events. As a result, such probes can also cause false reports or, even worse, mask critical security threats.

• Do you know if your NetFlow/IPFIX probe is correctly extracting important information from network traffic?
• Are you sure that the probe does not generate unexpected flow records?
• What are its performance limits and when does it start to drop traffic and create incomplete records?

By using tools in this project, it is possible to create a comprehensive test environment for testing NetFlow/IPFIX probes in various realistic environments. This makes it easy to verify that the probe you are using to monitor your network correctly reports network status and handles your network traffic. In addition, automated tests can be used to repeatedly verify functionality across different probe configurations or after software updates.

💡 If you want to test your probe or want to replicate our test environment in your lab, contact us and let's discuss the possibilities of cooperation.

Setup and installation

At the moment, RHEL 8/9 and derived distributions (e.g. Oracle Linux or Rocky Linux) are supported. On other distributions, compiling and running tools should be also possible, but installation of dependencies may vary.

You can prepare a complete development environment including all dependencies or, for example, just an environment for running Python tools. In the case of automated environment preparation, we use Ansible playbooks that are prepared only for RHEL 8/9 and its derived distributions (e.g. Oracle Linux or Rocky Linux).

See the following options:

<details> <summary>🔧 <b>Automated preparation of the complete development environment.</b></summary>

# Install all C++ dependencies and prepare Python environment.
# Default Ansible playbook "dev_machine.yaml" will run on your machine.
# If the prompt "BECOME password:" appears, enter the root password.
sudo dnf -y install epel-release
sudo dnf -y install ansible
./prepare_machine.sh

# If you want to build and optionally install C++ tools (e.g. ft-generator
# or ft-replay), run following commands:
mkdir build && cd build
cmake -DCMAKE_INSTALL_PREFIX=/usr ..
make -j8
sudo make install

# If you want to run Python tools, active Python environment (pipenv) and
# follow the instructions in the README of the individual tools.
# Note: If you compiled C++ tools in the previous step, you must return to
# the root of the project directory first.
. venv/bin/activate

Note: For RHEL 8 (and derived distributions), the system Python will be upgraded to version 3.9, as older versions are no longer supported.

</details> <details> <summary>🔧 <b>Automated preparation of the environment for running Python tools only.</b></summary>

# Prepare Python environment using Ansible playbook "python_machine.yaml".
# If the prompt "BECOME password:" appears, enter the root password.
sudo dnf -y install epel-release
sudo dnf -y install ansible
./prepare_machine.sh -p ansible/python_machine.yaml

# Active Python environment (pipenv) and follow the instructions
# in the README of the individual tools.
. venv/bin/activate

Note: For RHEL 8 (and derived distributions), the system Python will be upgraded to version 3.9, as older versions are no longer supported.

</details> <details> <summary>🔧 <b>Manual preparation of the environment for running Python tools.</b></summary>

If you previously performed the automated installation using the above procedure, you do not need to repeat these steps. These instructions are just in case you can't use Ansible for some reason.

Running the tools requires at least Python 3.9 (see the output of python --version). If an older version is installed on your system, please upgrade first. In addition to Python itself, you need to have pip and setuptools installed. For example, on RHEL 9+ (and derived distributions) you only need to run:

sudo dnf -y install python3 python3-setuptools python3-pip

Next, you need to prepare the Python environment with the dependencies installed:

# Prepare Python virtual environment
pip3 install --upgrade pip
python3 -m venv --prompt flowtest venv

# Activate it
. venv/bin/activate

# Install dependencies
python3 -m pip install pipenv
pipenv install --dev

With active virtual environment, you can now run Python tools.

</details> <details> <summary>🔧 <b>Manual installation of dependencies and build of C++ tools (RHEL & Rocky/Oracle Linux).</b></summary>

If you previously performed the automated installation using the above procedure, you do not need to repeat these steps. These instructions are just in case you can't use Ansible for some reason.

# Enable EPEL RPM repository
sudo dnf -y install epel-release

# Enable system "CodeReady Linux Builder (CRB)" RPM repository.
# Depending on your system, the repository name, i.e. "crb", may vary.
# For example:
# - on RHEL 8 distributions (excluding Oracle Linux) use: "powertools"
# - on Oracle Linux 8 use: "ol8_codeready_builder"
# - on Oracle Linux 9 use: "ol9_codeready_builder"
sudo dnf -y install "dnf-command(config-manager)"
sudo dnf config-manager --set-enabled crb

# Install libraries and build tools
sudo dnf -y install git cmake make rpm-build gcc gcc-c++
sudo dnf -y install libpcap-devel xxhash-libs xxhash-devel glpk-devel
sudo dnf -y install kernel-header libbpf-devel openssl-devel
sudo dnf -y install python3.9-devel
# On RHEL 9+ (and derived distributions) install additional library
sudo dnf -y install libxdp-devel

# Build and optionally install C++ tools (e.g. ft-generator or ft-replay):
mkdir build && cd build
cmake -DCMAKE_INSTALL_PREFIX=/usr ..
make -j8
sudo make install

</details> <details> <summary>🔧 <b>Manual installation of dependencies and build of C++ tools (Debian & Ubuntu)</b></summary>

This manual may not be complete and may vary slightly depending on the distribution and its version. Suggestions for improvements are welcome.

Note: The "sudo" tool must be installed and the user must have administrative rights.

# Install libraries and build tools
sudo apt-get install -y git cmake gcc g++ pkg-config
sudo apt-get install -y libpcap-dev libxxhash-dev libglpk-dev libbpf-dev libssl-dev
sudo apt-get install -y install python3.9-devel
# On Debian 12+ and Ubuntu 22.04+ install additional library
sudo apt-get install -y libxdp-dev

# Build and optionally install C++ tools (e.g. ft-generator or ft-replay):
mkdir build && cd build
cmake -DCMAKE_INSTALL_PREFIX=/usr -DENABLE_RPMBUILD=Off ..
make -j8
sudo make install

</details>

Datasets