<div align="center"> <img src="images/logo.png" width="40%" alt="WARP17, The Stateful Traffic Generator"> </div> <p> <div align="center"> <img src="https://travis-ci.org/Juniper/warp17.svg?branch=dev%2Fcommon" alt="Build status"> </div> </p>

WARP17, The Stateful Traffic Generator for L1-L7 is a lightweight solution for generating high volumes of session based traffic with very high setup rates. WARP17 currently focuses on L5-L7 application traffic (e.g., HTTP) running on top of TCP as this kind of traffic requires a complete TCP implementation. Nevertheless, WARP17 also supports application traffic running on top of UDP.

Developing network components or services usually requires expensive proprietary solutions for validating the implemented functionalities and scalability or performance requirements. WARP17 is a platform agnostic tool based on DPDK which:

• allows extremely fast generation and sustaining of stateful sessions
• offers configurable TCP/UDP infrastructure which can be used for generating high connection setup and data rates for application traffic
• is Linux based so all the openly available tools can be integrated by the users of WARP17.

The WARP17 TCP/IP implementation runs completely in user-space thus avoiding the additional latency in the kernel stack. From a hardware perspective, WARP17 will be able to run on all the platforms that are supported by DPDK.

Performance benchmarks

You can find all the performance tests descriptions and the reference hardware architecture details in the doc folder.

TCP setup and data rates for RAW application traffic

NOTE: In the case when we only want to test the TCP control implementation (i.e., the TCP 3-way handshake and TCP CLOSE sequence), WARP17 achieved the maximum setup rate of 8.5M clients/s and 8.5M servers/s, so a total of 17M TCP sessions are handled every second.

The tests set up 20 million TCP sessions (i.e., 10 million TCP clients and 10 million TCP servers) on which clients continuously send fixed size requests (with random payload) and wait for fixed size responses from the servers.

• TCP raw traffic link utilization reaches line rate (40Gbps) as we increase the size of the requests and responses. When line rate is achieved the number of packets that actually make it on the wire decreases (due to the link bandwidth):

<div align="center"> <img src="./benchmarks/tcp_raw_link_usage.png" width="49%" alt="TCP raw link usage"> <img src="./benchmarks/tcp_raw_pps.png" width="49%" alt="TCP raw pps"> </div>

• TCP raw traffic setup rate is stable at approximately 7M sessions per second (3.5M TCP clients and 3.5M TCP servers per second)

<div align="center"> <img src="./benchmarks/tcp_raw_setup.png" width="49%" alt="TCP raw setup rate"> </div>

TCP setup and data rates for HTTP application traffic

The tests set up 20 million TCP sessions (i.e., 10 million TCP clients and 10 million TCP servers) on which the clients continuously send HTTP GET requests and wait for the HTTP responses from the servers.

• HTTP traffic link utilization reaches line rate (40Gbps) as we increase the size of the requests and responses. When line rate is achieved the number of packets that actually make it on the wire decreases (due to the link bandwidth):

<div align="center"> <img src="./benchmarks/tcp_http_link_usage.png" width="49%" alt="HTTP link usage"> <img src="./benchmarks/tcp_http_pps.png" width="49%" alt="HTTP pps"> </div>

• HTTP traffic setup rate is stable at approximately 7M sessions per second (3.5M HTTP clients and 3.5M HTTP servers per second)

<div align="center"> <img src="./benchmarks/tcp_http_setup.png" width="49%" alt="HTTP setup rate"> </div>

UDP setup and data rates for RAW application traffic

The tests continuously send UDP fixed size packets (with random payload) from 10 million clients which are processed on the receing side by 10 million UDP listeners.

• UDP packets are generated at approximately 22 Mpps (for small packets) and as we reach the link bandwidth the rate decreases.

<div align="center"> <img src="./benchmarks/udp_raw_link_usage.png" width="49%" alt="UDP raw link usage"> <img src="./benchmarks/udp_raw_pps.png" width="49%" alt="UDP raw pps"> </div>

Installing and configuring

Prerequisites

Any 64 bit Linux distribution will do, however we have been testing this with Ubuntu Server 18.04 LTS. In addition we have made an OVF virtual machine image available, details can be found in the respective documentation.

Install DPDK

Run the automated script with <version> as 19.11.3 (the latest LTS supported by warp17)

# ./build_dpdk.sh -v <version>

Install Google Protocol Buffers

Run dep_install.sh as root from the source folder

# ./dep_install.sh

Get WARP17

Get the warp17-<ver>.tgz archive or clone the desired release.

Compile WARP17

tar xfz warp17-<ver>.tgz
cd warp17-<ver>
make

Configure Python virtualenv (on Ubuntu)

sudo apt-get install python3-pip
sudo pip install virtualenv
virtualenv warp17-venv
source warp17-venv/bin/activate
pip install -r python/requirements.txt

Once installed, whenever python tests need to run the virtual environment must be activated:

source warp17-venv/bin/activate

To exit the virtual environment and return to the default python interpretor and libraries:

deactivate

Configure DPDK ports

Use the $RTE_SDK/usertools/dpdk-setup.sh script (as described in the DPDK Guide). Select which ports to be controlled by the IGB UIO module: option Bind Ethernet/Crypto device to IGB UIO module.

How to run

From the top directory of WARP17:

./build/warp17 <dpdk-command-line-args> -- <warp17-command-line-args>

Running as non-root

After compiling WARP17 change the owner of the binary to root (in order to allow access to /proc/self/pagemap:):

sudo chown root build/warp17

Set the suid bit on the binary in order to allow the user to keep permissions:

sudo chmod u+s build/warp17

Command-line arguments

DPDK command-line arguments

• -c <hex_mask>: bitmask specifying which physical cores the application will use. Each bit corresponds to a physical core (0-<max_cores>).
• -n <chan_no> : number of memory channels to be used.
• -m <mem_in_MB>: total memory available to the application (in MB).

Please check section 3.1 of the DPDK App Guide for more info about DPDK command-line arguments.

NOTE: For now WARP17 supports at most 64 cores.

WARP17 command-line arguments

• --version: prints version and exit.

• --help: prints the help and exit.

• --qmap <port>.<hex_mask>: bitmask specifying which physical cores will control the physical port <eth_port>.

• --qmap-default max-c: maximize the number of independent cores handling each physical port.

• --qmap-default max-q: maximize the number of transmit queues per physical port.

• --tcb-pool-sz: configure the size of the TCP control block pool (one TCB is used per TCP connection endpoint). The size of the pool will be given by the argument of this option multiplied by 1024. By default 10M TCBs are allocated.

• --ucb-pool-sz: configure the size of the UDP control block pool (one UCB is used per UDP connection endpoint). The size of the pool will be given by the argument of this option multiplied by 1024. By default 10M UCBs are allocated.

• --mbuf-pool-sz: configure the size of the packet pool. The size of the pool will be given by the argument of this option multiplied by 1024. By default 768K packets are allocated.

• --mbuf-sz: configure the size of a packet fragment (mbuf) in bytes. By default fragments are 2048 bytes.

• --mbuf-hdr-pool-sz: configure the size of the packet headers pool. The size of the pool will be given by the argument of this option multiplied by 1024. By default 512K packet headers are allocated.

• --mpool-any-sock: configure if memory pools should be created from any available memory if the local socket memory is exhausted. By default this feature is disabled as it might affect performance.

• --ring-if-pairs: configure the number of in-memory-ring-based interfaces. NOTE: please check section Using In-Memory-Ring-Based Interfaces for more information.

• --kni-ifs: configure the number of kni interfaces. NOTE: please check section Using Kernel Network Interface (KNI) Interfaces for more information.

• --cmd-file=<file>: CLI command file to be executed when the application starts

NOTE: Options qmap, qmap-default max-c/max-q, cannot be combined. Only one can be passed at a given time.

NOTE: Users are encouraged to use the "qmap-default max-q" option whenever ethernet ports are on the same socket as the PKT cores as this usually gives the best performance!

NOTE: The lowest two cores will be dedicated to CLI and management processing, and can not be assigned to a physical port for packet processing using the --qmap option!

Example (on a x86 server with 32G RAM for WARP17 and 4 memory channels):

• Determine the number of physical cores:

  $ lscpu | grep "CPU(s)"
  CPU(s):                12
  

  Decide how many cores WARP17 should use. In this example we consider WARP17 uses 8 cores:

  • cores 6, 7, 8, 9, 10, 11 for packet processing
  • cores 0, 1 for CLI and management

  Based on that we determine the bitmask corresponding to the ids of the cores we would like to use. The bit index in the bit mask corresponds to the core id:

  Bitmask:  0  0  0  0      1   1  1  1     1