Nighthawk

A L7 (HTTP/HTTPS/HTTP2) performance characterization tool

Current state

Nighthawk currently offers:

• A load testing client which supports HTTP/1.1 and HTTP/2 over HTTP and HTTPS. (HTTPS certificates are not yet validated).
• A simple test server which is capable of generating dynamic response sizes, as well as inject delays.
• A binary to transform nighthawk output to well-known formats, allowing integration with other systems and dashboards.

Navigating the codebase

See navigating the codebase for a description of the directory structure.

Additional Documentation

See the howto directory for documentation aimed at specific use-cases.

Building Nighthawk

Prerequisites

Note that Nighthawk uses Envoy's code directly, so building Envoy is a prerequisite for building Nighthawk. Start by looking at Envoy's building documentation.

Compiler requirements

The main supported way of building Nighthawk is with the Clang compiler. At least Clang/LLVM 12+ is needed to successfully build Nighthawk.

Bazel

Both Envoy and Nighthawk use the Bazel build tool. The steps required to set up Bazel are documented in Envoy's Quick start Bazel build for developers.

Building on Ubuntu

This section outlines the steps needed to build on Ubuntu. Note that these steps include commands that are documented in the prerequisites section above.

Install required packages

Run the following command to install the required packages.

sudo apt-get install \
   autoconf \
   automake \
   cmake \
   curl \
   libtool \
   make \
   ninja-build \
   patch \
   python3-pip \
   unzip \
   virtualenv

Install Clang/LLVM

Note that depending on the chosen Ubuntu version, you may need to manually install a never version of Clang/LLVM. The installed version of Clang can be verified by running:

clang -v

If you do need to install a newer version, be sure to use Ubuntu's update-alternatives or a similar approach to switch to using the newer Clang/LLVM. See issue#832 for one possible approach.

Run the following commands to install Clang/LLVM.

sudo apt install -y lld clang llvm lld lldb
sudo apt install -y clang-{format,tidy,tools} clang-doc clang-examples

Install Bazelisk instead of bazel

Bazelisk is recommended, since it automatically chooses and downloads the appropriate Bazel version. If you already have Bazel installed, it is strongly recommended to remove it.

Run the following to remove bazel.

sudo apt-get remove bazel

Run the following to install Bazelisk.

sudo wget -O /usr/local/bin/bazel https://github.com/bazelbuild/bazelisk/releases/latest/download/bazelisk-linux-$([ $(uname -m) = "aarch64" ] && echo "arm64" || echo "amd64")
sudo chmod +x /usr/local/bin/bazel

Clone Nighthawk and setup Clang as the compiler

Run the following to clone the Nighthawk repository and instruct Bazel to use Clang.

git clone https://github.com/envoyproxy/nighthawk
cd nighthawk/
echo "build --config=clang" >> user.bazelrc

Install Python libraries

It is advisable to use the same version of Python as the one listed at the top of tools/base/requirements.txt. While other versions may also work, the chances of success are greatest if using the same one.

Recommended: Use virtualenv to avoid conflicts between Nighthawk's Python package version requirements and other versions already on your system:

python3 -m venv ~/my_nh_venv
source ~/my_nh_venv/bin/activate

Note: Avoid creating the environment under the Nighthawk project directory.

Install Python packages required for Nighthawk (whether using virtualenv or not):

pip3 install --user -r tools/base/requirements.txt

If pip3 install fails, you will need to troubleshoot the Python environment before attempting to build and test Nighthawk.

Build and testing Nighthawk

You can now use the CI script to build Nighthawk.

ci/do_ci.sh build

Or to execute its tests.

ci/do_ci.sh test

Note that after building completes, the Nighthawk binaries are located in the bazel-bin/ directory located at the root of the cloned Nighthawk repository.

Using the Nighthawk client CLI

For using the Nighthawk test server, see here.

➜ bazel-bin/nighthawk_client --help

<!-- BEGIN USAGE -->

USAGE:

bazel-bin/nighthawk_client  [--user-defined-plugin-config <string>] ...
[--latency-response-header-name <string>]
[--stats-flush-interval-duration <duration>]
[--stats-flush-interval <uint32_t>]
[--stats-sinks <string>] ... [--no-duration]
[--simple-warmup]
[--request-source-plugin-config <string>]
[--request-source <uri format>] [--label
<string>] ... [--multi-target-use-https]
[--multi-target-path <string>]
[--multi-target-endpoint <string>] ...
[--experimental-h2-use-multiple-connections]
[--nighthawk-service <uri format>]
[--jitter-uniform <duration>] [--open-loop]
[--experimental-h1-connection-reuse-strategy
<mru|lru>] [--no-default-failure-predicates]
[--failure-predicate <string:uint64_t>] ...
[--termination-predicate <string:uint64_t>]
... [--trace <uri format>]
[--sequencer-idle-strategy <spin|poll
|sleep>] [--max-concurrent-streams
<uint32_t>] [--max-requests-per-connection
<uint32_t>] [--max-active-requests
<uint32_t>] [--max-pending-requests
<uint32_t>] [--transport-socket <string>]
[--upstream-bind-config <string>]
[--tls-context <string>]
[--request-body-size <uint32_t>]
[--request-header <string>] ...
[--request-method <GET|HEAD|POST|PUT|DELETE
|CONNECT|OPTIONS|TRACE>] [--address-family
<auto|v4|v6>] [--burst-size <uint32_t>]
[--prefetch-connections] [--output-format
<json|human|yaml|dotted|fortio
|experimental_fortio_pedantic|csv
|prometheus>] [-v <trace|debug|info|warn
|error|critical>] [--concurrency <string>]
[--tunnel-tls-context <string>]
[--tunnel-uri <string>] [--tunnel-protocol
<http1|http2|http3>]
[--http3-protocol-options <string>] [-p
<http1|http2|http3>] [--h2] [--timeout
<uint32_t>] [--duration <uint32_t>]
[--connections <uint32_t>] [--rps
<uint32_t>] [--] [--version] [-h] <uri
format>


Where:

--user-defined-plugin-config <string>  (accepted multiple times)
WIP - will throw unimplemented error. Optional configurations for
plugins that collect data about responses received by NH and attach a
corresponding UserDefinedOutput to the Result. Example (json):
{name:"nighthawk.fake_user_defined_output"
,typed_config:{"@type":"type.googleapis.com/nighthawk.FakeUserDefinedO
utputConfig",fail_data:"false"}}

--latency-response-header-name <string>
Set an optional header name that will be returned in responses, whose
values will be tracked in a latency histogram if set. Can be used in
tandem with the test server's response option
"emit_previous_request_delta_in_response_header" to record elapsed
time between request arrivals. Default: ""

--stats-flush-interval-duration <duration>
Time interval (in Duration) between flushes to configured stats sinks.
For example '1s' or '1.000000001s'. Mutually exclusive with
--stats-flush-interval.

--stats-flush-interval <uint32_t>
Time interval (in seconds) between flushes to configured stats sinks.
Mutually exclusive with --stats-flush-interval-duration. Default: 5.

--stats-sinks <string>  (accepted multiple times)
Stats sinks (in json) where Nighthawk metrics will be flushed. This
argument is intended to be specified multiple times. Example (json):
{name:"envoy.stat_sinks.statsd"
,typed_config:{"@type":"type.googleapis.com/envoy.config.metrics.v3.St
atsdSink",tcp_cluster_name:"statsd"}}

--no-duration
Request infinite execution. Note that the default failure predicates
will still be added. Mutually exclusive with --duration.

--simple-warmup
Perform a simple single warmup request (per worker) before starting
execution. Note that this will be reflected in the counters that
Nighthawk writes to the output. Default is false.

--request-source-plugin-config <string>
[Request
Source](https://github.com/envoyproxy/nighthawk/blob/main/docs/root/ov
erview.md#requestsource) plugin configuration in json. Mutually
exclusive with --request-source. Example (json):
{name:"nighthawk.stub-request-source-plugin"
,typed_config:{"@type":"type.googleapis.com/nighthawk.request_source.S
tubPluginConfig",test_value:"3"}}

--request-source <uri format>
Remote gRPC source that will deliver to-be-replayed traffic. Each
worker will separately connect to this source. For example
grpc://127.0.0.1:8443/. Mutually exclusive with
--request_source_plugin_config.

--label <string>  (accepted multiple times)
Label. Allows specifying multiple labels which will be persisted in
structured output formats.

--multi-target-use-https
Use HTTPS to connect to the target endpoints. Otherwise HTTP is used.
Mutually exclusive with providing a URI.

--multi-target-path <string>
The single absolute path Nighthawk should request from each target
endpoint. Required when using --multi-target-endpoint. Mutually
exclusive with providing a URI.

--multi-target-endpoint <string>  (accepted multiple times)
Target endpoint in the form IPv4:port, [IPv6]:port, or DNS:port. This
argument is intended to be specified multiple times. Nighthawk will
spread traffic across all endpoints with round robin distribution.
Mutually exclusive with providing a URI.

--experimental-h2-use-multiple-connections
DO NOT USE: This option is deprecated, if this beha