CppBenchmark

<br/> <br/>

C++ Benchmark Library allows to create performance benchmarks of some code to investigate average/minimal/maximal execution time, items processing processing speed, I/O throughput. CppBenchmark library has lots of features and allows to make benchmarks for different kind of scenarios such as micro-benchmarks, benchmarks with fixtures and parameters, threads benchmarks, produsers/consummers pattern.

CppBenchmark API reference

Contents

• Features
• Requirements
• How to build?
• How to create a benchmark?
• Benchmark examples
  • Example 1: Benchmark of a function call
  • Example 2: Benchmark with cancelation
  • Example 3: Benchmark with static fixture
  • Example 4: Benchmark with dynamic fixture
  • Example 5: Benchmark with parameters
  • Example 6: Benchmark class
  • Example 7: Benchmark I/O operations
  • Example 8: Benchmark latency with auto update
  • Example 9: Benchmark latency with manual update
  • Example 10: Benchmark threads
  • Example 11: Benchmark threads with fixture
  • Example 12: Benchmark single producer, single consumer pattern
  • Example 13: Benchmark multiple producers, multiple consumers pattern
  • Example 14: Dynamic benchmarks
• Command line options

Features

• Cross platform (Linux, MacOS, Windows)
• Micro-benchmarks
• Benchmarks with static fixtures and dynamic fixtures
• Benchmarks with parameters (single, pair, triple parameters, ranges, ranges with selectors)
• Benchmark infinite run with cancelation
• Benchmark items processing speed
• Benchmark I/O throughput
• Benchmark latency with High Dynamic Range (HDR) Histograms
• Benchmark threads
• Benchmark producers/consumers pattern
• Different reporting formats: console, csv, json
• Colored console progress and report

Requirements

• Linux
• MacOS
• Windows
• cmake
• gcc
• git
• gil
• python3

Optional:

• clang
• CLion
• Cygwin
• MSYS2
• MinGW
• Visual Studio

How to build?

Install gil (git links) tool

pip3 install gil

Setup repository

git clone https://github.com/chronoxor/CppBenchmark.git
cd CppBenchmark
gil update

Linux

cd build
./unix.sh

MacOS

cd build
./unix.sh

Windows (Cygwin)

cd build
unix.bat

Windows (MSYS2)

cd build
unix.bat

Windows (MinGW)

cd build
mingw.bat

Windows (Visual Studio)

cd build
vs.bat

How to create a benchmark?

1. Build CppBenchmark library
2. Create a new *.cpp file
3. Insert #include "benchmark/cppbenchmark.h"
4. Add benchmark code (examples for different scenarios you can find below)
5. Insert BENCHMARK_MAIN() at the end
6. Compile the *.cpp file and link it with CppBenchmark library
7. Run it (see also possible command line options)

Benchmark examples

Example 1: Benchmark of a function call

#include "benchmark/cppbenchmark.h"

#include <math.h>

// Benchmark sin() call for 5 seconds (by default).
// Make 5 attemtps (by default) and choose one with the best time result.
BENCHMARK("sin")
{
    sin(123.456);
}

BENCHMARK_MAIN()

Report fragment is the following:

===============================================================================
Benchmark: sin()
Attempts: 5
Duration: 5 seconds
-------------------------------------------------------------------------------
Phase: sin()
Average time: 6 ns/op
Minimal time: 6 ns/op
Maximal time: 6 ns/op
Total time: 858.903 ms
Total operations: 130842248
Operations throughput: 152336350 ops/s
===============================================================================

Example 2: Benchmark with cancelation

#include "benchmark/cppbenchmark.h"

// Benchmark rand() call until it returns 0.
// Benchmark will print operations count required to get 'rand() == 0' case.
// Make 10 attemtps and choose one with the best time result.
BENCHMARK("rand-till-zero", Settings().Infinite().Attempts(10))
{
    if (rand() == 0)
        context.Cancel();
}

BENCHMARK_MAIN()

Report fragment is the following:

===============================================================================
Benchmark: rand()-till-zero
Attempts: 10
-------------------------------------------------------------------------------
Phase: rand()-till-zero
Average time: 15 ns/op
Minimal time: 15 ns/op
Maximal time: 92 ns/op
Total time: 159.936 mcs
Total operations: 10493
Operations throughput: 65607492 ops/s
===============================================================================

Example 3: Benchmark with static fixture

Static fixture will be constructed once per each benchmark, will be the same for each attempt / operation and will be destructed at the end of the benchmark.

#include "macros.h"

#include <list>
#include <vector>

template <typename T>
class ContainerFixture
{
protected:
    T container;

    ContainerFixture()
    {
        for (int i = 0; i < 1000000; ++i)
            container.push_back(rand());
    }
};

BENCHMARK_FIXTURE(ContainerFixture<std::list<int>>, "std::list<int>.forward")
{
    for (auto it = container.begin(); it != container.end(); ++it)
        ++(*it);
}

BENCHMARK_FIXTURE(ContainerFixture<std::list<int>>, "std::list<int>.backward")
{
    for (auto it = container.rbegin(); it != container.rend(); ++it)
        ++(*it);
}

BENCHMARK_FIXTURE(ContainerFixture<std::vector<int>>, "std::vector<int>.forward")
{
    for (auto it = container.begin(); it != container.end(); ++it)
        ++(*it);
}

BENCHMARK_FIXTURE(ContainerFixture<std::vector<int>>, "std::vector<int>.backward")
{
    for (auto it = container.rbegin(); it != container.rend(); ++it)
        ++(*it);
}

BENCHMARK_MAIN()

Report fragment is the following:

===============================================================================
Benchmark: std::list<int>-forward
Attempts: 5
Duration: 5 seconds
-------------------------------------------------------------------------------
Phase: std::list<int>-forward
Average time: 6.332 ms/op
Minimal time: 6.332 ms/op
Maximal time: 6.998 ms/op
Total time: 4.958 s
Total operations: 783
Operations throughput: 157 ops/s
===============================================================================
Benchmark: std::list<int>-backward
Attempts: 5
Duration: 5 seconds
-------------------------------------------------------------------------------
Phase: std::list<int>-backward
Average time: 7.883 ms/op
Minimal time: 7.883 ms/op
Maximal time: 8.196 ms/op
Total time: 4.911 s
Total operations: 623
Operations throughput: 126 ops/s
==================================