Overview

MMTests is a configurable test suite that runs performance tests against arbitrary workloads. This is not the only test framework but care is taken to make sure the test configurations are accurate, representative and reproducible. Reporting and analysis is common across all benchmarks. Support exists for gathering additional telemetry while tests are running and hooks exist for more detailed tracing using ftrace or perf.

Quick Introduction

The top-level directory has a single driver script called run-mmtests.sh which reads a config file that describes how the benchmarks should be configured and executed. In some cases, the same benchmarking tool may be used with different configurations that stresses the scenario.

A test run can have any name. A common use case is simply to compare kernel versions but it can be anything —different compiler, different userspace package, different benchmark configuration etc.

Monitors can be optionally configured, but care should be taken as there is a possibility that they introduce overhead of their own. Hence, for some performance sensitive tests it is preferable to have no monitoring.

Many of the tests download external benchmarks. An attempt will be made to download from a mirror if it exists. To get an idea where the mirror should be located, grep for MIRROR_LOCATION= in shellpacks/.

A basic invocation of the suite is

$ ./bin/autogen-configs
$ ./run-mmtests.sh --no-monitor --config configs/config-workload-stream-single 5.8-vanilla
$ ./run-mmtests.sh --no-monitor --config configs/config-workload-stream-single 5.9-vanilla
$ cd work/log
$ ../../compare-kernels.sh
$ mkdir /tmp/html/
$ ../../compare-kernels.sh --format html --output-dir /tmp/html > /tmp/html/index.html

The first step is optional. Some configurations are auto-generated from a template, particularly the filesystem-specific ones.

Note that perhaps Math::Gradient may need to be installed from CPAN for the reporting to work. Similarly, R should be installed if attempting to highlight whether performance differences are statistically relevant.

A tutorial with some more details and the full output of each step is available here:

• MMTests Tutorial

Running Benchmarks with MMTests

Configuration

All available configurations are stored in configs/.

For example config-workload-stream-single can be used to run tests that measure sustainable memory bandwidth and computation rates for vector kernels (i.e. workloads). Similarly there are network, disk and scheduler configs.

The config file can take many options, in the form of export-ed variables. there is a functional sample config file available in config.

Some options are universal, others are specific to the test. Some of the universal ones are:

• MMTESTS: A list of what tests will be run.
• AUTO_PACKAGE_INSTALL: Whether packages necessary for building or running benchmarks should be automatically installed, without asking any confirmation (takes a yes or a no). Creating a file called .mmtests-auto-package-install and putting it in / would be equivalent of having this set to yes.
• MMTESTS_NUMA_POLICY: Whether numad or numactl should be used for deciding (typically, for restricting) on what CPUs and/or NUMA nodes the benchmark will run. It accepts several values. none, numad or interleave, are the simplest, but the following ones can also be used:
  • fullbind_single_instance_node
  • fullbind_single_instance_cpu
  • membind_single_instance_node
  • cpubind_single_instance_node
  • membind_single_instance_node
  • membind_single_instance_node
  • cpubind_largest_nonnode0_memory, in which case, MMTESTS_NODE_ID should also be defined
  • cpubind_node_nrcpus, in which case MMTESTS_NUMA_NODE_NRCPUS should also be defined. If none is used or the option is not present, nothing is done in terms of NUMA pinning of the benchmarks.
• MMTESTS_TUNED_PROFILE: Whether or not the tuned tool should be used and, if yes, with which profile. In fact, the option takes the name of the desired profile (which should be present in the system). If this is defined tuned is started and stopped around the execution of the benchmarks.
• SWAP_CONFIGURATION, SWAP_PARTITIONS, SWAP_SWAPFILE_SIZEMB: It's possible to use a different swap configuration than what is provided by default.
• TESTDISK_RAID_DEVICES, TESTDISK_RAID_MD_DEVICE, TESTDISK_RAID_OFFSET, TESTDISK_RAID_SIZE, TESTDISK_RAID_TYPE: If the target machine has partitions suitable for configuring RAID, they can be specified here. This RAID partition is then used for all the tests.
• TESTDISK_PARTITION: Use this partition for all tests.
• TESTDISK_FILESYSTEM, TESTDISK_MKFS_PARAM, TESTDISK_MOUNT_ARGS: The filesystem, mkfs parameters and mount arguments for the test partitions.
• TESTDISK_DIR: A directory passed to the test. If not set, defaults to SHELLPACK_TEMP. The directory is supposed to contain a precreated environment (eg. a specifically created filesystem mounted with desired mount options).
• STORAGE_CACHE_TYPE, STORAGE_CACHING_DEVICE, STORAGE_BACKING_DEVICE: It's also possible to use storage caching. STORAGE_CACHE_TYPE is either "dm-cache" or "bcache". The devices specified with STORAGE_CACHING_DEVICE and STORAGE_BACKING_DEVICE are used to create the cache device which then is used for all the tests.

Platform Specific Configuration

It is possible to retrieve information about the characteristics of the system where the benchmarks will be running, and use them inside a config file.

For instance:

• MEMTOTAL_BYTES: Tells how much memory there is in the system.
• NUMCPUS: Tells how many CPUs are present in the system.

It is possible to add the following to the config file:

. $SHELLPACK_INCLUDE/include-sizes.sh
get_numa_details

This will give access to more information about the system topology, such as:

• NUMLLCS: Number of Last Level Caches present in the system.
• NUMNODES: Number of NUMA nodes.

Benchmark configurations can then be refined, by taking advantage of the knowledge of the platform characteristics.

For an example check config-workload-stream-omp-llcs, where this is done: STREAM_THREADS=$NUMLLCS. Or config-scheduler-schbench, which has: SCHBENCH_THREADS=$(((NUMCPUS/NUMNODES)-1))

Running Benchmarks

The entry point to running benchmarks is run-mmtests.sh. If run with -h or --help the available options are shown:

 run-mmtests [-mnpb] [-c config-file] test-name

  Options:
  -m|--run-monitors         Run with monitors enabled as specified by the configuration
  -n|--no-monitor           Only execute the benchmark, do not monitor it
  -p|--performance          Set the performance cpufreq governor before starting
  -c|--config               Configuration file to read (default: ./config)
  -b|--build-only           Only build the benchmark, do not execute it

If no config file is specified, the one in ./config is used.

After a run, the benchmark results as well as any data that can be useful for a report will be available in the work/log directory. (more specifically, in work/log/TEST_RUN/iter-0, for a run called TEST_RUN).

Note that often a configuration will run more than just one single benchmark (this depends on the value of the MMTESTS option in the config itself), resulting in some subdirectories being present in the results directory.

Running MMTests as root

Configuring the system for running a benchmark may include doing some changes to the system itself that can only be done with root provileges.

For instance, the config-workload-thpchallenge-fio-defrag config does:

echo always > /sys/kernel/mm/transparent_hugepage/defrag

If starting run-mmtests.sh as a "regular user" doing something like that will fail. benchmarks should still complete (most likely with some warnings) but results will likely not be the ones expected.

In fact, MMTests is intended to be run as root. For most of the changes that it applies to the system, the framework is careful to (try to) undo them. It is however fair to say that MMTests is best used on machines that can be redeployed and reset to a clean known state both before and after running a benchmark.

Monitoring

A full list of available monitors is in monitors/.

The following options, to be defined in the config file, can be used to control monitoring:

• RUN_MONITOR: yes or no switch for deciding whether monitoring should happen or not, during the execution of the benchmarks. If set to no, even if monitors are defined, they will be ignored (but see MONITOR_ALWAYS below). It can be overridden by the --run-monitors and --no-monitor command line parameters. I.e., --run-monitors means we will always run monitors, even if we have RUN_MONITOR=no in the config file (and vice versa, for --no-monitor and RUN_MONITOR=yes).
• MONITORS_ALWAYS: Basically, another override. In fact, monitors defined here will be started even if we have RUN_MONITOR=no and/or --no-monitor.
• MONITORS_GZIP: A list of monitors to be used during the benchmarks. Their output will be saved in compressed (with gzip) log files.
• MONITORS_WITH_LATENCY: A list of monitors to be used during the benchmarks with their output augmented with some additional time