PerfKit Benchmarker

PerfKit Benchmarker is an open effort to define a canonical set of benchmarks to measure and compare cloud offerings. It's designed to operate via vendor provided command line tools. The benchmark default settings are not tuned for any particular platform or instance type. These settings are recommended for consistency across services. Only in the rare case where there is a common practice like setting the buffer pool size of a database do we change any settings.

This README is designed to give you the information you need to get running with the benchmarker and the basics of working with the code. The wiki contains more detailed information:

• FAQ
• Tech Talks
• Governing rules
• Community meeting decks and notes
• Design documents
• You are always welcome to open an issue, or to join us on #PerfKitBenchmarker on freenode to discuss issues you're having, pull requests, or anything else related to PerfKitBenchmarker

Licensing

PerfKit Benchmarker provides wrappers and workload definitions around popular benchmark tools. We made it very simple to use and automate everything we can. It instantiates VMs on the Cloud provider of your choice, automatically installs benchmarks, and runs the workloads without user interaction.

Due to the level of automation you will not see prompts for software installed as part of a benchmark run. Therefore you must accept the license of each of the benchmarks individually, and take responsibility for using them before you use the PerfKit Benchmarker.

Moving forward, you will need to run PKB with the explicit flag --accept-licenses.

In its current release these are the benchmarks that are executed:

• aerospike: Apache v2 for the client and GNU AGPL v3.0 for the server
• bonnie++: GPL v2
• cassandra_ycsb: Apache v2
• cassandra_stress: Apache v2
• cloudsuite3.0: CloudSuite 3.0 license
• cluster_boot: MIT License
• coremark: EEMBC
• copy_throughput: Apache v2
• fio: GPL v2
• gpu_pcie_bandwidth: NVIDIA Software Licence Agreement
• hadoop_terasort: Apache v2
• hpcc: Original BSD license
• hpcg: BSD 3-clause
• iperf: UIUC License
• memtier_benchmark: GPL v2
• mesh_network: HP license
• mongodb: Deprecated. GNU AGPL v3.0
• mongodb_ycsb: GNU AGPL v3.0
• multichase: Apache v2
• netperf: HP license
• oldisim: Apache v2
• object_storage_service: Apache v2
• pgbench: PostgreSQL Licence
• ping: No license needed.
• silo: MIT License
• scimark2: public domain
• speccpu2006: SPEC CPU2006
• SHOC: BSD 3-clause
• sysbench_oltp: GPL v2
• TensorFlow: Apache v2
• tomcat: Apache v2
• unixbench: GPL v2
• wrk: Modified Apache v2
• ycsb (used by mongodb, hbase_ycsb, and others): Apache v2

Some of the benchmarks invoked require Java. You must also agree with the following license:

• openjdk-7-jre: GPL v2 with the Classpath Exception

SPEC CPU2006 benchmark setup cannot be automated. SPEC requires that users purchase a license and agree with their terms and conditions. PerfKit Benchmarker users must manually download cpu2006-1.2.iso from the SPEC website, save it under the perfkitbenchmarker/data folder (e.g. ~/PerfKitBenchmarker/perfkitbenchmarker/data/cpu2006-1.2.iso), and also supply a runspec cfg file (e.g. ~/PerfKitBenchmarker/perfkitbenchmarker/data/linux64-x64-gcc47.cfg). Alternately, PerfKit Benchmarker can accept a tar file that can be generated with the following steps:

• Extract the contents of cpu2006-1.2.iso into a directory named cpu2006
• Run cpu2006/install.sh
• Copy the cfg file into cpu2006/config
• Create a tar file containing the cpu2006 directory, and place it under the perfkitbenchmarker/data folder (e.g. ~/PerfKitBenchmarker/perfkitbenchmarker/data/cpu2006v1.2.tgz).

PerfKit Benchmarker will use the tar file if it is present. Otherwise, it will search for the iso and cfg files.

Getting Started

To quickly get started running PKB, follow one of our tutorials:

• Beginner tutorial for an in-depth but beginner friendly look at PKB's architectures, flags, and even data visualization, using GCP's Cloud Shell & netperf benchmarks.
• Docker tutorial to run PKB in just a few steps, using GCP & docker.
• Continue reading below for installation & setup on all Clouds + discussion of many topics like flags, configurations, preprovisioned data, & how to make contributions.

Installation and Setup

Before you can run the PerfKit Benchmarker, you need account(s) on the cloud provider(s) you want to benchmark (see providers). You also need the software dependencies, which are mostly command line tools and credentials to access your accounts without a password. The following steps should help you get up and running with PKB.

Python 3

The recommended way to install and run PKB is in a virtualenv with the latest version of Python 3 (at least Python 3.12). Most Linux distributions and recent Mac OS X versions already have Python 3 installed at /usr/bin/python3.

If Python is not installed, you can likely install it using your distribution's package manager, or see the Python Download page.

<!-- copybara:strip_begin(internal) --> <!--* pragma: { seclinter_this_is_fine: true } *--> <!-- copybara:strip_end -->

# install pyenv to install python on persistent home directory
curl https://pyenv.run | bash

# add to path
echo 'export PATH="$HOME/.pyenv/bin:$PATH"' >> ~/.bashrc
echo 'eval "$(pyenv init -)"' >> ~/.bashrc
echo 'eval "$(pyenv virtualenv-init -)"' >> ~/.bashrc

# update bashrc
source ~/.bashrc

# install python 3.12 and make default
pyenv install 3.12
pyenv global 3.12

<!-- copybara:strip_begin(internal) --> <!--* pragma: { seclinter_this_is_fine: false } *--> <!-- copybara:strip_end -->

Install PerfKit Benchmarker

Download the latest PerfKit Benchmarker release from GitHub. You can also clone the working version with:

$ cd $HOME
$ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git

Install Python library dependencies:

$ pip3 install -r $HOME/PerfKitBenchmarker/requirements.txt

You may need to install additional dependencies depending on the cloud provider you are using. For example, for AWS:

$ cd $HOME/PerfKitBenchmarker/perfkitbenchmarker/providers/aws
$ pip3 install -r requirements.txt

Preprovisioned data

Some benchmarks may require data to be preprovisioned in a cloud. To preprovision data, you will need to obtain the data and then upload it to that cloud. See more information below about which benchmarks require preprovisioned data and how to upload it to different clouds.

Running a Single Benchmark

PerfKit Benchmarker can run benchmarks both on Cloud Providers (GCP, AWS, Azure, DigitalOcean) as well as any "machine" you can SSH into.

Example run on GCP

$ ./pkb.py --project=<GCP project ID>