Attabench

| :warning: WARNING | |:---------------------------| | This package has been largely superseded by the Swift Collections Benchmark package. That package provides a portable benchmarking solution that works on all platforms that Swift supports, and it is being maintained by the Swift Standard Library team.|

Attabench is a microbenchmarking app for macOS, designed to measure and visualize the performance of Swift code.

Table of Contents

• Background
• Installation
• Usage
• Control What Gets Measured
• Change How Data Gets Displayed
• Get Chart Images Out of Attabench
• Create Your Own Benchmarks
• Get Surprised by Results
• Internal Details: The Attabench Protocol

Background

This app is for microbenchmarking low-level algorithms with one degree of freedom (usually size). It works by repeatedly performing the same operation on random data of various sizes, while continuously charting the results in nice plots. Attabench's default log-log plots are ideal for seeing your algorithm's performance at a glance.

Attabench was originally created to supply nice log-log charts for my dotSwift 2017 talk and Optimizing Collections book. At the time, it seemed easier to build a custom chart renderer from scratch using Core Graphics than to mess with a bunch of CSV files and pivot tables in Excel. (It has to be noted though that this opinion has been somewhat weakened during the implementation process.)

Attabench was made in a hurry for a single use case, so its code is what polite people might call a little messy. But it's shockingly fun to play with, and the graphs it produces are chock full of strange and wonderful little mysteries.

If you find Attabench useful in your own project, please consider buying a copy of my book! It contains a lot of benchmarks made with Attabench; I'm positive you'll find it entertaining and informative.

Installation

Follow these steps to compile Attabench on your own:

1. Clone this repo to your Mac.

   git clone https://github.com/attaswift/Attabench.git Attabench --recursive
   cd Attabench
   

2. Install Carthage if you don't already have it. (This assumes you have Homebrew installed.)

   brew install carthage
   

3. Retrieve and build dependencies (SipHash, BTree and GlueKit).

   carthage bootstrap --platform Mac
   

4. Open the project file in Xcode 9, then build and run the Attabench target.

   open Attabench.xcodeproj
   

Usage

Attabench has two document formats: a benchmark document defining what to test (with the extension .attabench), and a results document containing benchmark timings (extension .attaresult). Results documents remember which benchmark they came from, so you can stop Attabench and restart any particular benchmark run at any point. You may create many result documents for any benchmark.

When the app starts up, it prompts you to open a benchmark document. Each benchmark contains executable code for one or more individually measurable tasks that can take some variable input. The repository contains two examples, so you don't need to start from sratch:

• SampleBenchmark.attabench is a simple example benchmark with just three tasks. It is a useful starting point for starting your own benchmarks.

• OptimizingCollections.attabench is an example of a real-life benchmark definition. It was used to generate the charts in the Optimizing Collections book. (See if you can reproduce my results!)

We are going to look at how to define your own benchmark files later; let's just play with the app first.

Once you load a benchmark, you can press <kbd>⌘-R</kbd> to start running benchmarks with the parameters displayed in the toolbar and the left panel. The chart gets updated in real time as new measurements are made.

You can follow the benchmarking progress by looking at the status bar in the middle panel. Below it there is a console area that includes Attabench status messages. If the benchmark prints anything on standard output or standard error during its run, that too will get included in the console area.

Control What Gets Measured

You can use the checkboxes in the list inside the left panel to control which tasks get executed. If you have many tasks, you can filter them by name using the search field on the bottom. (You can build simple expressions using negation, conjunction (AND) and disjunction (OR) -- for example, typing dog !brown, cat in the search field will get you all tasks whose name includes either dog but not brown, or it includes cat.) To check/uncheck many tasks at once, just select them all and press any of their checkboxes.

The two pop up buttons in the tool bar lets you select the size interval on which you want to run your tasks. Attabench will smoothly sample the interval on a logarithmic curve that perfectly fits the charts.

While there is an active benchmark running, whenever you change something on the left side of the window, the benchmark is immediately stopped and restarted with the new parameters. This includes typing anything in the search bar -- only visible tasks get run. Be careful not to interrupt long-running measurements.

The run options panel in the bottom controls how many times any particular task is executed before a measurement is reported. As a general rule, the task is repeated Iterations times, and the fastest result is used as the measurement. However, when the Min Duration field is set, each task will keep repeating until the specified time has elapsed; this smooths out the charts of super quick benchmarks that would otherwise have really noisy results. On the other hand, a task will not get repeated when it has cumulatively taken more time than the time interval in Max Duration field. (This will get you results quicker for long-running tasks.) So with the Duration fields, any particular task may get run for either more or less times than what you set in the Iteration field.

Change How Data Gets Displayed

The right panel is used to configure how the chart is rendered. Feel free to experiment by tweaking these controls; they only change the appearance of how the results get displayed; they do not affect any currently running benchmark.

The pop up button on the top lets select one from a handful of built-in visual themes to radically change the chart's appearance. For example, the Presentation theme is nice for white-on-black presentation decks. (You currently need to modify the source of the app if you need to change these themes or create your own.)

The three Scales checkboxes lets you enable/disable amortized time display or switch to linear scales on any of the two axes. These are occasionally useful.

The Curves panel includes two pop up buttons for selecting what data to display. For the actual curves, you can choose from the following kinds of data:

• None to disable the curve altogether
• Minimum to show the minimum of all collected measurements.
• Average selects the arithmetic mean of collected samples. This is often the most informative, so this is the default choice.
• Maximum displays the slowest measurement only. This is probably not that useful on its own, but it was really cheap to implement! (And it can be interesting to combine it with the stddev-based error bands.)
• Sample Count is the odd one out: it displays the count of measurements made, not their value. This is a bit of a hack, but it is very useful for determining if you have taken enough measurements. (To get the best view, switch to a linear "time" scale.)

There is also an optional Error Band that you can display around each curve. Here are the available options for these bands:

• None disables them. This is the minimalist choice.
• Maximum paints a faintly colored band between the minimum and maximum measured values.
• The μ + σ option replaces the maximum value with the sum of the average and the standard deviation. (This is the 68% in the 68-95-99.7 rule.)
• μ + 2σ doubles the standard deviation from the previous option. ("95%")
• μ + 3σ goes triple. ("99.7%")

The bottom band is always set to the minimum value, in all cases except None. (E.g., μ - σ can easily go below zero, which looks really bad on a log scale.)

A word of warning: I know nothing about statistics, and I'm not qualified to do proper statistical analysis. I chose these options because they produced cool-looking charts that seemed to tell me something meaningful about the spread of the data. These sigma expressions look suitably scientific, but they are likely not the greate