APerf

[!NOTE] Leave us feedback at https://github.com/aws/aperf/discussions/329

What is APerf?

APerf is a CLI tool used for performance monitoring and debugging. It records a wide range of performance-related system metrics or data over a sampling period, such as CPU utilization, memory availability, and PMU counters, and writes them into an archive on disk. APerf's recording is low overhead, and aims to utilize <5% of one CPU. To view the data, APerf processes one or more collected archives, performs analysis, and generates an HTML report. In the report, users can refer to the analytical findings for potential performance issues, or they can browse through all collected data to get a holistic understanding of the systems under test.

Why does APerf exist?

Traditionally, performance issues in applications are investigated by recreating them locally and collecting data/metrics using monitoring tools like sysstat, perf, sysctl, ebpf, etc... or by running these tools remotely. Installing and executing various performance monitoring tools is a manual process and prone to errors. Even after collecting data, understanding the output of these tools requires deep domain specific knowledge.

The goal of APerf is to accelerate the performance debugging process by enabling experienced users to deep dive into extensive data and less experienced users to identify issues without specialized knowledge. This is achieved by,

• Consolidating the collection of a wide array of system metrics into a single command.
• Efficiently visualizing data in an interactive report.
• Performing automated analysis to highlight potential performance issues.

[!TIP] Check out the APerf guide and demo video at https://www.youtube.com/watch?v=mSrDZuxWFtw

Quick Start

Get started with APerf in under 2 minutes:

# 1. Download and extract latest release
arch=$(uname -m); curl -sL $(curl -s https://api.github.com/repos/aws/aperf/releases/latest | grep "browser_download_url.*$arch.*\.tar\.gz" | cut -d'"' -f4) | tar -xz && echo "✓ aperf available at ./aperf-<version>-$arch/aperf"

# 2. Set kernel permissions (non-root users)
sudo sysctl -w kernel.perf_event_paranoid=-1
sudo sysctl -w kernel.kptr_restrict=0
ulimit -n 65535

# 3. Record performance data every 1 second for 60 seconds
./aperf record -r my_run -i 1 -p 60

# 4. Generate report, can be run another machine
./aperf report -r my_run -n my_report

# 5. Open my_report/index.html in your browser

What data does APerf collect?

| Data Type | Description | |--------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------| | Metadata | | | systeminfo | System information and EC2 metadata if running on EC2 instances | | kernel_config | Kernel Configuration (/boot/config) | | sysctl | Sysctl variable configuration settings | | Performance Data | | | cpu_utilization | CPU Utilization, both per CPU and aggregate CPU utilization | | vmstat | Virtual Memory Utilization | | diskstats | Disk Utilization per Disk | | interrupts | Interrupt Data per Interrupt Line per CPU | | perf_stat | CPU Performance Counters | | processes | CPU utilization of running processes | | netstat | TCP/IP stats | | ena_stat | ENA (ethtool) stats | | efa_stat | EFA stats | | meminfo | Memory usage information | | perf_profile | Performance profile data (enabled through the --profile option and the perf binary) | | java_profile | JVM profile data (enabled through the --profile-java option and the async-profiler binary) | | hotline | Memory and branch predictor hot spot detection (needs to be built with the Hotline feature and run on metal instance only) | | memalloc | Memory allocation data including buddyinfo, pagetypeinfo, and slabinfo (some data requires root privileges) | | APerf Execution Data | | | aperf_runlog | The log messages | | aperf_stats | Execution time of each collection interval (including the total time and every data's collection time) |

Installation

Download the binary from the Releases page. APerf record and report are fully supported on Linux. Only report generation (aperf report) is supported on macOS and Windows.

Building from source

1. Download the source code from the Releases page.

2. Install requirements: Rust toolchain (v1.61.0+), Node.js (v16.16.0+), and build tools

   # Install Rust
   curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
   
   # Install Node.js via nvm
   curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.0/install.sh | bash
   source ~/.bashrc  # or source ~/.zshrc if using zsh
   nvm install 16.16.0
   nvm use 16.16.0
   
   # Install build tools
   # On Ubuntu:
   sudo apt install -y build-essential # or on Amazon Linux: sudo yum install kernel-devel
   

3. Build APerf:

   cargo build --release
   

   The binary will be located at target/release/aperf.

Optional - Building with Hotline

Hotline is APerf's in-memory latency and branch hotspot analyzer. It uses ARM Statistical Profiling Extension (SPE) to sample micro-architectural events directly from the CPU pipeline, giving you precise, low-overhead visibility into where your workloads spend time and why.

Hotline produces two categories of analysis:

• Memory latency hotspots — identifies instructions with the highest memory access latency, broken down by execution, issue, and translation latency. A completion node view shows the cache hierarchy distribution (L1/L2/L3/DRAM) for each hot instruction, so you can see whether a load is bottlenecked on a specific cache level.
• Branch misprediction hotspots — identifies instructions with the most branch samples and their misprediction rates.

Hotline can only be built and run on a baremetal Graviton instance. For now, it is not released as a binary. To use the feature, please build from source following the below instructions:

On Ubuntu 22.04:

sudo apt-get update
sudo apt install -y build-essential
sudo apt install linux-modules-extra-$(uname -r)
# KPTI isolates kernel and user space page tables, but interferes with performance monitoring. Disable:
sudo nano /etc/default/grub  # Add/modify: GRUB_CMDLINE_LINUX_DEFAULT="kpti=off"
sudo update-grub
sudo reboot
sudo modprobe arm_spe_pmu
sudo sh -c 'echo 0 > /proc/sys/kernel/kptr_restrict'
sudo sh -c 'echo -1 > /proc/sys/kernel/perf_event_paranoid'
sudo chmod +r /proc/kallsyms
sudo apt-get install libdw-dev libelf-dev libcapstone-dev zlib1g-dev liblzma-dev libbz2-dev libzstd-dev

On Amazon Linux 2 / Amazon Linux 2023: