Intel® PerfSpect

Getting PerfSpect | Running PerfSpect | Building PerfSpect

Intel® PerfSpect is a command-line tool designed to help you analyze and optimize Linux servers and the software running on them. Whether you’re a system administrator, a developer, or a performance engineer, PerfSpect provides comprehensive insights and actionable recommendations to enhance performance and efficiency.

We welcome bug reports and enhancement requests, which can be submitted via the Issues section on GitHub. For those interested in contributing to the code, please refer to the guidelines outlined in the CONTRIBUTING.md file.

Getting PerfSpect

Pre-built PerfSpect releases are available in the repository's Releases. Download and extract perfspect.tgz.

wget -qO- https://github.com/intel/PerfSpect/releases/latest/download/perfspect.tgz | tar -xvz
cd perfspect

Running PerfSpect

PerfSpect includes a suite of commands designed to analyze and optimize both system and software performance.

Usage:
  perfspect [command] [flags]

Commands

| Command | Description | Reference | | ------- | ----------- | --------- | | metrics | CPU core and uncore metrics | options | | report | System configuration and health | options | | benchmark | Performance benchmarks | options | | telemetry | System telemetry | options | | flamegraph | Software call-stacks as flamegraphs | options | | lock | Software hot spot, cache-to-cache and lock contention | options | | config | Modify system configuration | options |

[!TIP] Run perfspect [command] -h to view command-specific help text. See perfspect -h for global options.

Additional commands: update checks for and applies application updates (Intel network only), and extract extracts embedded resources (for developers).

Metrics Command

The metrics command generates reports containing CPU architectural performance characterization metrics in HTML and CSV formats. Run perfspect metrics.

Live Metrics

The metrics command supports two modes -- default and "live". Default mode behaves as above -- metrics are collected and saved into report files for review. The "live" mode prints the metrics to stdout where they can be viewed in the console and/or redirected into a file or observability pipeline. Run perfspect metrics --live.

Metrics Without Root Permissions

If neither sudo nor root access is available, an administrator must apply the following configuration to the target system(s):

• sysctl -w kernel.perf_event_paranoid=0
• sysctl -w kernel.nmi_watchdog=0
• write '125' to all perf_event_mux_interval_ms files found under /sys/devices/*, for example, for i in $(find /sys/devices -name perf_event_mux_interval_ms); do echo 125 > $i; done

Once the configuration changes are applied, use the --noroot flag on the command line, for example, perfspect metrics --noroot.

Refining Metrics to a Specific Time Range

After collecting metrics, you can generate new summary reports for a specific time interval using the metrics trim subcommand. This is useful when you've collected metrics for an entire workload but want to analyze only a specific portion, excluding setup, teardown, or other unwanted phases.

The time range can be specified using either absolute timestamps (seconds since epoch) or relative offsets from the beginning/end of the data. At least one time parameter must be specified.

The trimmed CSV and HTML summary files will be placed in a new output directory. The output directory can be specified using the --output flag.

Examples:

# Skip the first 10 seconds and last 5 seconds
$ ./perfspect metrics trim --input perfspect_2025-11-28_09-21-56 --start-offset 10 --end-offset 5

# Use absolute timestamps (seconds since epoch)
$ ./perfspect metrics trim --input perfspect_2025-11-28_09-21-56 --start-time 1764174327 --end-time 1764174351

Prometheus Endpoint

The metrics command can expose metrics via a Prometheus compatible metrics endpoint. This allows integration with Prometheus monitoring systems. To enable the Prometheus endpoint, use the --prometheus-server flag. By default, the endpoint listens on port 9090. The port can be changed using the --prometheus-server-addr flag. Run perfspect metrics --prometheus-server. See the example daemonset for deploying in Kubernetes.

See perfspect metrics -h for the extensive set of options and examples.

Report Command

The report command generates system configuration reports in a variety of formats. All categories of information are collected by default. See perfspect report -h for all options.

It's possible to report a subset of information by providing command line options. Note that by specifying only the txt format, it is printed to stdout, as well as written to a report file.

$ ./perfspect report --bios --format txt
BIOS
====
Vendor:       Intel Corporation
Version:      EGSDCRB1.SYS.1752.P05.2401050248
Release Date: 01/05/2024

Benchmark Command

The benchmark command runs performance micro-benchmarks to evaluate system health and performance characteristics. All benchmarks are run by default unless specific benchmarks are selected. A brief system summary is included in the output by default.

[!IMPORTANT] Benchmarks should be run on idle systems to ensure accurate measurements and to avoid interfering with active workloads.

Examples:

./perfspect benchmark                  # Run all benchmarks with system summary
./perfspect benchmark --speed --power  # Run specific benchmarks
./perfspect benchmark --no-summary     # Exclude system summary from output

See perfspect benchmark -h for all options.

| Benchmark | Description | | --------- | ----------- | | speed | runs each stress-ng cpu-method for 1s each, reports the geo-metric mean of all results. | | power | runs stress-ng to load all cpus to 100% for 60s. Uses turbostat to measure power. | | temperature | runs the same micro benchmark as 'power', but extracts maximum temperature from turbostat output. | | frequency | runs avx-turbo to measure scalar and AVX frequencies across processor's cores. Note: Runtime increases with core count. | | memory | runs Intel(r) Memory Latency Checker (MLC) to measure memory bandwidth and latency. Note: MLC is not included with PerfSpect. It can be downloaded from Intel Memory Latency Checker. Once downloaded, extract the Linux executable and place it in the perfspect/tools/x86_64 directory. | | cache | runs MLC to measure L1, L2, and L3 cache idle latency (ns) and maximum bandwidth (GB/s). See Note above about downloading MLC. | | storage | runs fio for 2 minutes across multiple I/O patterns to measure storage latency, IOPs, and bandwidth. Use --storage-dir to override the default location (/tmp). Minimum 32GB disk space required. |

Telemetry Command

The telemetry command reports CPU utilization, instruction mix, disk stats, network stats, and more on the specified target(s). All telemetry types are collected by default. To choose telemetry types, see the additional command line options (perfspect telemetry -h).

Additional Telemetry via Environment Variables

The following optional telemetry sources can be enabled via environment variables:

| Environment Variable | Description | | -------------------- | ----------- | | PERFSPECT_GAUDI_HLSMI_PATH | Full path to the Gaudi hl-smi binary. When set, enables collection of Intel Gaudi accelerator telemetry (temperature, utilization, memory, power). | | PERFSPECT_PDU_HOST | Raritan PDU hostname or IP address for power telemetry. Requires all four PDU variables to be set. | | PERFSPECT_PDU_USER | Username for PDU authentication. | | PERFSPECT_PDU_PASSWORD | Password f