<img width="100%" src="src/images/hotspot-logo.png">

Hotspot - the Linux perf GUI for performance analysis

This project is a KDAB R&D effort to create a standalone GUI for performance data. As the first goal, we want to provide a UI like KCachegrind around Linux perf. Looking ahead, we intend to support various other performance data formats under this umbrella.

Table of Contents

<!-- TOC generated with https://github.com/jonschlinkert/markdown-toc To update, run `markdown-toc -i README.md` --> <!-- toc -->

• Screenshots
  • Visualizing Data
  • Time Line
  • Record Data
• Getting Hotspot
  • Gentoo
  • Via package manager
  • For any Linux distro: AppImage
• Building Hotspot
• Using
  • Off-CPU Profiling
  • Embedded Systems
  • Import Export
  • tracepoints
  • Disassembler
  • Rust v0 Mangling
• Known Issues
  • Broken Backtraces
  • debuginfod
  • Missing Features
  • Recording with perf without super user rights
  • Export File Format
• Qt Creator
• License

<!-- tocstop -->

Screenshots

Here are some screenshots showing the most important features of Hotspot in action:

Visualizing Data

The main feature of Hotspot is the graphical visualization of a perf.data file.

Note: Inlined functions have a darker border than non-inlined ones.

Time Line

The time line allows filtering the results by time, process or thread. The data views are updated accordingly.

Record Data

You can also launch perf from Hotspot, to profile a newly started application or to attach to already running process(es). Do take the caveats below into account though.

Getting Hotspot

Note: Hotspot is not yet packaged on all Linux distributions. In such cases, or if you want to use the latest version, please use the AppImage which will work on any recent Linux distro just fine.

Gentoo

Hotspot ebuilds are available from our overlay (https://github.com/KDAB/kdab-overlay).

Via package manager

For any Linux distro: AppImage

You can either head over to the most current release or the continuous build. In both cases you'll find the AppImage under "Assets" that you can download. Untar the AppImage file (in case of the latest release), then make it executable and then run it.

Please use the latest build to get the most recent version. If it doesn't work, please report a bug and test the latest stable version.

Note: Your system libraries or preferences are not altered. In case you'd like to remove Hotspot again, simply delete the downloaded file. Learn more about AppImage here.

To find out how to debug the AppImage, see HACKING.

Building Hotspot

Building Hotspot from source gives you the latest and greatest, but you'll have to make sure all its dependencies are available. Most users should probably install Hotspot from the distro package manager or as an AppImage.

For everyone that wants to contribute to Hotspot or use the newest version without the AppImage detailed notes are found at HACKING.

Using

General

First of all, record some data with perf. To get backtraces, you will need to enable the dwarf callgraph mode:

perf record --call-graph dwarf <your application>
...
[ perf record: Woken up 58 times to write data ]
[ perf record: Captured and wrote 14.874 MB perf.data (1865 samples) ]

Now, if you have Hotspot available on the same machine, all you need to do is launch it. It will automatically open the perf.data file in the current directory (similar to perf report).

Alternatively, you can specify the path to the data file on the console:

hotspot /path/to/perf.data

Command Line options

Depending on your needs you may want to pass additional command line options to Hotspot. This allows to one-time set configuration options that are found in the GUI under "Settings" and also allows to convert Linux perf data files into the smaller and portable perfdata format (see Import / Export for details on that). All command line options are shown with --help:

Usage: hotspot [options] [files...]
Linux perf GUI for performance analysis.

Options:
  -h, --help               Displays help on commandline options.
  --help-all               Displays help including Qt specific options.
  -v, --version            Displays version information.
  --sysroot <path>         Path to sysroot which is used to find libraries.
  --kallsyms <path>        Path to kallsyms file which is used to resolve
                           kernel symbols.
  --debugPaths <paths>     Colon separated list of paths that contain debug
                           information. These paths are relative to the
                           executable and not to the current working directory.
  --extraLibPaths <paths>  Colon separated list of extra paths to find
                           libraries.
  --appPath <path>         Path to folder containing the application executable
                           and libraries.
  --sourcePaths <paths>    Colon separated list of search paths for the source
                           code.
  --arch <path>            Architecture to use for unwinding.
  --exportTo <path>        Path to .perfparser output file to which the input
                           data should be exported. A single input file has to
                           be given too.
  --perf-binary <path>     Path to the perf binary.
  --objdump-binary <path>  Path to the objdump binary.

Arguments:
  files                    Optional input files to open on startup, i.e.
                           perf.data files.

Off-CPU Profiling

Hotspot supports a very powerful way of doing wait-time analysis, or off-CPU profiling. This analysis is based on kernel tracepoints in the linux scheduler. By recording that data, we can find the time delta during which a thread was not running on the CPU, but instead was off-CPU. There can be multiple reasons for that, all of which can be found using this technique:

• synchronous I/O, e.g. via read() or write()
• page faults, e.g. when accessing mmap()'ed file data
• calls to nanosleep() or yield()
• lock contention via futex() etc.
• preemption
• and probably many more

By leveraging kernel trace points in the scheduler, the overhead is pretty manageable and we only pay a price, when the process is actually getting switched out. Most notably we do not pay a price when e.g. a mutex lock operation can be handled directly in user-space.

To do off-CPU analysis with Hotspot, you need to record the data with a very specific command:

perf record \
    -e cycles \                             # on-CPU profiling
    -e sched:sched_switch --switch-events \ # off-CPU profiling
    --sample-cpu \                          # track on which core code is executed
    -m 8M \                                 # reduce chance of event loss
    --aio -z \                              # reduce disk-I/O overhead and data size
    --call-graph dwarf \                    # we definitely want backtraces
    <your application>

Alternatively, you can use the off-CPU check box in Hotspot's integrated record page.

During the analysis, you can then switch between the "cycles" cost view for on-CPU data to the "off-CPU time" cost view for wait-time analysis. Often, you will want to change between both, e.g. to find places in your code which may require further parallelization (see also Amdahl's law).

The "sched:sched_switch" cost will also be sh