uftrace

uftrace is a function call graph tracer for C, C++, Rust and Python programs.

It hooks into the entry and exit of each function, recording timestamps as well as the function's arguments and return values. uftrace is capable of tracing both user and kernel functions, as well as library functions and system events providing an integrated execution flow in a single timeline.

Initially, uftrace only supported function tracing with compiler support. However, it now allows users to trace function calls without recompilation by analyzing instructions in each function prologue and dynamically and selectively patching those instructions.

Users can also write and run scripts for each function entry and exit using python/luajit APIs to create custom tools for their specific purposes.

uftrace offers various filters to reduce the amount of trace data and provides visualization using Chrome trace viewer and flame graph or call-graph diagrams for graphviz and mermaid, allowing for a big picture view of the execution flow.

It was heavily inspired by the ftrace framework of the Linux kernel and the name uftrace stems from the combination of user and ftrace.

It can record data from:

User space C/C++/Rust functions, by either dynamically patching functions using -P., or else selective NOP patching using code compiled with -pg, -finstrument-functions or -fpatchable-function-entry=N.
C/C++/Rust Library functions (through PLT hooking)
Python functions (using Python's trace/profile infrastructure)
Kernel functions (using the ftrace framework in Linux kernel)
Kernel trace events (using event tracing framework in Linux kernel)
Task creation, termination and scheduling events (using Linux perf_event)
User space events in the target binary or libraries (using SystemTap SDT ABI)
PMU counter values for given functions (using Linux perf_event)

With the recorded data, uftrace can:

Show colored and nested function call graphs.
Show arguments and return values symbolically using libc function prototypes and DWARF debug information.
Apply filters to minimize the amount of trace data in both record and replay time.
Extract metadata from traces. (e.g. system information on which the trace was taken)
Generate symbol tables and memory maps of the traced program and library functions.
Generate task relationship trees (parent and children) of nested programs in traces.

It supports many commands and filters such as filtering by function call duration for analysis of program execution and performance.

uftrace-live-demo

Homepage: https://github.com/namhyung/uftrace
Tutorial: https://github.com/namhyung/uftrace/wiki/Tutorial
Chat: https://discord.gg/MENMKaCWqD (Discord)
Mailing list: uftrace@googlegroups.com
Lightning talk: https://youtu.be/LNav5qvyK7I

Features

uftrace traces each function in the executable and shows time durations.

Usually, for this to be possible, the program needs to be compiled with -pg or -fpatchable-function-entry=5 (=2 is enough on aarch64). With full dynamic tracing (-P.|--patch=.), uftrace works on all executables (as long they are not stripped, or symbol information is available from a separate file).

uftrace hooks into the PLT in the given executable file to trace library calls and with (-l|--nest-libcall), it also hooks into the procedure linkage tables (PLTs) of shared libraries. The depth can be limited using -D<num>, where 1 is flat call tracing.

Using (-a|--auto-args), uftrace automatically records arguments and return values of known functions. Without extra debug information, this includes the API functions of standard (C language or system) libraries. This can be combined with -P. or -l: For example, -la traces nested library calls, even in stripped executables.

In addition, -a implies --srcline, so it records the source line location info, and this info can be shown by uftrace replay --srcline and in uftrace tui. Users can directly open the editor at the source location as shown in https://uftrace.github.io/slide/#120.

If debug information for the program (gcc -g) is available, --auto-args works even on functions inside the user-compiled programs.

In case argument information is not available, argument specifications like (-A udev_new@arg1/s) can be passed on the command line or an options file.

Example:

$ uftrace record -la -A udev_new@arg1/s lsusb >/dev/null
$ uftrace replay -f+module
or simply:
$ uftrace -la -A udev_new@arg1/s -f+module lsusb  # -f+module adds the module name
# DURATION     TID        MODULE NAME   FUNCTION
 306.339 us [ 23561]            lsusb | setlocale(LC_TYPE, "") = "en_US.UTF-8";
   1.163 us [ 23561]            lsusb | getopt_long(1, 0x7fff7175f6a8, "D:vtP:p:s:d:Vh") = -1;
            [ 23561]            lsusb | udev_new("POSIXLY_CORRECT") {
   0.406 us [ 23561] libudev.so.1.7.2 |   malloc(16) = 0x55e07277a7b0;
   2.620 us [ 23561]            lsusb | } /* udev_new */
            [ 23561]            lsusb | udev_hwdb_new() {
   0.427 us [ 23561] libudev.so.1.7.2 |   calloc(1, 200) = 0x55e07277a7d0;
   5.829 us [ 23561] libudev.so.1.7.2 |   fopen64("/etc/systemd/hwdb/hwdb.bin", "re") = 0;

Furthermore, it can show detailed execution flow at function level, and report which functions had the longest execution time. It also shows information about the execution environment.

You can set up filters to exclude or include specific functions when tracing. In addition, function arguments and return values can be saved and shown later.

It supports multi-process and/or multi-threaded applications. With root privileges and if the kernel was built with CONFIG_FUNCTION_GRAPH_TRACER=y, kernel functions can be traced as well.

How to build and install uftrace

On Linux distros, misc/install-deps.sh can be used to install required software(s) for building uftrace. Those are for optional and advanced features, but are highly recommended.

$ sudo misc/install-deps.sh

Once you installed required software(s) on your system, it can be built and installed like following:

$ ./configure
$ make
$ sudo make install

For details about installation and dependencies, please refer to INSTALL.md

How to use uftrace

These are the commands supported by uftrace:

record : runs a program and saves the trace data
replay : shows program execution in the trace data
report : shows performance statistics in the trace data
live : does record and replay in a row (default)
info : shows system and program info in the trace data
dump : shows low-level trace data
recv : saves the trace data from network
graph : shows function call graph in the trace data
script : runs a script for recorded trace data
tui : show text user interface for graph and report

You can use -h or --help option to see available commands and options.

$ uftrace
uftrace -- function (graph) tracer for userspace

 usage: uftrace [COMMAND] [OPTION...] [<program>]

 COMMAND:
   record          Run a program and saves the trace data
   replay          Show program execution in the trace data
   report          Show performance statistics in the trace data
   live            Do record and replay in a row (default)
   info            Show system and program info in the trace data
   dump            Show low-level trace data
   recv            Save the trace data from network
   graph           Show function call graph in the trace data
   script          Run a script for recorded trace data
   tui             Show text user interface for graph and report

Try `uftrace --help' or `man uftrace [COMMAND]' for more information.

If omitted, it defaults to the live command which is almost same as running record and replay subcommand in a row (but does not record the trace info to files).

For recording, the executable needs to be compiled with the -pg (or -finstrument-functions) option which generates profiling code (calling mcount or __cyg_profile_func_enter/exit) for each function.

Note that, there's an experimental support for dynamic tracing on x86_64 and AArch64(ARM64) which doesn't require such (re-)compilations. Also, recent compilers have some options to help uftrace to reduce tracing overhead with similar way (although it still needs recompilation of your program). Please see dynamic tracing section for more details.

$ uftrace tests/t-abc
# DURATION    TID     FUNCTION
  16.134 us [ 1892] | __monstartup();
 223.736 us [ 1892] | __cxa_atexit();
            [ 1892] | main() {
            [ 1892] |   a() {
            [ 1892] |     b() {
            [ 1892] |       c() {
   2.579 us [ 1892] |         getpid();
   3.739 us [ 1892] |       } /* c */
   4.376 us [ 1892] |     } /* b */
   4.962 us [ 1892] |   } /* a */
   5.769 us [ 1892] | } /* main */

For more analysis, you'd be bet

Uftrace

Install / Use

README

uftrace

Features

How to build and install uftrace

How to use uftrace