Tracexec
Tracer for execve{,at} and pre-exec behavior, launcher for debuggers.
Install / Use
/learn @kxxt/TracexecREADME
tracexec
A small utility for tracing execve{,at} and pre-exec behavior.
tracexec helps you to figure out what and how programs get executed when you execute a command.
It's useful for debugging build systems, understanding what shell scripts actually do, figuring out what programs does a proprietary software run, etc.
Showcases
Perfetto Trace Export
tracexec supports exporting exec traces to perfetto trace format, which could be viewed in the Perfetto UI. The trace follows a tree format in the UI, where processes resulting from successful execs are represented as slices and exec failures are represented as instant events.
The following video shows analyzing the build process of tracexec with itself:
https://github.com/user-attachments/assets/fc825d55-eb2f-43cd-b454-a11b5f9b44bc
The shape of the traces in the Perfetto UI could give you a rough idea of how parallel the build is at process-level. The trace tree and details of slices enable identification of bottlenecks, troubleshooting, and a deep understanding of how the build works.
Start collecting a perfetto trace with the following command:
tracexec collect --format=perfetto -o out.pftrace -- cmd
TUI mode with pseudo terminal
In TUI mode with a pseudo terminal, you can view the details of exec events and interact with the processes within the pseudo terminal at ease.
Tracing setuid binaries
With root privileges, you can also trace setuid binaries and see how they work. But do note that this is not compatible with seccomp-bpf optimization so it is much less performant. You can use eBPF mode which is more performant in such scenarios.
sudo tracexec --user $(whoami) tui -t -- sudo ls
Nested setuid binary tracing is also possible: A real world use case is to trace extra-x86_64-build(Arch Linux's build tool that requires sudo):
In this real world example, we can easily see that _FORTIFY_SOURCE is redefined from 2 to 3, which lead to a compiler error.
Use tracexec as a debugger launcher
tracexec can also be used as a debugger launcher to make debugging programs easier. For example, it's not trivial or convenient to debug a program executed by a shell/python script(which can use pipes as stdio for the program). The following video shows how to use tracexec to launch gdb to detach two simple programs piped together by a shell script.
https://github.com/kxxt/tracexec/assets/18085551/72c755a5-0f2f-4bf9-beb9-98c8d6b5e5fd
Please read the gdb-launcher example for more details.
eBPF mode
The eBPF mode is currently experimental. The minimum supported kernel version is 5.17.
The following examples shows how to use eBPF in TUI mode.
The eBPF command also supports regular log and collect subcommands.
System-wide Exec Tracing
sudo -E tracexec ebpf tui
Follow Fork mode with eBPF
sudo -E tracexec --user $(whoami) ebpf tui -t -- bash
Log mode
In log mode, by default, tracexec will print filename, argv and the diff of the environment variables and file descriptors.
example: tracexec log -- bash (In an interactive bash shell)
Reconstruct the command line with --show-cmdline
$ tracexec log --show-cmdline -- <command>
# example:
$ tracexec log --show-cmdline -- firefox
Try to reproduce stdio in the reconstructed command line
--stdio-in-cmdline and --fd-in-cmdline can be used to reproduce(hopefully) the stdio used by a process.
But do note that the result might be inaccurate when pipes, sockets, etc are involved.
tracexec log --show-cmdline --stdio-in-cmdline -- bash
Show the interpreter indicated by shebang with --show-interpreter
And show the cwd with --show-cwd.
$ tracexec log --show-interpreter --show-cwd -- <command>
# example: Running Arch Linux makepkg
$ tracexec log --show-interpreter --show-cwd -- makepkg -f
Usage
General CLI help:
Core crate of tracexec [Internal implementation! DO NOT DEPEND ON!]
Usage: tracexec [OPTIONS] <COMMAND>
Commands:
log Run tracexec in logging mode
tui Run tracexec in TUI mode, stdin/out/err are redirected to /dev/null by default
generate-completions Generate shell completions for tracexec
collect Collect exec events and export them
ebpf Experimental ebpf mode
help Print this message or the help of the given subcommand(s)
Options:
--color <COLOR> Control whether colored output is enabled. This flag has no effect on TUI mode. [default: auto] [possible values: auto, always, never]
-C, --cwd <CWD> Change current directory to this path before doing anything
-P, --profile <PROFILE> Load profile from this path
--no-profile Do not load profiles
-u, --user <USER> Run as user. This option is only available when running tracexec as root
-h, --help Print help
-V, --version Print version
TUI Mode:
Run tracexec in TUI mode, stdin/out/err are redirected to /dev/null by default
Usage: tracexec tui [OPTIONS] -- <CMD>...
Arguments:
<CMD>... command to be executed
Options:
--successful-only
Only show successful calls
--fd-in-cmdline
[Experimental] Try to reproduce file descriptors in commandline. This might result in an unexecutable cmdline if pipes, sockets, etc. are involved.
--stdio-in-cmdline
[Experimental] Try to reproduce stdio in commandline. This might result in an unexecutable cmdline if pipes, sockets, etc. are involved.
--resolve-proc-self-exe
Resolve /proc/self/exe symlink
--no-resolve-proc-self-exe
Do not resolve /proc/self/exe symlink
--hide-cloexec-fds
Hide CLOEXEC fds
--no-hide-cloexec-fds
Do not hide CLOEXEC fds
--timestamp
Show timestamp information
--no-timestamp
Do not show timestamp information
--inline-timestamp-format <INLINE_TIMESTAMP_FORMAT>
Set the format of inline timestamp. See https://docs.rs/chrono/latest/chrono/format/strftime/index.html for available options.
--seccomp-bpf <SECCOMP_BPF>
Controls whether to enable seccomp-bpf optimization, which greatly improves performance [default: auto] [possible values: auto, on, off]
--polling-interval <POLLING_INTERVAL>
Polling interval, in microseconds. -1(default) disables polling.
--show-all-events
Set the default filter to show all events. This option can be used in combination with --filter-exclude to exclude some unwanted events.
--filter <FILTER>
Set the default filter for events. [default: warning,error,exec,tracee-exit]
--filter-include <FILTER_INCLUDE>
Aside from the default filter, also include the events specified here. [default: <empty>]
--filter-exclude <FILTER_EXCLUDE>
Exclude the events specified here from the default filter. [default: <empty>]
-t, --tty
Allocate a pseudo terminal and show it alongside the TUI
-f, --follow
Keep the event list scrolled to the bottom
--terminate-on-exit
Instead of waiting for the root child to exit, terminate when the TUI exits
--kill-on-exit
Instead of waiting for the root child to exit, kill when the TUI exits
-A, --active-pane <ACTIVE_PANE>
Set the default active pane to use when TUI launches [possible values: terminal, events]
-L, --layout <LAYOUT>
Set the layout of the TUI when it launches [possible values: horizontal, vertical]
-F, --frame-rate <FRAME_RATE>
Set the frame rate of the TUI (60 by default)
-m, --max-events <MAX_EVENTS>
Max number of events to keep in TUI (0=unlimited)
--scrollback-lines <SCROLLBACK_LINES>
Number of scrollback lines to keep in the pseudo terminal (1000 by default)
--theme <THEME_FILE>
Path to a theme file to use for the TUI.
-D, --default-external-command <DEFAULT_EXTERNAL_COMMAND>
Set the default external command to run when using "Detach, Stop and Run Command" feature in Hit Manager
-b, --add-breakpoint <BREAKPOINTS>
Add a new breakpoint to the tracer. This option can be used multiple times. The format is <syscall-stop>:<pattern-type>:<pattern>, where syscall-stop can be sysenter or sysexit, pattern-type can be argv-regex, in-filename or exact-filename. For example, sysexit:in-filename:/bash
-h, --help
Print help
Log Mode:
Run tracexec in logging mode
Usage: tracexec log [OPTIONS] -- <CMD>...
Arguments:
<CMD>... command to be executed
Options:
--more-colors
More colors
--less-colors
Less colors
--show-cmdline
P
