Multimonitor
A convinient command line utility to log system and process metrics.
Install / Use
/learn @baryluk/MultimonitorREADME
multimonitor
A convenient command line utility to log system and process metrics.
$ multimonitor --utc_nice --gpu=min --process valley_x64 --process Xorg
# Waiting for process valley_x64
# Waiting for process valley_x64
# Waiting for process valley_x64
# For process name valley_x64 found pids: [1996445]
# For process name Xorg found pids: [2066]
# ticks_per_second: 100
# With interval 200 ms and 100 ticks/s, expect CPU% error of ± 5.0%
# Xorg
# valley_x64 |
# | |
# 1996445 2066
DATETIME-UTC TIME RELTIME GPU% VRAM SCLK CPU% RSS CPU% RSS
2020-12-31T22:28:44.688313 709668.550807 0.200063 0.0% 309.6MiB 386.7MHz 0.00% 77MiB 5.00% 407MiB
2020-12-31T22:28:44.888319 709668.750763 0.400018 0.0% 309.6MiB 386.7MHz 0.00% 77MiB 0.00% 407MiB
2020-12-31T22:28:45.088325 709668.950799 0.600054 0.0% 311.6MiB 326.5MHz 15.00% 82MiB 10.00% 407MiB
2020-12-31T22:28:45.288331 709669.150770 0.800025 0.0% 311.6MiB 326.5MHz 0.00% 82MiB 0.00% 407MiB
2020-12-31T22:28:45.488337 709669.350800 1.000056 0.0% 311.6MiB 326.5MHz 0.00% 82MiB 10.00% 407MiB
2020-12-31T22:28:45.688343 709669.550775 1.200030 0.0% 311.6MiB 326.5MHz 0.00% 82MiB 5.00% 407MiB
2020-12-31T22:28:45.888350 709669.750795 1.400050 0.0% 311.6MiB 326.5MHz 0.00% 82MiB 5.00% 407MiB
2020-12-31T22:28:46.088356 709669.950932 1.600187 0.0% 311.6MiB 326.5MHz 0.00% 82MiB 0.00% 407MiB
2020-12-31T22:28:46.288362 709670.150692 1.799947 0.0% 294.1MiB 588.2MHz 5.01% 82MiB 5.01% 407MiB
2020-12-31T22:28:46.488368 709670.350815 2.000071 0.0% 294.1MiB 588.2MHz 0.00% 82MiB 5.00% 407MiB
2020-12-31T22:28:46.688374 709670.550755 2.200010 0.0% 294.1MiB 588.2MHz 5.00% 82MiB 5.00% 407MiB
2020-12-31T22:28:46.888381 709670.750808 2.400063 0.0% 294.1MiB 588.2MHz 0.00% 82MiB 10.00% 407MiB
2020-12-31T22:28:47.088387 709670.950767 2.600023 0.0% 294.1MiB 588.2MHz 0.00% 82MiB 5.00% 407MiB
2020-12-31T22:28:47.288393 709671.150816 2.800071 0.0% 298.2MiB 724.0MHz 5.00% 82MiB 10.00% 407MiB
2020-12-31T22:28:47.488399 709671.350754 3.000009 0.0% 298.2MiB 724.0MHz 0.00% 82MiB 0.00% 407MiB
2020-12-31T22:28:47.688405 709671.550808 3.200063 0.0% 298.2MiB 724.0MHz 0.00% 82MiB 0.00% 407MiB
2020-12-31T22:28:47.888411 709671.750837 3.400093 0.0% 298.2MiB 724.0MHz 0.00% 82MiB 5.00% 407MiB
2020-12-31T22:28:48.088418 709671.950742 3.599998 0.0% 298.2MiB 724.0MHz 0.00% 82MiB 0.00% 407MiB
Installation
To build multimonitor from source, you will need a D programming language
compiler. GDC, LDC2 and DMD are all supported. On most Linux distributions it is
easiest to install gdc, which is part of gcc, and prepackaged on most
distributions.
After obtaining source code, just execute ./build.sh (you can adjust options in
that script), or use dub to build it.
You should then get a multimonitor binary to use.
Options
Multimonitor - sample information about system and processes.
--sub Launch a single external command, monitor it just like
--pid and finish once all of them finish
--pids List of process pids to monitor
--process List of process names to monitor
--process_map Assign short names to processes, i.e. a=firefox,b=123
--cpu Overall CPU stats, i.e. load, average and max frequency
--loadavg System-wide load average. Avaiable: none, min (1-min avg),
med (+5 min avg), max (+runnables and tasks count,
and forks per second)
--cpu_temp CPU temperature
--sched CPU scheduler details
--vm Virtual memory subsystem
--interrupts Interrupts details
--io System-wide IO details. Available: none, min, max
--net System-wide networking metrics
--gpu Gather GPU stats. Available: none, min, max
--mangohud_fps Gather FPS information for given processes using MangoHud RPC
--exec Run external command with arbitrary output once per sample
--exec_async Run external command with arbitrary output asynchronously
--pipe Run external command and consume output lines as they come
--async_delay_msec Change how often to run --exec_async and --gpu commands.
(default: 200ms)
--wait_for_all Wait until all named processes are up
--find_new_when_dead If the named process is dead, try searching again
--exit_when_dead Stop collecting metrics and exit, when any of requested
pids exits too.
--sum_all_matching For named processes, sum all matching processes metrics
(sum CPU, smart memory sum)
--auto_output Automatically create timestamped output file in current
working directory with data, instead of using standard
output. (default: false)
--interval_msec Target interval for main metric sampling and output.
(default: 200ms)
--duration_sec How long to log. (default: forever)
--time Time mode, one of: relative, boottime, absolute, all.
(default: all)
--utc_nice Show absolute time as ISO 8601 formated date and time in
UTC timezone. Otherwise Unix time is printed.
(default: false)
-H --human_friendly Use human friendly (pretty), but still fixed units
(default: true)
--verbose Show timeing loop debug info
-h --help This help information.
Primary purpose is debugging processes, system load, memory usage, memory leaks, GPU usage, frame rate tests, etc.
Combination of ps, top, iotop, powertop, radeontop, vmstat, free,
mpstat, pidstat, slabtop, cpufreq-info, mangohud and more, all in one.
In some areas the accuracy is significantly better than any of the above tools.
Supported
- Extremely accurate timestamps. Absolute (ISO 8601), monotonic from system start, and relative from tool startup.
- Very accurate repetition rate (usually <10us).
- Syscall delay compensation.
- Asynchronous sampling of expensive statistics.
- Automatic compensation of delays when calculating rates.
- Rich set of available metrics:
- System CPU
- System Memory
- System IO, total and per-device
- CPU temperature, frequency, and scaling governor
- GPU frequency, temperature, load and memory usage
- (multiple) Process CPU usage, thread count
- (multiple) Process memory usage
- (multiple) OpenGL / Vulkan FPS / frame time measurements (using MangoHud RPC system)
- (multiple) Process IO statistics
- (multiple) Process scheduling and IO priority logging
- System networking statistics
- System-wide and per-process context switches and interrupts
- (multiple) Custom asynchronous metrics (using external scripts), like:
- System power / current from PSU / SMBus
- Number of files in a directory
- ZFS statistics
- GPU power save mode
- Window resolution of a benchmarked game
- Many more easy to add on the fly
- Custom events annotation (i.e. using external scripts)
- Ability to sample some metrics at slower rate than main metrics.
- Human and machine friendly output in one format.
- Self documenting output. Clear units.
- Monitor processes by pid, or by name.
- Continue monitoring even if pid dies, or stop. Configurable.
- Ability to sum multiple pids or names. I.e. sum all processes with given name under one column.
- Autostart/prestart - wait for a process by name, and start logging as soon as one is found.
- Fast. 5Hz by default, 100Hz possible.
- Extremely low CPU and memory overhead (<0.5% CPU, <5MiB).
- Integration with Gnuplot for plotting.
- Automatic detection of various failures, like signals, interrupts, processes death, slow syscalls / preemption, system sleep, process SIGSTOP, clock jumps, etc. Automatically output nan, or empty lines when discontinuities are detected, so plotting can use the data correctly.
- Automatically calculate expected error / accuracy, and warn if it is low.
Many metrics support both rates and cumulative figures. Some other measure only "gauges" (i.e. memory usage, GPU load or frequency).
The output format is a simple text format with nicely aligned and annotated columns, but also optimized to be easily parsed by automated tools, most notably Gnuplot. Relative and absolute timestamps allows correlating with other tools and events, as well overlying multiple runs for comparison. Most of the data in various columns use fixed non-configurable formats. This is mostly done to speed-up processing and output, reduce memory allocations, and make it less likely for user to mess things up. It also means the logs produced now, will have exactly same format and units as the ones produced years from now, no matter
