Shournal

A (file-) journal for your shell

Log shell-commands and used files. Snapshot executed scripts. Fully automatic. <br> There are two kinds of people: those who backup, and those who have never lost their data. <br>

$ SHOURNAL_ENABLE
$ cat demo.sh
#!/usr/bin/env bash
echo hi | tee out.log
$ ./demo.sh
hi
$ shournal -q --wfile out.log
cmd-id 2 $?=0 2022-11-08 08:46 $ ./demo.sh
Working directory: /home/user
  1 written file:
     /home/user/out.log (3 bytes) Hash: 15349503233279147316
  1 read file:
     /home/user/demo.sh (42 bytes) Hash: 13559791986335963073 id 1
          #!/usr/bin/env bash
          echo hi | tee out.log

shournal records that out.log was written by the command ./demo.sh and created a backup of the script demo.sh because it was read by the bash interpreter.

shournal does not guess the files - it asks the Linux kernel. It's fast enough, see Overhead.

After installation and easy setup of the shell-integration the following questions may be answered within seconds:

• What files were modified, read or executed by a command? Or reverse: What shell-commands were used to create/modify or read from a certain file?
• You executed a script. What was the script-content by the time it was called?
• The command read a config-file - which one, and what was in it?
• The command ran for a long time - can a re-execution be avoided (s. --stat)?
• What other commands were executed during the same shell-session?
• What about working directory, command start- and end-time or the exit status ($?) ?

Besides output on the command-line in a human-readable format (or JSON) you can export (parts of) your command-history into a standalone html-file where it is displayed in an interactive time-line-plot. Further miscellaneous statistics are displayed in bar-plots, e.g. the commands with most file-modifications.

Using the external software shournal-to-snakemake an observed shell-command-series can be directly transformed into rules for the Snakemake workflow engine, a tool to create reproducible and scalable data analyses.

shournal runs on GNU/Linux or Microsoft Windows via the Windows Subsystem for Linux (WSL) using its fanotify edition.

For a more formal description please also check out our paper <br> Kirchner, T., Riege, K. & Hoffmann, S. Bashing irreproducibility with shournal. Sci Rep 14, 4872 (2024). https://doi.org/10.1038/s41598-024-53811-9

<br>

Examples

Please note: below examples make use of the shell-integration. <br>

• Create a file and ask shournal, how it came to be:

  $ SHOURNAL_ENABLE # monitor all commands using the shell-integration
  $ echo hi > foo
  $ shournal --query --wfile foo
  cmd-id 1 $?=0 2019-05-14 10:19 $ echo hi > foo
    1 written file:
       /home/user/foo (3 bytes) Hash: 15349503233279147316
  
  

• shournal can be configured, to store specific read files, like shell-scripts, within it's database. Sometimes old script versions are of interest. Query by read filename and optionally restore the files with --restore-rfiles:

  $ shournal -q --rname demo.sh
  cmd-id 34 $?=0 2022-04-21 15:15 $ ./demo.sh
    1 read file:
       /home/user/demo.sh (34 bytes) Hash: 16696055267278105544 id 3
            #!/usr/bin/env bash
            echo version1
  cmd-id 35 $?=0 2022-04-21 15:15 $ ./demo.sh
    1 read file:
       /home/user/demo.sh (34 bytes) Hash: 17683376525180966954 id 4
            #!/usr/bin/env bash
            echo version2
  $ shournal -q --rname demo.sh --restore-rfiles # restore read files
  ...
  2 file(s) restored at /tmp/shournal-restore-user
  

• List all commands which contained the string demo (<kbd>%</kbd> is wildcard):

  $ shournal -q -cmdtxt %demo%
  cmd-id 1 $?=0 2022-04-20 15:46 $ cat demo.sh
  ...
  cmd-id 2 $?=0 2022-04-20 15:46 $ ./demo.sh
  ...
  

• Are input files up to date, such that re-execution of the command can be avoided? Add --stat to the query, reporting current file statuses as U (up to date), M (modified), N (not exist) ERROR (in case of an error) or NA (not queried, only using json).

  $ cat foo > bar
  $ shournal -q -wf bar --stat
  cmd-id 1 $?=0 2025-02-10 11:38-11:38 $ cat foo > bar
  ...
  1 written file:
     /home/tycho/bar (3 bytes) Hash: 15349503233279147316 U
  1 read file:
     /home/tycho/foo (3 bytes) Hash: 15349503233279147316 id 404002 U
  

  To query only for changed input files, execute

  shournal -q -wf bar --stat --output-format json | grep -F 'COMMAND:' | \
    sed -n 's/COMMAND://p' | \
    jq -r '.fileReadEvents | .[] | .status + " " + .path' | grep -v ^U
  

• What commands were executed at the current working directory?

  shournal --query -cwd "$PWD"
  

• What commands were executed within a specific shell-session? The uuid can be taken from the command output of a previous query.

  shournal --query --shell-session-id $uuid
  

• Find out the value of a variable. For instance, the command echo $foo > bar was executed in the shell-session with id puLvkEizEe6CgvXjQlmnIQ==. If foo was set within that shell session, its value can often be retrieved by

  shournal -q -sid puLvkEizEe6CgvXjQlmnIQ==  | fgrep 'foo='
  

• For the full list of query-options, please enter

  shournal --query --help
  

Instead of printing the --query-results to terminal, you can also create fancy html-plots, by appending --output-format html -o out.html. Use an ordinary web-browser for display.

Installation

Binary releases

For Debian/Ubuntu-based distributions .deb-packages are available on the release-page. Three different editions are provided for different use-cases: most users will want to install shournal on a real host (or virtual machine) and shournal-docker inside Docker (or another container platform). shournal-fanotify does not contain the kernel backend and is targeted at institutions where the usage of out-of-tree kernel-modules is discouraged. <br> Only LTS-releases are officially supported, the packages are known to work from Debian 10 (Buster) and Ubuntu 18.04 (Bionic) onwards. Before installing shournal including its kernel backend, make sure, the kernel headers are installed: <br> Ubuntu: apt install linux-headers-generic <br> Ubuntu with HWE: apt install linux-headers-generic-hwe-$(lsb_release -rs) <br> Debian: apt install linux-headers-$(dpkg --print-architecture) <br> Install deb-packages as usual, e.g. <br> sudo apt install ./shournal_2.2_amd64.deb <br> To enable the shell-integration:

• for bash: put the following to the end of your ~/.bashrc <br> source /usr/share/shournal/SOURCE_ME.bash <br>
• for zsh: put the following to the end of your ~/.zshrc <br> source /usr/share/shournal/SOURCE_ME.zsh <br> and run SHOURNAL_ENABLE afterwards.

For any Linux, a flat binary is available on the release-page to be used without installation:

tar -xf shournal-fanotify*.tar.xz
cd shournal-fanotify/
sudo groupadd shournalmsenter
sudo chown root shournal-run-fanotify && sudo chmod u+s shournal-run-fanotify
./shournal-run-fanotify -e echo Hello World

# Source shournal's shell integration from bashrc/zshrc, e.g.
# echo "source '$PWD/SOURCE_ME.bash'" >> ~/.bashrc
# echo "source '$PWD/SOURCE_ME.zsh'" >> ~/.zshrc
# Enable with: SHOURNAL_ENABLE.

An update of shournal should be performed after all users have logged out, because the shell integrations need to be resourced. Further in case of the kernel module backend unloading the old version stops all running observations.

After installation: Depending on your distribution, additional steps might be necessary to enable the (recommended) uuidd-daemon. If systemd is in use, one may need to:

systemctl enable uuidd
systemctl start uuidd

Add yourself or other users to the group shournalk: <br> sudo adduser $USER shournalk (relogin to take affect). <br> You may override this group:

mkdir -p /etc/shournal.d/
echo GROUPNAME > /etc/shournal.d/kgroup

replacing GROUPNAME with the value of your choice. This rule takes into effect the next time shournal's kernel module is loaded ( so call e.g. modprobe -r shournalk; modprobe shournalk or reboot).

More details and advanced options (logging commands executed via ssh) can be found here.

Compile and install from source

Please refer to the instructions found within the compile-README.

FAQ

• Does shournal track file rename/move operations? <br> No, but most often it should not be a problem. Using the --wfile commandline-query-option, shournal finds the stored command by content (size, hash) and mtime, not by its name. For the name, --wname can be used. More concrete:

  shournal --exec sh -c 'echo foo > bar; mv bar bar_old'
  

  Querying for bar_old by content (--wfile-option) yields exactly the given command, however, --wname bar_old does not work (--wname bar of course works). To use the bar_old file name (and not content) as basis for a successful query, in this case --command-text -like '%bar_old%' can be used.
• What happens to an appended file? <br> How to get a "modification history"? Please read above rename/move-text first. Appending to a file is currently handled as if a new one was created - only the last command, which modified a given file can be found with good