rush -- a cross-platform command-line tool for executing jobs in parallel

rush is a tool similar to GNU parallel and gargs. rush borrows some idea from them and has some unique features, e.g., supporting custom defined variables, resuming multi-line commands, more advanced embeded replacement strings.

These features make rush suitable for easily and flexibly parallelizing complex workflows in fields like Bioinformatics (see examples).

Features
Performance
Installation
Usage
Examples
Special Cases
Contributors
Acknowledgements
Contact
License

Features

Major:

Supporting Linux, OS X and Windows (not CygWin)!
Avoid mixed line from multiple processes without loss of performance, e.g. the first half of a line is from one process and the last half of the line is from another process. (--line-buffer in GNU parallel)
Timeout (-t). (--timeout in GNU parallel)
Retry (-r). (--retry-failed --joblog in GNU parallel)
Safe exit after capturing Ctrl-C (not perfect, you may stop it by typing ctrl-c or closing terminal)
Continue (-c). (--resume --joblog in GNU parallel, <s/>sut it does not support multi-line commands, which are common in workflow</s>)
awk -v like custom defined variables (-v). (Using Shell variable in GNU parallel)
Keeping output in order of input (-k). (Same -k/--keep-order in GNU parallel)
Exit on first error(s) (-e). (not perfect, you may stop it by typing ctrl-c or closing terminal) (--halt 2 in GNU parallel)
Settable record delimiter (-D, default \n). (--recstart and --recend in GNU parallel)
Settable records sending to every command (-n, default 1). (-n/--max-args in GNU parallel)
Settable field delimiter (-d, default \s+). (Same -d/--delimiter in GNU parallel)
Practical replacement strings (like GNU parallel):
- {{}}, {} itself
- {{1,}}, {1,}.
- {#}, job ID. (Same in GNU parallel)
- {}, full data. (Same in GNU parallel)
- {n}, nth field in delimiter-delimited data. (Same in GNU parallel)
- Directory and file
  - {/}, dirname. ({//} in GNU parallel)
  - {%}, basename. ({/} in GNU parallel)
  - {.}, remove the last file extension. (Same in GNU parallel)
  - {:}, remove all file extensions (Not directly supported in GNU parallel)
  - {^suffix}, remove suffix (Not directly supported in GNU parallel)
  - {@regexp}, capture submatch using regular expression (Not directly supported in GNU parallel). There's a limitation here: curly brackets can't be used in the regular expression.
- Combinations
  - {%.}, {%:}, basename without extension
  - {2.}, {2/}, {2%.}, manipulate nth field
  - {file:}, {file:^_1}, remove all extensions of a preset variable (see below)
Preset variable (macro), e.g., rush -v p={^suffix} 'echo {p}_new_suffix', where {p} is replaced with {^suffix}. (Using Shell variable in GNU parallel)

Minor:

Dry run (--dry-run). (Same in GNU parallel)
Trim input data (--trim). (Same in GNU parallel)
Verbose output (--verbose). (Same in GNU parallel)

Differences between rush and GNU parallel on GNU parallel site.

Performance

Performance of rush is similar to gargs, and they are both slightly faster than parallel (Perl) and both slower than Rust parallel (discussion).

Note that speed is not the #.1 target, especially for processes that last long.

Installation

rush is implemented in Go programming language, executable binary files for most popular operating systems are freely available in release page.

Method 0: Conda

Install conda, then run

conda install -c conda-forge rush

Or use mamba, which is faster.

mamba install -c conda-forge rush

Method 1: Download binaries

rush v0.8.0

Tip: run rush -V to check update !!!

OS |Arch |File, |Download Count :------|:---------|:-------------------------------------------------------------------------------------------------------------------------------|:----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Linux |32-bit |rush_linux_386.tar.gz | Linux |64-bit|rush_linux_amd64.tar.gz | Linux |arm64 |rush_linux_arm64.tar.gz | OS X |64-bit|rush_darwin_amd64.tar.gz | OS X |arm64 |rush_darwin_arm64.tar.gz | Windows|32-bit |rush_windows_386.exe.tar.gz | Windows|64-bit|rush_windows_amd64.exe.tar.gz | FreeBSD|32-bit |rush_freebsd_386.tar.gz | FreeBSD|64-bit|rush_freebsd_amd64.tar.gz |