recur

recur is a command-line tool that runs a single command repeatedly until it succeeds or no more attempts are left. It implements optional exponential backoff with configurable jitter. It lets you write the success condition in Starlark.

Installation

Prebuilt binaries

Prebuilt binaries for FreeBSD (amd64), Linux (aarch64, riscv64, x86_64), macOS (arm64, x86_64), NetBSD (amd64), OpenBSD (amd64), and Windows (amd64, arm64, x86) are attached to releases.

Homebrew

You can install recur from Homebrew on macOS and Linux:

brew install recur

Go

Install Go, then run:

go install dbohdan.com/recur/v3@latest

Build requirements

Go 1.22
Task (go-task) 3.28

Usage

Command-line interface

Usage: recur [-h] [-V] [-a <attempts>] [-b <backoff>] [-c <condition>] [-d
<delay>] [-E] [-F] [-I] [-j <jitter>] [-m <max-delay>] [-O] [-R <path>] [-r
<reset-time>] [-s <seed>] [-t <timeout>] [-u] [-v] [--] <command> [<arg> ...]

Retry a command with exponential backoff and jitter.

Arguments:
  <command>
          Command to run

  [<arg> ...]
          Arguments to the command

Options:
  -h, --help
          Print this help message and exit

  -V, --version
          Print version number and exit

  -a, --attempts 10
          Maximum number of attempts (negative for unlimited)

  -b, --backoff 0
          Base for exponential backoff (duration)

  -c, --condition 'code == 0'
          Success condition (Starlark expression)

  -d, --delay 0
          Constant delay (duration)

  -E, --hold-stderr
          Buffer standard error for each attempt and only print it on success

  -F, --fib
          Add Fibonacci backoff

  -I, --replay-stdin
          Read standard input until EOF at the start and replay it on each
attempt

  -j, --jitter '0,0'
          Additional random delay (maximum duration or 'min,max' duration)

  -m, --max-delay 1h
          Maximum allowed sum of constant delay, exponential backoff, and
Fibonacci backoff (duration)

  -O, --hold-stdout
          Buffer standard output for each attempt and only print it on success

  -R, --report ''
          Report output (file path, '-' for stderr, or '' to disable; prefix
with 'json:' or 'text:' to override the format)

  -r, --reset -1s
          Minimum attempt time that resets exponential and Fibonacci backoff
(duration; negative for no reset)

  -s, --seed 0
          Random seed for jitter (0 for automatic)

  -t, --timeout -1s
          Timeout for each attempt (duration; negative for no timeout)

  -u, --unlimited, -f, --forever
          Unlimited attempts

  -v, --verbose
          Increase verbosity (up to 4 times)

Duration arguments take Go duration strings; for example, 0, 100ms, 2.5s, 0.5m, or 1h. The value of -j/--jitter must be either a single duration or two durations joined with a comma, like 1s,2s or 500ms, 0.5m.

If the maximum delay (-m/--max-delay) is shorter than the constant delay (-d/--delay), the constant delay will automatically increase the maximum delay to match it. Use -m/--max-delay after -d/--delay if you want a shorter maximum delay.

The following recur options run the command foo --config bar.cfg indefinitely. Every time foo exits, there is a delay that grows exponentially from two seconds to a minute. The delay resets back to two seconds if the command runs for at least five minutes.

recur --backoff 2s --condition False --forever --max-delay 1m --reset 5m foo --config bar.cfg

recur exits with the last command's exit code unless the user overrides this in the condition. When the command is not found during the last attempt, recur exits with code 127. recur exits with code 124 on timeout and 255 on internal error.

[!NOTE] The option -f/--forever is deprecated because of the ambiguity of "forever". (recur --forever still tries until success unless you change the success condition.) -f/--forever will remain available. It is not recommended for new usage: use -u/--unlimited instead.

Standard input

By default, the command run by recur inherits its standard input. This means that if standard input is a terminal, every attempt can read interactively from the terminal. If standard input is a pipe or redirected file, the data is consumed on the first attempt; later attempts see an immediate EOF.

To feed the command the same data on each attempt, use the -I/--replay-stdin option. With this option, recur reads its entire stdin into memory and replays it on each attempt.

$ echo hi | recur -a 3 -c False cat
hi
recur [00:00:00.0]: maximum 3 attempts reached

$ echo hi | recur -a 3 -c False -I cat
hi
hi
hi
recur [00:00:00.0]: maximum 3 attempts reached

Because the data is buffered in memory, --replay-stdin is not recommended for very large inputs.

Standard output and standard error

The command's standard output and standard error are passed through to recur's standard output and standard error by default. To buffer standard output and only print it on success, use -O/--hold-stdout. With this option, recur buffers the command's standard output and only prints it if the success condition is met or the condition expression calls stdout.flush().

$ recur -c 'attempt == 3' sh -c 'echo "$RECUR_ATTEMPT"'
1
2
3

$ recur -c 'attempt == 3' -O sh -c 'echo "$RECUR_ATTEMPT"'
3

$ recur -c 'stdout.flush() or attempt == 3' -O sh -c 'echo "$RECUR_ATTEMPT"'
1
2
3

The -E/--hold-stderr option and stderr.flush() method work similarly for standard error.

Because the data is buffered in memory, --hold-stdout and --hold-stderr are not recommended for commands that produce very large output.

Regular-expression matching

You can match regular expressions against recur's input and the command's output in your success condition using methods on the built-in objects stdin, stdout, and stderr:

stdin.search() — matches against standard input (requires -I/--replay-stdin)
stdout.search() — matches against standard output (requires -O/--hold-stdout)
stderr.search() — matches against standard error (requires -E/--hold-stderr)

These methods use Go regular expressions with the RE2 syntax.

The stdin, stdout, and stderr objects are None without their respective command-line option (-I/--replay-stdin, -O/--hold-stdout, or -E/--hold-stderr). Calling methods on None will result in an error.

Standard input, standard output, and standard error are not available directly as Starlark strings to reduce memory usage. The methods provide the only way to access them in conditions.

Matching standard input

The following example waits for the input to contain done on a line after status::

$ printf 'Status:\nDONE\n' | recur \
    --condition 'stdin.search(r"(?im)status:\s*done$")' \
    --replay-stdin \
    cat \
    ;
Status:
DONE

r"..." disables the processing of backslash escapes in the string. It is necessary because \s is not a valid backslash escape. The regular expression (?im)status:\s*done$ uses RE2 inline flags:

i for case-insensitive matching
m for multiline mode ($ matches the end of each line)

The condition evaluates to true when stdin.search() finds a match (returns a non-empty list) and false when no match is found (returns None).

Standard input is perhaps of limited use for retrying a command because it is read once and never changes. However, it can be used to exit early.

Matching standard output and standard error

This example extracts a status value from the command's output and validates it:

$ recur \
    --condition 'stdout.search(r"(?i)status:([^\n]+)", group=1, default="fail").strip().lower() != "fail"' \
    --hold-stdout \
    echo 'Status: OK' \
    ;
Status: OK

In this condition:

stdout.search(r"(?i)status:([^\n]+)", group=1, default="fail") searches for "status:" followed by text on the same line
- r"..." disables the processing of backslash escapes like \n in the string
- group=1 extracts just the captured text (for example, " OK" with a leading space)
- default="fail" returns "fail" if no match is found
.strip().lower() normalizes the extracted value

Matching against standard error with stderr.search works similarly.

Environment variables

recur sets the environment variable RECUR_ATTEMPT to the current attempt number so the command can access it. recur also sets RECUR_MAX_ATTEMPTS to the value of the -a/--attempts option and RECUR_ATTEMPT_SINCE_RESET to the attempt number since exponential and Fibonacci backoff were reset.

The following command succeeds on the last attempt:

$ recur sh -c 'echo "Attempt $RECUR_ATTEMPT of $RECUR_MAX_ATTEMPTS"; exit $((RECUR_MAX_ATTEMPTS - RECUR_ATTEMPT))'
Attempt 1 of 10
Attempt 2 of 10
Attempt 3 of 10
Attempt 4 of 10
Attempt 5 of 10
Attempt 6 of 10
Attempt 7 of 10
Attempt 8 of 10
Attempt 9 of 10
Attempt 10 of 10

Conditions

recur supports a limited form of scripting. You can define the success condition using an expression in [Starlark](https://laurent.le-brun.e

Recur

Install / Use

README