concur

A replacement for the parts of GNU Parallel that I like.

I like what parallel can do. As a network engineer, over the years I've had to do things like copy a firmware update file to a tens or low hundreds of routers, or ping a bunch of hosts to see which ones were up, and parallel made those jobs really easy.

You know what I don't like about parallel? It's got defaults that don't work for me (one job per core when I'm blocking on network I/O is hugely inefficient). Its CLI is convoluted (yes, it's a hard problem to solve in general, but still). It's got a 120-page manual or a 20-minute training video to teach you how to use it. It's written in Perl. It has this weird thing where it can't do CSV right out of the box and you have to do manual CPAN stuff. Ain't nobody got time for any of that. And worst of all, it has that weird gold-digging 'I agree under penalty of death to cite parallel in any academic work'. I get that it's reasonable to ask for credit for writing free software, but if everyone did that then the whole open source industry would drown itself in a pool of paperwork.

So I give you concur. I'm never going to claim it's as fully featured as parallel but it does the bits I need, it's a few hundred lines of go, it has sensible defaults for things which aren't compute-bound, it has easy to read json output. It can flag jobs which exit with a return code other than zero. It can run all your jobs, or it can stop when the first one teminates. It's got an easy syntax for subbing in the thing you're iterating over (each entry in a list of hosts, for example).

It doesn't do as many clever things as parallel but it does the subset that I want and it does them well and it does them using go's built-in concurrency. Thus, concur (which as a verb is a synonym for parallel).

This project is very much under active development so the screen scrapes here may not match what's in the latest code but it'll be close.

TL;DR

concur "dig @{{1}} ocw.mit.edu" 1.1.1.1 9.9.9.9 8.8.8.8 94.140.14.14 208.67.222.222 runs five digs in parallel and spits out some JSON about the results. Try it! You'll like it!

BIG IMPORTANT NOTE

This is a work in progress written by a guy who doesn't write code for a living. I believe this is pretty solid at its core and does what it says on the tin but it might have weird corner cases. It is by no means idiomatic go, although I tried. PRs and comments welcome. It doesn't have many tests and could use a restructure. Maybe someday.

Pre-built binary

You can grab a pre-built binary from the latest release.

building

You may want to build from scratch. I use just to manage building and testing so everything is in a justfile and done with goreleaser so it gets a little complicated. Check it out.

You can do that too, or you can just rungo build.

nix

A nix flake has been provided, run nix shell github:ewosborne/concur / nix profile install github:ewosborne/concur or clone and run nix build in git repo.

usage

Run commands concurrently

Usage:
  concur <command string> <list of hosts> [flags]

Flags:
      --any                  Return any (the first) job with exit code of zero
  -c, --concurrent string    Number of concurrent jobs (0 = no limit), 'cpu' or '1x' = one job per cpu core, '2x' = two jobs per cpu core (default "128")
      --first                First commanjobd regardless of exit code
      --flag-errors          Print a message to stderr for all completed jobs with an exit code other than zero
  -h, --help                 help for concur
  -j, --job-timeout string   Per-job timeout in time.Duration format (0 default, must be <= global timeout) (default "0")
  -l, --log string           Enable debug mode (one of d, i, w, e, or q for quiet). (default "e")
  -p, --pbar                 Display a progress bar which ticks up once per completed job
  -t, --timeout string       Global timeout in time.Duration format (0 default for no timeout) (default "0")
      --token string         Token to match for replacement (default "{{1}}")
  -v, --version              version for concur

Concur takes some mandatory arguments. The first is the command you wish to run concurrently. All subsequent arguments are whatever it is you want to change about what you run in parallel. It queues up all jobs and runs them on a number of worker goroutines until all jobs have finished - you control this number with -c/--concurrent. There is no timeout but you can add one with --timeout. There are also a couple of options to return before all jobs are done - --any and --first, see below. And --pbar displays a progress bar which counts off as jobs finish. I find this useful for longer running jobs like scping a file to a bunch of hosts.

concur also takes targets via stdin:

eric@Erics-MacBook-Air concur % ls /tmp/*foo
zsh: no matches found: /tmp/*foo
eric@Erics-MacBook-Air concur % echo "foo bar baz" | concur "touch /tmp/{{1}}.foo"
{
 "command": [
...
...
...
eric@Erics-MacBook-Air concur % ls /tmp/*foo
/tmp/bar.foo  /tmp/baz.foo  /tmp/foo.foo

Example

Here's an example which pings three different hosts:

concur "ping -c 1 {{1}}" www.mit.edu www.ucla.edu www.slashdot.org

This runs ping -c 1 for each of those three web sites, replacing {{1}} with each in turn. It returns JSON to stdout, suitable for piping into jq or gron. It sometimes complains but all complains go to stderr so even if there are errors you still get clean JSON on stdout.

There are two top-level keys in that JSON, command and info. info doesn't have much in it now but SystemRuntime tells you how long it took to finish everything.

command is where most of the fun is. Take a look at the example below. It's a list of information about each command which was run, sorted by runtime, fastest first. This means that concur "ping -c 1 {{1}}" www.mit.edu www.ucla.edu www.slashdot.org | jq '.command[0] | '.arg' + " " + .runtime' always gives you the host which responded first, but -- spoiler alert! -- there's a flag for that. concur "ping -c 1 {{1}}" www.mit.edu www.ucla.edu www.slashdot.org --any gives you the same thing, albeit the full JSON output, not just the runtime and argument name.

concur "ping -c 1 {{1}}" www.mit.edu www.ucla.edu www.slashdot.org | jq '.command[0]

Some things to note are: stdout is presented as an array, with each line of stdout a separate array element. stderr is stored the same way. The return code from the application is in returncode and the runtime for the command is runtime. There's also a jobstatus code, one of

const (
	TBD JobStatus = iota
	Started
	Running
	Finished
	Errored
)

Return code is only valid if jobstatus is Finished or Errored.

Here's the full JSON output from that sample ping.

{
 "command": [
  {
   "id": 0,
   "jobstatus": "Finished",
   "original": "ping -c 1 {{1}}",
   "substituted": "ping -c 1 www.mit.edu",
   "arg": "www.mit.edu",
   "stdout": [
    "PING e9566.dscb.akamaiedge.net (23.57.130.30): 56 data bytes",
    "64 bytes from 23.57.130.30: icmp_seq=0 ttl=59 time=7.901 ms",
    "",
    "--- e9566.dscb.akamaiedge.net ping statistics ---",
    "1 packets transmitted, 1 packets received, 0.0% packet loss",
    "round-trip min/avg/max/stddev = 7.901/7.901/7.901/0.000 ms",
    ""
   ],
   "stderr": [
    ""
   ],
   "starttime": "2024-12-27T16:34:50.917211-05:00",
   "endtime": "2024-12-27T16:34:50.937883-05:00",
   "runtime": "20.671667ms",
   "returncode": 0
  },
  {
   "id": 2,
   "jobstatus": "Finished",
   "original": "ping -c 1 {{1}}",
   "substituted": "ping -c 1 www.slashdot.org",
   "arg": "www.slashdot.org",
   "stdout": [
    "PING www.slashdot.org.cdn.cloudflare.net (104.18.4.215): 56 data bytes",
    "64 bytes from 104.18.4.215: icmp_seq=0 ttl=60 time=8.277 ms",
    "",
    "--- www.slashdot.org.cdn.cloudflare.net ping statistics ---",
    "1 packets transmitted, 1 packets received, 0.0% packet loss",
    "round-trip min/avg/max/stddev = 8.277/8.277/8.277/0.000 ms",
    ""
   ],
   "stderr": [
    ""
   ],
   "starttime": "2024-12-27T16:34:50.917172-05:00",
   "endtime": "2024-12-27T16:34:50.937845-05:00",
   "runtime": "20.672417ms",
   "returncode": 0
  },
  {
   "id": 1,
   "jobstatus": "Finished",
   "original": "ping -c 1 {{1}}",
   "substituted": "ping -c 1 www.ucla.edu",
   "arg": "www.ucla.edu",
   "stdout": [
    "PING d1zev4mn1zpfbc.cloudfront.net (18.161.21.115): 56 data bytes",
    "64 bytes from 18.161.21.115: icmp_seq=0 ttl=250 time=8.040 ms",
    "",
    "--- d1zev4mn1zpfbc.cloudfront.net ping statistics ---",
    "1 packets transmitted, 1 packets received, 0.0% packet loss",
    "round-trip min/avg/max/stddev = 8.040/8.040/8.040/nan ms",
    ""
   ],
   "stderr": [
    ""
   ],
   "starttime": "2024-12-27T16:34:50.917174-05:00",
   "endtime": "2024-12-27T16:34:50.93788-05:00",
   "runtime": "20.705917ms",
   "returncode": 0
  }
 ],
 "info": {
  "CoroutineLimit": 128,
  "SystemRuntime": "20ms"
 }
}

Flags

concur has a number of useful flags:

      --any                  Return any (the first) job with exit code of zero
  -c, --concurrent string    Number of concurrent jobs (0 = no limit), 'cpu' or '1x' = one job per cpu core, '2x' = two jobs per cpu core (default "128")
      --first                First commanjobd regardless of exit code
      --flag-errors          Print a message to stderr for all completed jobs with an exit code other than zero
  -h, --help                 help for concur
  -j, --job-timeout string   Per-job timeout in time.Duration format (0 default, must be <= global timeout) (default "0")
  -l, --log string           Enable debug mode (one of d, i, w, e, or q for quiet). (default "e")
  -p