Specwrk

Run your RSpec examples across many processors and many nodes for a single build. Or just many processes on a single node. Speeds up your slow (minutes/hours not seconds) test suite by running multiple examples in parallel. Optionally, retry failed examples up-to N-times to avoid breaking deployments.

One CLI command to:

1. Start a queue server for your current build
2. Seed the queue server with all possible examples in the current project
3. Execute

Install

Start by adding specwrk to your project or installing it.

$ bundle add specwrk -g development,test

$ gem install specwrk

CLI

$ specwrk --help

Commands:
  specwrk seed [DIR]            # Seed the server with a list of specs for the run
  specwrk serve                 # Start a queue server
  specwrk start [DIR]           # Start a server and workers, monitor until complete
  specwrk version               # Print version
  specwrk watch                 # Start a server and workers, watch for file changes in the current directory, and execute specs
  specwrk work                  # Start one or more worker processes

specwrk start -c 8 spec/

Intended for quick ad-hoc local host development or single-node CI runs. This command starts a queue server, seeds it with examples from the spec/ directory, and starts 8 worker processes. It will report the ultimate success or failure.

$ specwrk start --help
Command:
  specwrk start

Usage:
  specwrk start [DIR]

Description:
  Start a server and workers, monitor until complete

Arguments:
  DIR                               # Relative spec directory to run against

Options:
  --uri=VALUE                       # HTTP URI of the server to pull jobs from. Overrides SPECWRK_SRV_URI, default: "http://localhost:5138"
  --key=VALUE, -k VALUE             # Authentication key clients must use for access. Overrides SPECWRK_SRV_KEY, default: ""
  --run=VALUE, -r VALUE             # The run identifier for this job execution. Overrides SPECWRK_RUN, default: "main"
  --timeout=VALUE, -t VALUE         # The amount of time to wait for the server to respond. Overrides SPECWRK_TIMEOUT, default: "5"
  --network-retries=VALUE           # The number of times to retry in the event of a network failure. Overrides SPECWRK_NETWORK_RETRIES, default: "1"
  --id=VALUE                        # The identifier for this worker. Overrides SPECWRK_ID. If none provided one in the format of specwrk-worker-8_RAND_CHARS-COUNT_INDEX will be used
  --count=VALUE, -c VALUE           # The number of worker processes you want to start, default: 1
  --output=VALUE, -o VALUE          # Directory where worker output is stored. Overrides SPECWRK_OUT, default: ".specwrk/"
  --seed-waits=VALUE, -w VALUE      # Number of times the worker will wait for examples to be seeded to the server. 1sec between attempts. Overrides SPECWRK_SEED_WAITS, default: "10"
  --port=VALUE, -p VALUE            # Server port. Overrides SPECWRK_SRV_PORT, default: "5138"
  --bind=VALUE, -b VALUE            # Server bind address. Overrides SPECWRK_SRV_BIND, default: "127.0.0.1"
  --store-uri=VALUE                 # Directory where server state is stored. Required for multi-node or multi-process servers.
  --group-by=VALUE                  # How examples will be grouped for workers; fallback to file if no timings are found. Overrides SPECWERK_SRV_GROUP_BY: (file/timings), default: "timings"
  --[no-]verbose                    # Run in verbose mode, default: false
  --max-retries=VALUE               # Number of times an example will be re-run should it fail, default: 0
  --help, -h                        # Print this help

specwrk serve

Only start the server process. Intended for use in CI pipelines.

$ specwrk serve --help
Command:
  specwrk serve

Usage:
  specwrk serve

Description:
  Start a queue server

Options:
  --port=VALUE, -p VALUE            # Server port. Overrides SPECWRK_SRV_PORT, default: "5138"
  --bind=VALUE, -b VALUE            # Server bind address. Overrides SPECWRK_SRV_BIND, default: "127.0.0.1"
  --key=VALUE, -k VALUE             # Authentication key clients must use for access. Overrides SPECWRK_SRV_KEY, default: ""
  --output=VALUE, -o VALUE          # Directory where worker or server output is stored. Overrides SPECWRK_OUT, default: ".specwrk/"
  --store-uri=VALUE                 # Directory where server state is stored. Required for multi-node or multi-process servers.
  --group-by=VALUE                  # How examples will be grouped for workers; fallback to file if no timings are found. Overrides SPECWERK_SRV_GROUP_BY: (file/timings), default: "timings"
  --[no-]verbose                    # Run in verbose mode, default: false
  --[no-]single-run                 # Act on shutdown requests from clients, default: false
  --help, -h                        # Print this help

specwrk seed spec/

Seed the configured server with examples from the spec/ directory. Intended for use in CI pipelines.

specwrk seed --help
Command:
  specwrk seed

Usage:
  specwrk seed [DIR]

Description:
  Seed the server with a list of specs for the run

Arguments:
  DIR                               # Relative spec directory to run against

Options:
  --uri=VALUE                       # HTTP URI of the server to pull jobs from. Overrides SPECWRK_SRV_URI, default: "http://localhost:5138"
  --key=VALUE, -k VALUE             # Authentication key clients must use for access. Overrides SPECWRK_SRV_KEY, default: ""
  --run=VALUE, -r VALUE             # The run identifier for this job execution. Overrides SPECWRK_RUN, default: "main"
  --timeout=VALUE, -t VALUE         # The amount of time to wait for the server to respond. Overrides SPECWRK_TIMEOUT, default: "5"
  --network-retries=VALUE           # The number of times to retry in the event of a network failure. Overrides SPECWRK_NETWORK_RETRIES, default: "1"
  --max-retries=VALUE               # Number of times an example will be re-run should it fail, default: 0
  --help, -h                        # Print this help

specwrk work -c 8

Starts 8 worker processes which will pull examples off the seeded server. Intended for use in CI pipelines.

$ specwrk work --help
Command:
  specwrk work

Usage:
  specwrk work

Description:
  Start one or more worker processes

Options:
  --id=VALUE                        # The identifier for this worker. Overrides SPECWRK_ID. If none provided one in the format of specwrk-worker-8_RAND_CHARS-COUNT_INDEX will be used
  --count=VALUE, -c VALUE           # The number of worker processes you want to start, default: 1
  --output=VALUE, -o VALUE          # Directory where worker output is stored. Overrides SPECWRK_OUT, default: ".specwrk/"
  --seed-waits=VALUE, -w VALUE      # Number of times the worker will wait for examples to be seeded to the server. 1sec between attempts. Overrides SPECWRK_SEED_WAITS, default: "10"
  --uri=VALUE                       # HTTP URI of the server to pull jobs from. Overrides SPECWRK_SRV_URI, default: "http://localhost:5138"
  --key=VALUE, -k VALUE             # Authentication key clients must use for access. Overrides SPECWRK_SRV_KEY, default: ""
  --run=VALUE, -r VALUE             # The run identifier for this job execution. Overrides SPECWRK_RUN, default: "main"
  --timeout=VALUE, -t VALUE         # The amount of time to wait for the server to respond. Overrides SPECWRK_TIMEOUT, default: "5"
  --network-retries=VALUE           # The number of times to retry in the event of a network failure. Overrides SPECWRK_NETWORK_RETRIES, default: "1"
  --help, -h                        # Print this help

specwrk watch -c 8

Starts 8 worker processes in watch mode for the current directory. Watched spec files will be distributed across the processes. By default, only looks at _spec.rb files. Configure a watchfile to map file changes to spec files (i.e. modification of app/models/user.rb should run spec/models/user_spec.rb and spec/system/users_spec.rb).

$ specwrk watch --help
Command:
  specwrk watch

Usage:
  specwrk watch

Description:
  Start a server and workers, watch for file changes in the current directory, and execute specs

Options:
  --watchfile=VALUE                 # Path to watchfile configuration, default: "Specwrk.watchfile.rb"
  --count=VALUE, -c VALUE           # The number of worker processes you want to start, default: 1
  --help, -h                        # Print this help

Configuring your test environment

If your test suite tracks state, starts servers, etc. and you plan on running many processes on the same node, you'll need to make adjustments to avoid conflicting port usage or database/state mutations.

specwrk workers will have TEST_ENV_NUMBER={i} set to help you configure approriately.

Rails

Rails has had easy multi-process test setup for a while now by creating unique test databases per process. For my rails v7.2 app which uses PostgreSQL and Capybara, I made these changes to my spec/rails_helper.rb:

++ if ENV["TEST_ENV_NUMBER"]
++   ActiveRecord::TestDatabases.create_and_load_schema(
++     ENV["TEST_ENV_NUMBER"].to_i, env_name: ActiveRecord::ConnectionHandling::DEFAULT_ENV.call
++   )
++ end
-- Capybara.server_port = 5550
++ Capybara.server_port = 5550 + ENV.fetch("TEST_ENV_NUMBER", "1").to_i
++ Capybara.always_include_port = true
-- ActiveRecord::Migration.maintain_test_schema!
++ ActiveRecord::Migration.maintain_test_schema! unless ENV["SPECWRK_SEED"]

YMMV, but please submit an issue if your setup required more configuration.

CI

Run specwrk in CI in either a single-node or multi-node configuration.

Single-node, multi-process

Single-node, multi-process works best when you only have a single node running tests, but that node has many unused CPUs. This is similar to running specwrk locally with `bundl