Exq

Exq is a job processing library compatible with Resque / Sidekiq for the Elixir language.

Exq uses Redis as a store for background processing jobs.
Exq handles concurrency, job persistence, job retries, reliable queueing and tracking so you don't have to.
Jobs are persistent so they would survive across node restarts.
You can use multiple Erlang nodes to process from the same pool of jobs.
Exq uses a format that is Resque/Sidekiq compatible.
- This means you can use it to integrate with existing Rails / Django projects that also use a background job that's Resque compatible - typically with little or no changes needed to your existing apps. However, you can also use Exq standalone.
- You can also use the Sidekiq UI to view job statuses, as Exq is compatible with the Sidekiq stats format.
- You can run both Exq and Toniq in the same app for different workers.
Exq supports uncapped amount of jobs running, or also allows a max limit per queue.
Exq supports job retries with exponential backoff.
Exq supports configurable middleware for customization / plugins.
Exq tracks several stats including failed busy, and processed jobs.
Exq stores in progress jobs in a backup queue (using the Redis RPOPLPUSH command). This means that if the system or worker is restarted while a job is in progress, the job will be re_enqueued when the node is restarted and not lost.
Exq provides an optional web UI that you can use to view several stats as well as rate of job processing.
When shutting down Exq will attempt to let workers terminate gracefully, with a configurable timeout.
There is no time limit to how long a job can run for.

Do you need Exq?

While you may reach for Sidekiq / Resque / Celery by default when writing apps in other languages, in Elixir there are some good options to consider that are already provided by the language and platform. So before adding Exq or any Redis backed queueing library to your application, make sure to get familiar with OTP and see if that is enough for your needs. Redis backed queueing libraries do add additional infrastructure complexity and also overhead due to serialization / marshalling, so make sure to evaluate whether it is an actual need or not.

Some OTP related documentation to look at:

GenServer: http://elixir-lang.org/getting-started/mix-otp/genserver.html
Task: https://hexdocs.pm/elixir/Task.html
GenStage: https://hexdocs.pm/gen_stage/GenStage.html
Supervisor: http://elixir-lang.org/getting-started/mix-otp/supervisor-and-application.html
OTP: http://erlang.org/doc/

If you need a durable jobs, retries with exponential backoffs, dynamically scheduled jobs in the future - that are all able to survive application restarts, then an externally backed queueing library such as Exq could be a good fit.

Getting Started

Pre-requisite

This assumes you have an instance of Redis to use. The easiest way to install it on OSX is via brew:

> brew install redis

To start it:

> redis-server

Screencast on elixircasts.io:

If you prefer video instructions, check out the screencast on elixircasts.io which details how to install and use the Exq library: https://elixircasts.io/elixir-job-processing-with-exq

Installation

Add :exq to your mix.exs deps (replace version with the latest hex.pm package version):

defp deps do
  [
    # ... other deps
    {:exq, "~> 0.23.0"}
  ]
end

Then run mix deps.get.

Configuration

By default, Exq will use configuration from your config.exs file. You can use this to configure your Redis host, port, password, as well as namespace (which helps isolate the data in Redis). If you would like to specify your options as a Redis URL, that is also an option using the url config key (in which case you would not need to pass the other Redis options).

Configuration options may optionally be given in the {:system, "VARNAME"} format, which will resolve to the runtime environment value.

Other options include:

The queues list specifies which queues Exq will listen to for new jobs.
The concurrency setting will let you configure the amount of concurrent workers that will be allowed, or :infinite to disable any throttling.
The name option allows you to customize Exq's registered name, similar to using Exq.start_link([name: Name]). The default is Exq.
If the option start_on_application is false, Exq won't be started automatically when booting up you Application. You can start it with Exq.start_link/1.
The shutdown_timeout is the number of milliseconds to wait for workers to finish processing jobs when the application is shutting down. It defaults to 5000 ms.
The mode option can be used to control what components of Exq are started. This would be useful if you want to only enqueue jobs in one node and run the workers in different node.
- :default - starts worker, enqueuer and API.
- :enqueuer - starts only the enqueuer.
- :api - starts only the api.
- [:api, :enqueuer] - starts both enqueuer and api.
The backoff option allows you to customize the backoff time used for retry when a job fails. By default exponential time scaled based on job's retry_count is used. To change the default behavior, create a new module which implements the Exq.Backoff.Behaviour and set backoff option value to the module name.

config :exq,
  name: Exq,
  host: "127.0.0.1",
  port: 6379,
  password: "optional_redis_auth",
  namespace: "exq",
  concurrency: :infinite,
  queues: ["default"],
  poll_timeout: 50,
  scheduler_poll_timeout: 200,
  scheduler_enable: true,
  max_retries: 25,
  mode: :default,
  shutdown_timeout: 5000

Concurrency

Exq supports concurrency setting per queue. You can specify the same concurrency option to apply to each queue or specify it based on a per queue basis.

Concurrency for each queue will be set at 1000:

config :exq,
  host: "127.0.0.1",
  port: 6379,
  namespace: "exq",
  concurrency: 1000,
  queues: ["default"]

Concurrency for q1 is set at 10_000 while q2 is set at 10:

config :exq,
  host: "127.0.0.1",
  port: 6379,
  namespace: "exq",
  queues: [{"q1", 10_000}, {"q2", 10}]

Job Retries

Exq will automatically retry failed job. It will use an exponential backoff timing similar to Sidekiq or delayed_job to retry failed jobs. It can be configured via these settings:

config :exq,
  host: "127.0.0.1",
  port: 6379,
  ...
  scheduler_enable: true,
  max_retries: 25

Note that scheduler_enable has to be set to true and max_retries should be greater than 0.

Dead Jobs

Any job that has failed more than max_retries times will be moved to dead jobs queue. Dead jobs could be manually re-enqueued via Sidekiq UI. Max size and timeout of dead jobs queue can be configured via these settings:

config :exq,
  dead_max_jobs: 10_000,
  dead_timeout_in_seconds: 180 * 24 * 60 * 60, # 6 months

OTP Application

You can add Exq into your OTP application list, and it will start an instance of Exq along with your application startup. It will use the configuration from your config.exs file.

def application do
  [
    applications: [:logger, :exq],
    #other stuff...
  ]
end

When using Exq through OTP, it will register a process under the name Elixir.Exq - you can use this atom where expecting a process name in the Exq module.

If you would like to control Exq startup, you can configure Exq to not start anything on application start. For example, if you are using Exq along with Phoenix, and your workers are accessing the database or other resources, it is recommended to disable Exq startup and manually add it to the supervision tree.

This can be done by setting start_on_application to false and adding it to your supervision tree:

config :exq,
   start_on_application: false

def start(_type, _args) do
  children = [
    # Start the Ecto repository
    MyApp.Repo,
    # Start the endpoint when the application starts
    MyApp.Endpoint,
    # Start the EXQ supervisor
    Exq,
  ]

Sentinel

Exq uses Redix client for communication with redis server. The client can be configured to use sentinel via redis_options. Note: you need to have Redix 0.9.0+.

config :exq
  redis_options: [
    sentinel: [sentinels: [[host: "127.0.0.1", port: 6666]], group: "exq"],
    database: 0,
    password: nil,
    timeout: 5000,
    name: Exq.Redis.Client,
    socket_opts: []
  ]

Using IEx

If you'd like to try Exq out on the iex console, you can do this by typing:

> mix deps.get

and then:

> iex -S mix

Standalone Exq

You can run Exq standalone from the command line, to run it:

> mix do app.start, exq.run

Workers

Enqueuing jobs:

To enqueue jobs:

{:ok, ack} = Exq.enqueue(Exq, "default", MyWorker, ["arg1", "arg2"])

{:ok, ack} = Exq.enqueue(Exq, "default", "MyWorker", ["arg1", "arg2"])

## Don't retry job in per worker
{:ok, ack} = Exq.e

Exq

Install / Use

README

Exq