Resque

Introduction

Resque (pronounced like "rescue") is a Redis-backed library for creating background jobs, placing those jobs on multiple queues, and processing them later.

For the backstory, philosophy, and history of Resque's beginnings, please see the blog post (2009).

Background jobs can be any Ruby class or module that responds to perform. Your existing classes can easily be converted to background jobs or you can create new classes specifically to do work. Or, you can do both.

Resque is heavily inspired by DelayedJob (which rocks) and comprises three parts:

1. A Ruby library for creating, querying, and processing jobs
2. A Rake task for starting a worker which processes jobs
3. A Sinatra app for monitoring queues, jobs, and workers.

Resque workers can be given multiple queues (a "queue list"), distributed between multiple machines, run anywhere with network access to the Redis server, support priorities, are resilient to memory bloat / "leaks," tell you what they're doing, and expect failure.

Resque queues are persistent; support constant time, atomic push and pop (thanks to Redis); provide visibility into their contents; and store jobs as simple JSON packages.

The Resque frontend tells you what workers are doing, what workers are not doing, what queues you're using, what's in those queues, provides general usage stats, and helps you track failures.

Resque 3.0 requires Ruby 3.0.0 or newer.

Version Support:

• Ruby: 3.0, 3.1, 3.2, 3.3, 3.4+
• Redis gem: 4.0+
• Rack: 2.x or 3.x
• Rails (for ActiveJob): 7.2+ (requires Ruby 3.1+ for Rails 8.0+)

Resque 3.0

Resque 3.0 is a major release that modernizes the codebase with updated dependency requirements:

• Requires Ruby 3.0+ (drops support for Ruby 2.x)
• Requires redis gem 4.0+ (tested with redis gem 4.x and 5.x)
• Requires Sinatra 2.0+ (with Rack 2.x or 3.x support via appropriate Sinatra version)
• Maintains compatibility with Rails 7.2+ and ActiveJob

If you need Ruby 2.x support, please use Resque 2.x.

Example

Resque jobs are Ruby classes (or modules) which respond to the perform method. Here's an example:

class Archive
  @queue = :file_serve

  def self.perform(repo_id, branch = 'master')
    repo = Repository.find(repo_id)
    repo.create_archive(branch)
  end
end

The @queue class instance variable determines which queue Archive jobs will be placed in. Queues are arbitrary and created on the fly - you can name them whatever you want and have as many as you want.

To place an Archive job on the file_serve queue, we might add this to our application's pre-existing Repository class:

class Repository
  def async_create_archive(branch)
    Resque.enqueue(Archive, self.id, branch)
  end
end

Now when we call repo.async_create_archive('masterbrew') in our application, a job will be created and placed on the file_serve queue.

Later, a worker will run something like this code to process the job:

klass, args = Resque.reserve(:file_serve)
klass.perform(*args) if klass.respond_to? :perform

Which translates to:

Archive.perform(44, 'masterbrew')

Let's start a worker to run file_serve jobs:

$ cd app_root
$ QUEUE=file_serve rake resque:work

This starts one Resque worker and tells it to work off the file_serve queue. As soon as it's ready it'll try to run the Resque.reserve code snippet above and process jobs until it can't find any more, at which point it will sleep for a small period and repeatedly poll the queue for more jobs.

Installation

Add the gem to your Gemfile:

gem 'resque'

Next, install it with Bundler:

$ bundle

Rack

In your Rakefile, or some other file in lib/tasks (ex: lib/tasks/resque.rake), load the resque rake tasks:

require 'resque'
require 'resque/tasks'
require 'your/app' # Include this line if you want your workers to have access to your application

Rails

To make resque specific changes, you can override the resque:setup job in lib/tasks (ex: lib/tasks/resque.rake). GitHub's setup task looks like this:

task "resque:setup" => :environment do
  Grit::Git.git_timeout = 10.minutes
end

We don't want the git_timeout as high as 10 minutes in our web app, but in the Resque workers it's fine.

Running Workers

Resque workers are rake tasks that run forever. They basically do this:

start
loop do
  if job = reserve
    job.process
  else
    sleep 5 # Polling frequency = 5
  end
end
shutdown

Starting a worker is simple:

$ QUEUE=* rake resque:work

Or, you can start multiple workers:

$ COUNT=2 QUEUE=* rake resque:workers

This will spawn two Resque workers, each in its own process. Hitting ctrl-c should be sufficient to stop them all.

Priorities and Queue Lists

Resque doesn't support numeric priorities but instead uses the order of queues you give it. We call this list of queues the "queue list."

Let's say we add a warm_cache queue in addition to our file_serve queue. We'd now start a worker like so:

$ QUEUES=file_serve,warm_cache rake resque:work

When the worker looks for new jobs, it will first check file_serve. If it finds a job, it'll process it then check file_serve again. It will keep checking file_serve until no more jobs are available. At that point, it will check warm_cache. If it finds a job it'll process it then check file_serve (repeating the whole process).

In this way you can prioritize certain queues. At GitHub we start our workers with something like this:

$ QUEUES=critical,archive,high,low rake resque:work

Notice the archive queue - it is specialized and in our future architecture will only be run from a single machine.

At that point we'll start workers on our generalized background machines with this command:

$ QUEUES=critical,high,low rake resque:work

And workers on our specialized archive machine with this command:

$ QUEUE=archive rake resque:work

Running All Queues

If you want your workers to work off of every queue, including new queues created on the fly, you can use a splat:

$ QUEUE=* rake resque:work

Queues will be processed in alphabetical order.

Or, prioritize some queues above *:

# QUEUE=critical,* rake resque:work

Running All Queues Except for Some

If you want your workers to work off of all queues except for some, you can use negation:

$ QUEUE=*,!low rake resque:work

Negated globs also work. The following will instruct workers to work off of all queues except those beginning with file_:

$ QUEUE=*,!file_* rake resque:work

Note that the order in which negated queues are specified does not matter, so QUEUE=*,!file_* and QUEUE=!file_*,* will have the same effect.

Process IDs (PIDs)

There are scenarios where it's helpful to record the PID of a resque worker process. Use the PIDFILE option for easy access to the PID:

$ PIDFILE=./resque.pid QUEUE=file_serve rake resque:work

Running in the background

There are scenarios where it's helpful for the resque worker to run itself in the background (usually in combination with PIDFILE). Use the BACKGROUND option so that rake will return as soon as the worker is started.

$ PIDFILE=./resque.pid BACKGROUND=yes QUEUE=file_serve rake resque:work

Polling frequency

You can pass an INTERVAL option which is a float representing the polling frequency. The default is 5 seconds, but for a semi-active app you may want to use a smaller value.

$ INTERVAL=0.1 QUEUE=file_serve rake resque:work

When INTERVAL is set to 0 it will run until the queue is empty and then shutdown the worker, instead of waiting for new jobs.

The Front End

Resque comes with a Sinatra-based front end for seeing what's up with your queue.

Standalone

If you've installed Resque as a gem running the front end standalone is easy:

$ resque-web

It's a thin layer around rackup so it's configurable as well:

$ resque-web -p 8282

If you have a Resque config file you want evaluated just pass it to the script as the final argument:

$ resque-web -p 8282 rails_root/config/initializers/resque.rb

You can also set the namespace directly using resque-web:

$ resque-web -p 8282 -N myapp

or set the Redis connection string if you need to do something like select a different database:

$ resque-web -p 8282 -r localhost:6379:2

Passenger

Using Passenger? Resque ships with a config.ru you can use. See Phusion's guide:

Apache: https://www.phusionpassenger.com/library/deploy/apache/deploy/ruby/ Nginx: https://www.phusionpassenger.com/library/deploy/nginx/deploy/ruby/

Rack::URLMap

If you want to load Resque on a subpath, possibly alongside other apps, it's easy to do with Rack's URLMap:

require 'resque/server'

run Rack::URLMap.new \
  "/"       => Your::App.new,
  "/resque" => Resque::Server.new

Check examples/demo/config.ru for a functional example (including HTTP basic auth).

Rails

You can also mount Resque on a subpath in your existing Rails app by adding require 'resque/server' to the top of your routes file or in an initializer then adding this to routes.rb:

mount Resque::Server.new, :at => "/resque"

Jobs

What should you run in the background? Anything that takes any time at all. Slow INSERT statements, disk manipulating, data processing, etc.

At GitHu