Kue

Kue is no longer maintained

Please see e.g. Bull as an alternative. Thank you!

Kue is a priority job queue backed by redis, built for node.js.

PROTIP This is the latest Kue documentation, make sure to also read the changelist.

Upgrade Notes (Please Read)

Installation

Latest release:
```
$ npm install kue
```

Master branch:

$ npm install http://github.com/Automattic/kue/tarball/master

Features

Delayed jobs
Distribution of parallel work load
Job event and progress pubsub
Job TTL
Optional retries with backoff
Graceful workers shutdown
Full-text search capabilities
RESTful JSON API
Rich integrated UI
Infinite scrolling
UI progress indication
Job specific logging
Powered by Redis

Overview

Creating Jobs
Jobs Priority
Failure Attempts
Failure Backoff
Job TTL
Job Logs
Job Progress
Job Events
Queue Events
Delayed Jobs
Processing Jobs
Processing Concurrency
Pause Processing
Updating Progress
Graceful Shutdown
Error Handling
Queue Maintenance
Redis Connection Settings
User-Interface
JSON API
Parallel Processing With Cluster
Securing Kue
Testing
Screencasts
License

Creating Jobs

First create a job Queue with kue.createQueue():

var kue = require('kue')
  , queue = kue.createQueue();

Calling queue.create() with the type of job ("email"), and arbitrary job data will return a Job, which can then be save()ed, adding it to redis, with a default priority level of "normal". The save() method optionally accepts a callback, responding with an error if something goes wrong. The title key is special-cased, and will display in the job listings within the UI, making it easier to find a specific job.

var job = queue.create('email', {
    title: 'welcome email for tj'
  , to: 'tj@learnboost.com'
  , template: 'welcome-email'
}).save( function(err){
   if( !err ) console.log( job.id );
});

Job Priority

To specify the priority of a job, simply invoke the priority() method with a number, or priority name, which is mapped to a number.

queue.create('email', {
    title: 'welcome email for tj'
  , to: 'tj@learnboost.com'
  , template: 'welcome-email'
}).priority('high').save();

The default priority map is as follows:

{
    low: 10
  , normal: 0
  , medium: -5
  , high: -10
  , critical: -15
};

Failure Attempts

By default jobs only have one attempt, that is when they fail, they are marked as a failure, and remain that way until you intervene. However, Kue allows you to specify this, which is important for jobs such as transferring an email, which upon failure, may usually retry without issue. To do this invoke the .attempts() method with a number.

 queue.create('email', {
     title: 'welcome email for tj'
   , to: 'tj@learnboost.com'
   , template: 'welcome-email'
 }).priority('high').attempts(5).save();

Failure Backoff

Job retry attempts are done as soon as they fail, with no delay, even if your job had a delay set via Job#delay. If you want to delay job re-attempts upon failures (known as backoff) you can use Job#backoff method in different ways:

    // Honor job's original delay (if set) at each attempt, defaults to fixed backoff
    job.attempts(3).backoff( true )

    // Override delay value, fixed backoff
    job.attempts(3).backoff( {delay: 60*1000, type:'fixed'} )

    // Enable exponential backoff using original delay (if set)
    job.attempts(3).backoff( {type:'exponential'} )

    // Use a function to get a customized next attempt delay value
    job.attempts(3).backoff( function( attempts, delay ){
      //attempts will correspond to the nth attempt failure so it will start with 0
      //delay will be the amount of the last delay, not the initial delay unless attempts === 0
      return my_customized_calculated_delay;
    })

In the last scenario, provided function will be executed (via eval) on each re-attempt to get next attempt delay value, meaning that you can't reference external/context variables within it.

Job TTL

Job producers can set an expiry value for the time their job can live in active state, so that if workers didn't reply in timely fashion, Kue will fail it with TTL exceeded error message preventing that job from being stuck in active state and spoiling concurrency.

queue.create('email', {title: 'email job with TTL'}).ttl(milliseconds).save();

Job Logs

Job-specific logs enable you to expose information to the UI at any point in the job's life-time. To do so simply invoke job.log(), which accepts a message string as well as variable-arguments for sprintf-like support:

job.log('$%d sent to %s', amount, user.name);

or anything else (uses util.inspect() internally):

job.log({key: 'some key', value: 10});
job.log([1,2,3,5,8]);
job.log(10.1);

Job Progress

Job progress is extremely useful for long-running jobs such as video conversion. To update the job's progress simply invoke job.progress(completed, total [, data]):

job.progress(frames, totalFrames);

data can be used to pass extra information about the job. For example a message or an object with some extra contextual data to the current status.

Job Events

Job-specific events are fired on the Job instances via Redis pubsub. The following events are currently supported:

enqueue the job is now queued
start the job is now running
promotion the job is promoted from delayed state to queued
progress the job's progress ranging from 0-100
failed attempt the job has failed, but has remaining attempts yet
failed the job has failed and has no remaining attempts
complete the job has completed
remove the job has been removed

For example this may look something like the following:

var job = queue.create('video conversion', {
    title: 'converting loki\'s to avi'
  , user: 1
  , frames: 200
});

job.on('complete', function(result){
  console.log('Job completed with data ', result);

}).on('failed attempt', function(errorMessage, doneAttempts){
  console.log('Job failed');

}).on('failed', function(errorMessage){
  console.log('Job failed');

}).on('progress', function(progress, data){
  console.log('\r  job #' + job.id + ' ' + progress + '% complete with data ', data );

});

Note that Job level events are not guaranteed to be received upon process restarts, since restarted node.js process will lose the reference to the specific Job object. If you want a more reliable event handler look for Queue Events.

Note Kue stores job objects in memory until they are complete/failed to be able to emit events on them. If you have a huge concurrency in uncompleted jobs, turn this feature off and use queue level events for better memory scaling.

kue.createQueue({jobEvents: false})

Alternatively, you can use the job level function events to control whether events are fired for a job at the job level.

var job = queue.create('test').events(false).save();

Queue Events

Queue-level events provide access to the job-level events previously mentioned, however scoped to the Queue instance to apply logic at a "global" level. An example of this is removing completed jobs:

queue.on('job enqueue', function(id, type){
  console.log( 'Job %s got queued of type %s', id, type );

}).on('job complete', function(id, result){
  kue.Job.get(id, function(err, job){
    if (err) return;
    job.remove(function(err){
      if (err) throw err;
      console.log('removed completed job #%d', job.id);
    });
  });
});

The events available are the same as mentioned in "Job Events", however prefixed with "job ".

Delayed Jobs

Delayed jobs may be scheduled to be queued for an arbitrary distance in time by invoking the .delay(ms) method, passing the number of milliseconds relative to now. Alternatively, you can pass a JavaScript Date object with a specific time in the future. This automatically flags the Job as "delayed".

var email = queue.create('email', {
    title: 'Account renewal required'
  , to: 'tj@learnboost.com'
  , template: 'renewal-email'
}).delay(milliseconds)
  .priority('high')
  .save();

Kue will check the delayed jobs with a timer, promoting them if the scheduled delay has been exceeded, defaulting to a check of top 1000 jobs every second.

Processing Jobs

Proces

Kue

Install / Use

README

Kue