Bullmq
BullMQ - Message Queue and Batch processing for NodeJS, Python, Elixir and PHP based on Redis
Install / Use
/learn @taskforcesh/BullmqREADME
🛠 Tutorials
You can find tutorials and news in this blog: https://blog.taskforce.sh/
News 🚀
🌐 Language agnostic BullMQ
Do you need to work with BullMQ on platforms other than Node.js? If so, check out the BullMQ Proxy
Official FrontEnd
Supercharge your queues with a professional front end:
- Get a complete overview of all your queues.
- Inspect jobs, search, retry, or promote delayed jobs.
- Metrics and statistics.
- and many more features.
Sign up at Taskforce.sh
🚀 Sponsors 🚀
<table cellspacing="0" cellpadding="0" border="0"> <tr> <td> <a href="https://www.dragonflydb.io/"> <img src="https://raw.githubusercontent.com/dragonflydb/dragonfly/main/.github/images/logo-full.svg" width=550 alt="Dragonfly" /> </a> </td> <td> Dragonfly is a new Redis™ drop-in replacement that is fully compatible with BullMQ and brings some important advantages over Redis™ such as massive better performance by utilizing all CPU cores available and faster and more memory efficient data structures. Read more <a href="https://www.dragonflydb.io/docs/integrations/bullmq">here</a> on how to use it with BullMQ. </td> </tr> </table>Used by
Some notable organizations using BullMQ:
<table cellspacing="0" cellpadding="0"> <tr> <td valign="center"> <a href="https://github.com/microsoft/lage"> <img src="https://files.gitbook.com/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LUuDmt_xXMfG66Rn1GA%2Fuploads%2FUvwInTAmk7hxAViDwJzU%2Fclipart1565701.png?alt=media" width="150" alt="Microsoft" /> </a> </td> <td valign="center"> <a href="https://github.com/vendure-ecommerce/vendure"> <img src="https://files.gitbook.com/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LUuDmt_xXMfG66Rn1GA%2Fuploads%2FvT30DUqsi61gL8edn3R2%2Fwordmark-logo.png?alt=media" width="150" alt="Vendure" /> </a> </td> <td valign="center"> <a href="https://github.com/datawrapper/datawrapper"> <img src="https://files.gitbook.com/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LUuDmt_xXMfG66Rn1GA%2Fuploads%2FCJ5XmotpBBsuSgD8CilC%2Fdatawrapper-logo.png?alt=media" width="150" alt="Datawrapper" /> </a> </td> <td valign="center"> <a href="https://github.com/nestjs/bull/tree/master/packages/bullmq"> <img src="https://876297641-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LUuDmt_xXMfG66Rn1GA%2Fuploads%2FfAcGye182utFUtPKdLqJ%2FScreenshot%202022-02-15%20at%2011.32.39.png?alt=media&token=29feb550-f0bc-467d-a290-f700701d7d15" width="150" alt="Nest" /> </a> </td> <td valign="center"> <a href="https://langfuse.com"> <img src="https://langfuse.com/langfuse_logo.svg" width="150" alt="Langfuse" /> </a> </td> </tr> <tr> <td valign="center"> <a href="https://github.com/teamcurri"> <img src="https://user-images.githubusercontent.com/659829/161662129-ae645bc4-c1e9-48ff-997e-4cee281a964a.png" width="150" alt="Curri" /> </a> </td> <td valign="center"> <a href="https://novu.co"> <img src="https://assets.super.so/1e9f5a51-c4c6-4fca-b6e8-25fa0186f139/images/0f550019-16db-4a65-90d1-1bdb7d3c5f20/novu-logo-gradient-light-background2x.png" width="150" alt="Novu" /> </a> </td> </td> <td valign="center"> <a href="https://www.nocodb.com"> <img src="https://github.com/nocodb/nocodb/raw/develop/packages/nc-gui/assets/img/icons/512x512.png" width="50" alt="NoCodeDB" /> </a> </td> </td> <td valign="center"> <a href="https://infisical.com/"> <picture> <source media="(prefers-color-scheme: dark)" srcset="https://mintlify.s3-us-west-1.amazonaws.com/infisical/logo/dark.svg"> <img src="https://mintlify.s3-us-west-1.amazonaws.com/infisical/logo/light.svg" width="150" alt="Infisical" /> </picture> </a> </td> </tr> </table>The gist
Install:
$ yarn add bullmq
Add jobs to the queue:
import { Queue } from 'bullmq';
const queue = new Queue('Paint');
queue.add('cars', { color: 'blue' });
Process the jobs in your workers:
import { Worker } from 'bullmq';
const worker = new Worker('Paint', async job => {
if (job.name === 'cars') {
await paintCar(job.data.color);
}
});
Listen to jobs for completion:
import { QueueEvents } from 'bullmq';
const queueEvents = new QueueEvents('Paint');
queueEvents.on('completed', ({ jobId }) => {
console.log('done painting');
});
queueEvents.on(
'failed',
({ jobId, failedReason }: { jobId: string; failedReason: string }) => {
console.error('error painting', failedReason);
},
);
Adds jobs with parent-child relationship:
import { FlowProducer } from 'bullmq';
const flow = new FlowProducer();
const originalTree = await flow.add({
name: 'root-job',
queueName: 'topQueueName',
data: {},
children: [
{
name: 'child-job',
data: { idx: 0, foo: 'bar' },
queueName: 'childrenQueueName',
children: [
{
name: 'grandchild-job',
data: { idx: 1, foo: 'bah' },
queueName: 'grandChildrenQueueName',
},
{
name: 'grandchild-job',
data: { idx: 2, foo: 'baz' },
queueName: 'grandChildrenQueueName',
},
],
},
{
name: 'child-job',
data: { idx: 3, foo: 'foo' },
queueName: 'childrenQueueName',
},
],
});
This is just scratching the surface, check all the features and more in the official <a href="https://docs.bullmq.io">documentation</a>
Feature Comparison
Since there are a few job queue solutions, here is a table comparing them:
| Feature | BullMQ-Pro | BullMQ | Bull | Kue | Bee | Agenda | | :------------------------- | :-----------------------------------------: | :-------------------------: | :-------------: | :---: | -------- | ------ | | Backend | redis | redis | redis | redis | redis | mongo | | Observables | ✓ | | | | | | | Group Rate Limit | ✓ | | | | | | | Group Support | ✓ | | | | | | | Batches Support | ✓ | | | | | | | Parent/Child Dependencies | ✓ | ✓ | | | | | | Deduplication (Debouncing) | ✓ | ✓ | ✓ | | | | | Deduplication (Throttling) | ✓ | ✓ | ✓ | | | | | Priorities | ✓ | ✓ | ✓ | ✓ | | ✓ | | Concurrency | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | | Delayed jobs | ✓ | ✓ | ✓ | ✓ | | ✓ | | Global events | ✓ | ✓ | ✓ | ✓ | | | | Rate Limiter | ✓ | ✓ | ✓ | | |
