Background Jobs in Node.js with BullMQ and Redis

Why BullMQ

BullMQ is the successor to Bull, the most popular Node.js job queue. It runs on Redis, supports priorities, retries, rate limiting, delayed jobs, and repeatable jobs. It is production-ready and actively maintained.

Setting up a queue

import { Queue, Worker } from 'bullmq';
import Redis from 'ioredis';

const connection = new Redis({ host: 'localhost', port: 6379, maxRetriesPerRequest: null });

// Define the queue
const emailQueue = new Queue('email', { connection });

// Add a job
await emailQueue.add('welcome', {
  to: 'user@example.com',
  subject: 'Welcome!',
  template: 'welcome',
  data: { name: 'Alice' },
});

Processing jobs

const worker = new Worker('email', async (job) => {
  console.log(`Processing ${job.name} job ${job.id}`);

  switch (job.name) {
    case 'welcome':
      await sendWelcomeEmail(job.data);
      break;
    case 'reset-password':
      await sendPasswordResetEmail(job.data);
      break;
    default:
      throw new Error(`Unknown job type: ${job.name}`);
  }
}, {
  connection,
  concurrency: 5, // Process 5 jobs simultaneously
});

worker.on('completed', (job) => {
  console.log(`Job ${job.id} completed`);
});

worker.on('failed', (job, err) => {
  console.error(`Job ${job?.id} failed:`, err.message);
});

The concurrency: 5 setting processes up to 5 jobs in parallel. For CPU-bound jobs (PDF generation), set this to the number of CPU cores. For I/O-bound jobs (email, API calls), you can go higher.

Retry strategies

await emailQueue.add('welcome', jobData, {
  attempts: 3,
  backoff: {
    type: 'exponential',
    delay: 2000, // 2s, 4s, 8s
  },
});

Exponential backoff prevents hammering a failing service. After 3 failed attempts, the job moves to the failed state where you can inspect and retry it manually.

Delayed and scheduled jobs

// Send in 30 minutes
await emailQueue.add('reminder', jobData, {
  delay: 30 * 60 * 1000,
});

// Run every day at 9am
await emailQueue.add('daily-report', jobData, {
  repeat: {
    pattern: '0 9 * * *', // cron syntax
  },
});

Delayed jobs are perfect for "send a reminder if the user has not completed onboarding in 24 hours." Repeatable jobs replace cron for most use cases.

Job priorities

// High priority: password reset emails
await emailQueue.add('reset-password', data, { priority: 1 });

// Normal priority: marketing emails
await emailQueue.add('newsletter', data, { priority: 10 });

Lower number = higher priority. Password resets always process before newsletters.

Monitoring with Bull Board

import { createBullBoard } from '@bull-board/api';
import { BullMQAdapter } from '@bull-board/api/bullMQAdapter';
import { ExpressAdapter } from '@bull-board/express';

const serverAdapter = new ExpressAdapter();
serverAdapter.setBasePath('/admin/queues');

createBullBoard({
  queues: [
    new BullMQAdapter(emailQueue),
    new BullMQAdapter(exportQueue),
  ],
  serverAdapter,
});

app.use('/admin/queues', serverAdapter.getRouter());

Bull Board gives you a web UI showing active, completed, failed, and delayed jobs. You can retry failed jobs, view payloads, and monitor throughput.

Separating workers from the web process

Do not run workers in the same process as your web server. A CPU-intensive job will block request handling.

# Web process
node dist/server.js

# Worker process (separate)
node dist/worker.js

In Docker, these are separate containers. In PM2, separate process entries. The queue in Redis connects them — the web process adds jobs, the worker process consumes them.

When BullMQ is not enough

If you need cross-language job processing, exactly-once delivery, or multi-datacenter support, look at RabbitMQ or Kafka. For Node.js-only systems with thousands of jobs per minute, BullMQ is the sweet spot.