Real-Time Event Processing with Redis Streams

Why Streams over Pub/Sub

Redis Pub/Sub is fire-and-forget. If a subscriber is offline when a message is published, that message is gone. Streams persist messages and let consumers read from any point in the stream — including messages published while they were offline.

Producing events

import Redis from 'ioredis';
const redis = new Redis();

// Add an event to the stream
await redis.xadd('orders', '*',
  'action', 'created',
  'orderId', 'ord_123',
  'userId', 'usr_456',
  'total', '99.99',
);

The * tells Redis to auto-generate a timestamp-based ID. Each entry is a flat key-value map.

Consumer groups

Consumer groups let multiple consumers cooperate on processing a stream. Each message is delivered to exactly one consumer in the group:

// Create the consumer group (run once)
await redis.xgroup('CREATE', 'orders', 'order-processors', '0', 'MKSTREAM');

// Consumer reads pending messages
async function consume(consumerName: string) {
  while (true) {
    const results = await redis.xreadgroup(
      'GROUP', 'order-processors', consumerName,
      'COUNT', 10,
      'BLOCK', 5000,
      'STREAMS', 'orders', '>',
    );

    if (!results) continue;

    for (const [stream, messages] of results) {
      for (const [id, fields] of messages) {
        await processOrder(Object.fromEntries(
          fields.reduce((acc, val, i) =>
            i % 2 === 0 ? [...acc, [val, fields[i + 1]]] : acc, [] as [string, string][]
          )
        ));
        // Acknowledge the message
        await redis.xack('orders', 'order-processors', id);
      }
    }
  }
}

The > means "give me new messages that no one in this group has seen." BLOCK 5000 waits up to 5 seconds for new data instead of busy-polling.

Handling failures

If a consumer crashes before acknowledging a message, the message stays in the Pending Entries List (PEL). Another consumer can claim it:

// Claim messages that have been pending for more than 60 seconds
const stale = await redis.xautoclaim(
  'orders', 'order-processors', 'consumer-2',
  60000, // min idle time in ms
  '0',
);

This is Redis's answer to dead letter queues. Stale messages get redistributed automatically.

Trimming the stream

Streams grow unbounded unless you trim them. Two strategies:

// Keep only the last 10,000 entries
await redis.xtrim('orders', 'MAXLEN', '~', 10000);

// Keep entries from the last 7 days
await redis.xtrim('orders', 'MINID', '~', sevenDaysAgoId);

The ~ means "approximately" — Redis trims in bulk for performance rather than maintaining an exact count.

When to use Kafka instead

• Data volume: Streams live in memory. If you produce millions of events per hour and need days of retention, Kafka's disk-based storage is more economical.
• Multi-datacenter replication: Kafka has built-in cross-datacenter replication. Redis does not.
• Exactly-once processing: Kafka supports it natively. Redis Streams give you at-least-once with manual deduplication.

For most Node.js applications processing thousands of events per minute with hours of retention, Redis Streams are the simpler, cheaper choice.