The problem
User A connects to server 1. User B connects to server 2. User A sends a message to user B. Server 1 has no idea user B exists — it is on a different server.
The simple solution: Redis Pub/Sub
Every server subscribes to a Redis channel. When any server receives a message, it publishes to Redis. Every other server receives the message and forwards it to the relevant connected client.
import { createClient } from 'redis';
const publisher = createClient();
const subscriber = createClient();
await publisher.connect();
await subscriber.connect();
// When a message arrives from a WebSocket client
function handleMessage(roomId: string, message: string) {
publisher.publish(`room:${roomId}`, message);
}
// Every server subscribes
await subscriber.pSubscribe('room:*', (message, channel) => {
const roomId = channel.split(':')[1];
broadcastToLocalClients(roomId, message);
});
Connection tracking
Each server maintains a local Map<roomId, Set<WebSocket>>. When a client connects, add it to the map. When it disconnects, remove it. The map only contains clients connected to this server.
Why not a dedicated message broker
RabbitMQ or Kafka would work, but they are overkill for this pattern. We already had Redis for caching. Pub/Sub added zero new infrastructure. The messages are ephemeral — if a server misses a message because it was restarting, that is fine. The client reconnects and catches up.
Heartbeats and cleanup
WebSocket connections die silently. The TCP connection hangs open but the client is gone. We send a ping every 30 seconds. If three pings go unanswered, we close the connection and clean up.
setInterval(() => {
for (const ws of connections) {
if (!ws.isAlive) {
ws.terminate();
continue;
}
ws.isAlive = false;
ws.ping();
}
}, 30000);
Results at 50K concurrent connections
- Memory per server: 600MB for 12K connections
- Redis Pub/Sub latency: under 2ms p99
- Message delivery: under 50ms end-to-end
- Server restarts: zero message loss (clients reconnect within 3 seconds)
The architecture scales horizontally. Need more connections? Add more servers. Redis Pub/Sub handles the fan-out.
