The Circuit Breaker Pattern: Stop Cascading Failures

The cascade problem

Service A calls Service B. Service B is slow — 30-second timeouts instead of 50ms responses. Service A's thread pool fills up waiting for B. Now Service A is slow too. Service C calls A, and the cascade continues until the entire system is down.

This is not theoretical. I have seen a single unhealthy database replica take down a platform serving 500K users.

Three states of a circuit breaker

1. Closed: requests flow normally. Failures are counted.
2. Open: after N failures in a window, the circuit opens. All requests fail immediately without calling the downstream service.
3. Half-open: after a cooldown period, one probe request is allowed through. If it succeeds, the circuit closes. If it fails, the circuit reopens.

Implementation in Node.js

class CircuitBreaker {
  private state: 'closed' | 'open' | 'half-open' = 'closed';
  private failures = 0;
  private lastFailureTime = 0;

  constructor(
    private threshold: number = 5,
    private cooldownMs: number = 30000,
  ) {}

  async call<T>(fn: () => Promise<T>): Promise<T> {
    if (this.state === 'open') {
      if (Date.now() - this.lastFailureTime > this.cooldownMs) {
        this.state = 'half-open';
      } else {
        throw new Error('Circuit is open');
      }
    }

    try {
      const result = await fn();
      this.onSuccess();
      return result;
    } catch (error) {
      this.onFailure();
      throw error;
    }
  }

  private onSuccess() {
    this.failures = 0;
    this.state = 'closed';
  }

  private onFailure() {
    this.failures++;
    this.lastFailureTime = Date.now();
    if (this.failures >= this.threshold) {
      this.state = 'open';
    }
  }
}

Using it in practice

const paymentCircuit = new CircuitBreaker(5, 30000);

async function chargeUser(userId: string, amount: number) {
  try {
    return await paymentCircuit.call(() => paymentService.charge(userId, amount));
  } catch (error) {
    if (error.message === 'Circuit is open') {
      // Queue for retry or return a graceful degradation
      await retryQueue.add('charge', { userId, amount });
      return { status: 'queued' };
    }
    throw error;
  }
}

Use a library in production

The implementation above teaches the concept. In production, use a battle-tested library like opossum:

import CircuitBreaker from 'opossum';

const breaker = new CircuitBreaker(paymentService.charge, {
  timeout: 3000,
  errorThresholdPercentage: 50,
  resetTimeout: 30000,
});

breaker.on('open', () => metrics.increment('circuit.payment.open'));
breaker.on('close', () => metrics.increment('circuit.payment.close'));

const result = await breaker.fire(userId, amount);

opossum gives you percentage-based thresholds, timeout handling, event hooks for monitoring, and fallback functions — all things you do not want to build yourself.

Choosing thresholds

• Error threshold: 50 percent failure rate over a 10-second window is a reasonable starting point. Too low and you trip on transient errors. Too high and you are not protecting anything.
• Cooldown: 30 seconds is typical. Long enough for most transient issues to resolve. Short enough that you recover quickly.
• Timeout: set it lower than your HTTP client timeout. If the downstream normally responds in 50ms, a 3-second timeout is generous.

Monitor your circuits

A circuit breaker that opens silently is worse than no circuit breaker. Emit metrics on every state transition and alert when a circuit stays open for more than 2 minutes. If the circuit keeps tripping, the downstream service needs attention — the breaker is buying you time, not fixing the root cause.