Circuit Breaker Pattern: Building Resilient Microservices That Don't Cascade Failures

A slow dependency is more dangerous than a dead one: requests that wait 30 seconds to time out exhaust thread pools and propagate failure upstream across otherwise healthy services. Without a containment mechanism, a single degraded downstream call can saturate an entire distributed system in seconds. The Circuit Breaker pattern contains that failure with three states and two numbers: the request timeout, and the error rate that trips the breaker. The timeout matters most: two to three times your P99 latency, rather than whatever the client library ships as a default. Sized that way, the breaker trips before the thread pool drains.

The Problem: When Slow is Worse Than Dead

Consider a payment provider whose API starts responding slowly. Not down, just taking 20-30 seconds per request instead of the usual 200ms. The calling service waits, and incoming requests pile up behind it until the thread pool drains and memory climbs. A healthy service turns unhealthy, and the degradation moves upstream to every caller that depends on it.

The failure is hard to spot because monitoring stays green. Health checks pass and dashboards show every service as “up”. The only symptom is that callers time out.

Circuit Breaker: Your System’s Safety Valve

The Circuit Breaker pattern acts like an electrical circuit breaker in your house. When things go wrong, it trips, preventing damage from spreading. Unlike the one in your electrical panel, this one resets itself: it periodically tests whether the problem is fixed and restores traffic when it is.

The Three States

enum CircuitState {
  CLOSED = 'CLOSED',  // Normal operation, requests flow through
  OPEN = 'OPEN',  // Circuit tripped, requests fail immediately
  HALF_OPEN = 'HALF_OPEN' // Testing if service recovered
}

Think of it like a bouncer at a club:

• CLOSED: “Come on in, everything’s fine”
• OPEN: “Nobody gets in, there’s a problem inside”
• HALF_OPEN: “Let me check with one person if it’s safe now”

A TypeScript Implementation

The class below keeps failures in a rolling window and applies two independent trip conditions: an absolute failure count and an error rate over a minimum request volume.

interface CircuitBreakerConfig {
  failureThreshold: number;  // Failures before opening
  successThreshold: number;  // Successes to close from half-open
  timeout: number;  // Request timeout in ms
  resetTimeout: number;  // Time before trying half-open
  volumeThreshold: number;  // Min requests before evaluating
  errorThresholdPercentage: number; // Error % to trip
}

class CircuitBreaker {
  private state: CircuitState = CircuitState.CLOSED;
  private failureCount = 0;
  private successCount = 0;
  private lastFailureTime?: Date;
  private window = new RollingWindow(10000); // 10 second window

  constructor(private readonly config: CircuitBreakerConfig) {}

  getState(): CircuitState {
    return this.state;
  }

  async execute<T>(protectedFunction: () => Promise<T>): Promise<T> {
    // Check if we should attempt half-open
    if (this.state === CircuitState.OPEN) {
      if (this.shouldAttemptReset()) {
        this.state = CircuitState.HALF_OPEN;
      } else {
        throw new CircuitOpenError('Circuit breaker is OPEN');
      }
    }

    try {
      const result = await this.executeWithTimeout(protectedFunction);
      this.onSuccess();
      return result;
    } catch (error) {
      this.onFailure();
      throw error;
    }
  }

  private async executeWithTimeout<T>(fn: () => Promise<T>): Promise<T> {
    return Promise.race([
      fn(),
      new Promise<T>((_, reject) =>
        setTimeout(() => reject(new TimeoutError()), this.config.timeout)
      )
    ]);
  }

  private onSuccess(): void {
    this.failureCount = 0;
    this.window.recordSuccess();

    if (this.state === CircuitState.HALF_OPEN) {
      this.successCount++;
      if (this.successCount >= this.config.successThreshold) {
        this.state = CircuitState.CLOSED;
        this.successCount = 0;
      }
    }
  }

  private onFailure(): void {
    this.failureCount++;
    this.lastFailureTime = new Date();
    this.window.recordFailure();

    if (this.state === CircuitState.HALF_OPEN) {
      this.state = CircuitState.OPEN;
      this.successCount = 0;
      return;
    }

    // Check both absolute and percentage thresholds
    const stats = this.window.getStats();
    if (stats.totalRequests >= this.config.volumeThreshold) {
      const errorRate = (stats.failures / stats.totalRequests) * 100;
      if (errorRate >= this.config.errorThresholdPercentage ||
          this.failureCount >= this.config.failureThreshold) {
        this.state = CircuitState.OPEN;
      }
    }
  }

  private shouldAttemptReset(): boolean {
    if (!this.lastFailureTime) return false;
    return Date.now() - this.lastFailureTime.getTime() >= this.config.resetTimeout;
  }
}

Production Lessons

1. Timeout is Your Most Important Setting

Slow responses do more damage than hard failures, and the timeout is the only setting that catches them. A refused connection returns an error in milliseconds. A request that hangs holds its thread until something else gives up, which is the exact condition a breaker exists to prevent.

const config = {
  timeout: 3000,  // 3 seconds, sized against a P99 of 1.2s
  // NOT 30000!  // A 30s wait drains the thread pool before the breaker reacts
};

A worked example for a payment API:

• Normal P50: 180ms
• Normal P99: 1.2s
• Circuit breaker timeout: 3s (roughly 2.5x P99)

2. The Half-Open State Gotcha

A common trap: trip to half-open, send one request, succeed, close the circuit, then immediately fail again with full traffic. The fix: require multiple successes before closing.

// Don't do this
if (testRequest.succeeded) {
  this.state = CircuitState.CLOSED; // Boom! Full traffic returns
}

// Do this instead
if (++this.successCount >= this.config.successThreshold) {
  this.state = CircuitState.CLOSED; // Gradual recovery
}

3. Combine with Retry Logic (But Carefully)

Circuit breakers and retries can create feedback loops. Here’s a reliable combination:

class ResilientClient {
  private readonly circuitBreaker = new CircuitBreaker(clientConfig);

  async callWithResilience(request: Request): Promise<Response> {
    // Circuit breaker wraps retry logic, not vice versa
    return this.circuitBreaker.execute(async () => {
      return await this.retryWithBackoff(request, {
        maxAttempts: 3,
        backoffMs: [100, 200, 400],
        shouldRetry: (error) => {
          // Don't retry circuit breaker errors
          if (error instanceof CircuitOpenError) return false;
          // Don't retry client errors
          if (error.statusCode >= 400 && error.statusCode < 500) return false;
          return true;
        }
      });
    });
  }
}

4. Monitor the Right Metrics

What to track (in order of importance):

1. Circuit state changes - Alert immediately on OPEN
2. Reset attempt results - Failed resets = ongoing problem
3. Request rejection rate - Business impact metric
4. Time in OPEN state - Helps tune reset timeout

Example CloudWatch metrics:

// Custom metrics emitted on every state change (AWS SDK v3)
await cloudwatch.send(new PutMetricDataCommand({
  Namespace: 'CircuitBreakers',
  MetricData: [
    {
      MetricName: 'StateChange',
      Value: 1,
      Unit: 'Count',
      Dimensions: [
        { Name: 'ServiceName', Value: this.serviceName },
        { Name: 'FromState', Value: oldState },
        { Name: 'ToState', Value: newState }
      ]
    },
    {
      MetricName: 'RejectedRequests',
      Value: rejectedCount,
      Unit: 'Count',
      Dimensions: [{ Name: 'ServiceName', Value: this.serviceName }]
    }
  ]
}));

Advanced Patterns: Beyond Basic Circuit Breaking

Bulkheading: Isolated Circuit Breakers

Don’t use one circuit breaker for an entire service. Isolate critical paths:

class PaymentService {
  private readonly chargeBreaker = new CircuitBreaker(chargeConfig);
  private readonly refundBreaker = new CircuitBreaker(refundConfig);
  private readonly queryBreaker = new CircuitBreaker(queryConfig);

  async chargeCard(request: ChargeRequest): Promise<ChargeResponse> {
    // Charging failures don't affect refunds
    return this.chargeBreaker.execute(() => this.api.charge(request));
  }

  async refundPayment(request: RefundRequest): Promise<RefundResponse> {
    // Refunds stay available even if charges are failing
    return this.refundBreaker.execute(() => this.api.refund(request));
  }
}

This pattern proves valuable during high-traffic periods when one endpoint becomes overwhelmed while others remain available.

Fallback Strategies

Not all failures are equal. Sometimes you can degrade gracefully:

async getProductRecommendations(userId: string): Promise<Product[]> {
  try {
    return await this.recommendationBreaker.execute(
      () => this.mlService.getRecommendations(userId)
    );
  } catch (error) {
    if (error instanceof CircuitOpenError) {
      // Fallback to simple popularity-based recommendations
      return this.getPopularProducts();
    }
    throw error;
  }
}

Circuit Breaker Inheritance

For microservices calling other microservices, inherit circuit state:

// API Gateway
if (paymentServiceBreaker.getState() === CircuitState.OPEN) {
  // Don't even try to call order service which depends on payment
  return { error: 'Payment service unavailable', status: 503 };
}

Configuration Baselines by Dependency Type

Starting points for three common dependency classes. Tune them against your own latency profile rather than copying the numbers:

// External API (payment providers, third-party services)
const externalAPIConfig: CircuitBreakerConfig = {
  failureThreshold: 5,  // 5 consecutive failures
  successThreshold: 2,  // 2 successes to recover
  timeout: 5000,  // 5 second timeout
  resetTimeout: 30000,  // Try recovery after 30s
  volumeThreshold: 10,  // Need 10 requests minimum
  errorThresholdPercentage: 50  // 50% error rate trips
};

// Internal microservice
const internalServiceConfig: CircuitBreakerConfig = {
  failureThreshold: 10,  // More tolerant
  successThreshold: 3,
  timeout: 3000,  // Faster timeout
  resetTimeout: 10000,  // Faster recovery attempts
  volumeThreshold: 20,
  errorThresholdPercentage: 30  // More sensitive to error rates
};

// Database connections
const databaseConfig: CircuitBreakerConfig = {
  failureThreshold: 3,  // Quick to trip
  successThreshold: 5,  // Slow to recover
  timeout: 1000,  // Very fast timeout
  resetTimeout: 5000,  // Quick retry
  volumeThreshold: 5,
  errorThresholdPercentage: 20  // Very sensitive
};

Testing Circuit Breakers: Chaos Engineering

You can’t trust a circuit breaker you haven’t tested. A chaos test walks a mock dependency through gradual degradation and asserts that the breaker trips at the right point:

describe('Circuit Breaker Chaos Tests', () => {
  it('should handle gradual degradation', async () => {
    const scenarios = [
      { latency: 100, errorRate: 0 },  // Normal
      { latency: 500, errorRate: 0.1 },  // Slight degradation
      { latency: 2000, errorRate: 0.3 }, // Major degradation
      { latency: 5000, errorRate: 0.7 }, // Near failure
    ];

    for (const scenario of scenarios) {
      mockService.setScenario(scenario);
      await runLoadTest(1000); // 1000 requests

      if (scenario.errorRate > 0.5) {
        expect(breaker.getState()).toBe(CircuitState.OPEN);
      }
    }
  });
});

In production, AWS Fault Injection Simulator can randomly inject failures to verify that circuit breakers respond correctly.

Common Mistakes and Their Consequences

Mistake 1: Client-Side Only Circuit Breaking

Implementing circuit breakers only in clients leaves the server unable to protect itself when it has downstream issues:

// Bad: Client protects itself but server still overwhelmed
class Client {
  private breaker = new CircuitBreaker(clientConfig);
  async call() { return this.breaker.execute(() => fetch('/api')); }
}

// Good: Server also protects itself
class Server {
  private downstreamBreaker = new CircuitBreaker(databaseConfig);
  async handleRequest(req, res) {
    try {
      const data = await this.downstreamBreaker.execute(() =>
        this.database.query(req.query)
      );
      res.json(data);
    } catch (error) {
      if (error instanceof CircuitOpenError) {
        res.status(503).json({ error: 'Service temporarily unavailable' });
      }
    }
  }
}

Mistake 2: Sharing Circuit Breakers Across Unrelated Operations

A single circuit breaker for “database operations” causes read traffic to be blocked when only writes are failing:

// Bad: One breaker for everything
class UserService {
  private dbBreaker = new CircuitBreaker(databaseConfig);

  async getUser(id) {
    return this.dbBreaker.execute(() => db.query('SELECT...'));
  }

  async createUser(data) {
    return this.dbBreaker.execute(() => db.query('INSERT...'));
  }
}

// Good: Separate breakers for different operations
class UserService {
  private readBreaker = new CircuitBreaker(readConfig);
  private writeBreaker = new CircuitBreaker(writeConfig);

  async getUser(id) {
    return this.readBreaker.execute(() => db.query('SELECT...'));
  }

  async createUser(data) {
    return this.writeBreaker.execute(() => db.query('INSERT...'));
  }
}

Mistake 3: Not Considering Business Impact

Treating all services equally leads to situations where payment processing is blocked while lower-priority services like metrics collection continue. Business criticality must drive circuit breaker configuration.

The Implementation Checklist

When implementing circuit breakers, here’s a useful checklist:

• Set timeout to 2-3x your P99 latency
• Require multiple successes before closing from half-open
• Implement separate breakers for read/write operations
• Add fallback behavior for business-critical paths
• Export metrics for state changes and rejections
• Test with chaos engineering before production
• Document timeout and threshold choices
• Alert on circuit OPEN, not on individual failures
• Consider business priority in configuration
• Implement gradual recovery, not instant

When to Reach for a Breaker

A breaker earns its place wherever a synchronous call crosses a process boundary and the caller holds a scarce resource while it waits: a thread, a connection, a Lambda invocation. Payment providers, internal service calls, and database pools all qualify. Size the timeout first and the thresholds second, because the rest of the configuration only refines those two numbers.

There are cases where it is the wrong tool. Work pulled from a queue already has back-pressure, so a breaker adds state without adding protection. A dependency with a single caller and no retry traffic is usually served better by a plain timeout. And a dependency that fails intermittently for unrelated reasons will make the breaker flap between states, which costs more availability than it saves.

References

• Circuit Breaker - Martin Fowler - The canonical description of the circuit breaker pattern, including the three states (closed, open, half-open) and failure thresholds
• Circuit Breaker Pattern - Azure Architecture Center - Microsoft’s production-oriented guide covering implementation considerations, state transitions, and integration with retry logic
• opossum - Node.js Circuit Breaker - Official documentation for the opossum library, the most widely used circuit breaker implementation for Node.js
• opossum - npm - npm package page with install instructions, API summary, and weekly download stats
• Design Patterns for Microservices - Azure Architecture Center - Broader catalogue of resilience patterns that complement the circuit breaker, including bulkhead and retry