Circuit Breaker Builder

Protect a service from slow or unavailable upstreams: fail fast, fall back, and isolate the blast radius so one sick dependency can't drain your whole capacity.

Workflow

Set an explicit timeout on every outbound call. Configure connect and read timeouts separately; never rely on library defaults (often minutes or infinite). A sane starting point: connect timeout of 1-2 seconds, read timeout anchored to the dependency's observed p99 latency plus modest headroom - not a round number pulled from the air. The total budget must be shorter than your own caller's timeout so you fail before they give up. A call with no timeout is a guaranteed resource leak under load.
Wrap the call in a three-state breaker. Closed: calls pass through. Open: calls fail instantly without touching the dependency. Half-open: a few probe calls test recovery. Implement these states explicitly per dependency, not a single global breaker.
Trip on a rolling error rate, not a raw count. Open the breaker when the error rate crosses a threshold over a sliding window with a minimum request volume. Widely-used defaults from Hystrix and resilience4j are a good anchor: trip at ≥50% errors over the window, but only once at least 10-20 requests have been observed in it. A raw failure count flaps at low traffic and trips on noise.
Define a fallback per call site. While open, return immediately: stale cache for reads, queue-for-later or accept-and-reconcile for writes, a sane default or degraded response where one exists, and a clean fast error only when there is no safe degradation. The point is to stop spending threads, connections, and latency on a call you expect to fail.
Probe and recover from half-open. After a cooldown - 30-60 seconds is a reasonable starting range - allow a limited number of probe calls (single digits, e.g. 3-5). If they succeed, close the breaker. If any fail, re-open and reset (and back off) the cooldown. Never flood the recovering dependency with full traffic the instant the cooldown expires.
Give each dependency its own bulkhead. Isolate every upstream behind its own concurrency limit - a dedicated connection pool, thread pool, or concurrency semaphore. A saturated dependency then exhausts only its own bulkhead and leaves capacity for unrelated traffic. Never share one pool across multiple upstreams.
Emit metrics and alert. Record state transitions, trip counts, fallback rates, and timeout rates per dependency. Alert when a breaker opens. Tune thresholds and cooldowns against this real data, not against guesses.