Resilience Patterns

Distributed systems fail. The network is unreliable, dependencies misbehave, machines die, deploys roll out half-broken, traffic spikes, clocks drift. Resilience isn't avoiding failure — it's designing the system to absorb it gracefully.

The core insight: you cannot make calls to remote systems reliable. You can only make your service's response to their failures predictable. Every remote call is a place where a failure mode can leak into your system. The patterns here are the toolkit for putting boundaries around those failure modes.

The mindset

• Assume every dependency will fail. Not "could." Will. Plan for it.
• Latency is a failure mode. A call that takes 30 seconds is worse than one that fails fast — it ties up resources and cascades.
• Failures love to be correlated. When one thing breaks, lots of things break together. Patterns that protect against one isolated failure may collapse under correlated failure.
• The remedy can be worse than the disease. Naive retries cause thundering herds. Naive failovers cause split-brain. Every pattern below has a failure mode of its own.
• Test the failure path. Code that's never been exercised is broken; the failure path is the most-untested code in the system. Inject failures and watch what happens.

Timeouts — the foundation

Without timeouts, nothing else matters. A request without a timeout is a resource leak waiting to happen.