Resiliency

Distributed systems fail partially, not totally. Resilience is the discipline of staying useful when components, networks, or dependencies degrade.

Failure Domains

A failure domain is the set of components that fail together. Name failure domains before designing for them — unnamed domains produce unnamed blast radii.

• Crash failure — process exits; detectable immediately by the load balancer or orchestrator.
• Slow failure — process responds but takes too long; the most dangerous mode. Threads and connections fill; the caller eventually crashes too.
• Gray/Byzantine failure — process returns wrong data or errors intermittently; hardest to detect.
• Partial failure — some instances or shards fail while others serve normally.

Cascading failure: a degraded dependency holds resources long enough that the caller exhausts its own pools, propagating failure upstream. Root cause is almost always unbounded resource sharing across failure domains.

Bulkheads

Named after ship compartments: isolate resource pools so a breach in one dependency does not exhaust resources for all others.