Cascading failure
A failure that jumps from one part of a system to another: one slow or broken component overloads its callers, whose retries and queues in turn overload theirs. The first failure is the trigger, and the later ones are what turn a small incident into a long one.
Reliability
In Quorum#
From Series A, an Incident that nobody answers can start another when it ends, and the second records the first as its cause. A Slow query left alone starts a Retry storm, a Retry storm starts Connection exhaustion, and a Memory leak starts a GC pause storm. The second Incident starts only if the Design is weak to it. The pager names the chain and the postmortem card says what it followed. Any answer, even acknowledging the Incident, stops it from cascading, and a chain stops after one hop.
Design#
Read next#
Metastable failure, Retry with backoff and jitter, Thundering herd, Circuit breaker