Quorum
Play

Retry storm

The Database slowed for 20 s. Callers retried at once, and the retries kept it overloaded.

What happens#

The Database slowed for 20 s. Callers retried at once, and the retries kept it overloaded.

How to prevent it#

Backoff with jitter, circuit breakers, and load shedding.

The postmortem opens the first of these that you do not own yet:

  1. Backoff with jitter
  2. Circuit breakers
  3. Priority load shedding

Engineering terms#