Stop a retry storm · Quorum Wiki

Use this guide when a Retry storm incident runs, or when errors stay high after a Database slowdown has passed.

In a retry storm, the Database slows down for 20 seconds. Callers time out and retry at once, so the Database gets several calls for each request. The slowdown ends, but the retries keep the Database full. This is a metastable failure: the overloaded state holds itself up. The incident ends only when the Database has drained its backlog.

Before you start#

The call settings need Timeouts and retries. Until you own it, every Link waits 30 seconds and never retries. Backoff needs Backoff with jitter, and Breaker needs Circuit breakers.

The director picks this incident most often when a Link into the Database uses Retry 3×.

Recognize the storm#

The Database Load row stays red after the first seconds of the incident.
The error share on the HUD stays high, mostly timeout in Errors by cause on the Users Node.
With Alerting, the pager banner fires on errors.

Stop it now#

Press Space to pause, if you need time.
Select the first App Server that links to the Database. The Inspector opens.
Open the Calls section. It lists each Link out of the App Server by the Node at its end.
Under Database, change Retry 3× to Backoff. With backoff, retries wait and spread out, within a budget of 10% more calls. If you do not own Backoff, select No retries.
If you own Circuit breakers, select Breaker under the same Link. A breaker opens when half the calls on the Link fail. It sends nothing for 5 seconds and fails fast, then tries again.
Repeat steps 2 to 5 for every App Server and Worker that calls the Database.
If you own Priority load shedding, open Overload on each App Server and select Shed low value. The App Servers drop the cheapest kinds first, so less work reaches the Database.
Resume the game and watch the Database Load row.

If the Load row falls and the error share drops under 1%, the backlog is draining. The incident ends when it is empty.

If the Load stays red, the Database is too small for the normal load, too. Select Larger on the Database, or see Scale the database.

Prevent the next one#

On every Link into a data Node, set the retries to Backoff, not Retry 3×.
Add a Breaker on Links to Nodes that can slow down: the Database, a Search Engine, and the Payment Provider.
Set Shed low value on every App Server as a default.
Keep spare capacity on the Database. A storm grows only when the slowed Database cannot keep up with its retries.
When a postmortem card opens, take its action item. The Retry storm card offers Backoff with jitter first, then Circuit breakers, then Priority load shedding.

Check the result#

Select the Users Node and read Errors by cause. timeout must stay near zero.
Open the Dashboard (D). The Errors chart must fall back under its 1% rule after the red incident line.
Check that Apdex on the HUD returns to its target.

Practice#

The Retry Storm Sandbox starts with every App Server call to the Database set to a 1 s timeout and Retry 3×. The Database slows twice. The second time, you must also stay under a Burn limit.

Errors and retries explains how retries multiply load.
Retry with backoff and jitter, Circuit breaker, and Timeout in the glossary.