Errors and retries

A failed request earns nothing and counts as a frustrated user. Quorum gives every failure one cause, and each cause points to a different fix. Some failures also create more load, which is how small problems become outages.

Every failure has a cause#

Before research, the game shows errors as one number. The Structured logs research splits them by cause:

• stranded: no Node on the path accepts the kind.

• dropped: the Node was over capacity and had no room.

• refused: a limit said no, such as the Database's connection limit.

• down: the Node is in an outage.

• timeout: the caller stopped waiting.

• oom: the process ran out of memory and was killed.

• crash: a bug or a crash took the request down.

• stale: a read returned old data where the kind needs fresh data.

• blocked: the WAF blocked a real user by mistake.

• upstream: a managed service, such as the Payment Provider, failed.

• throttled: a rate limit rejected the request.

• duplicate: a retried checkout charged a card twice.

The breakdown turns "errors are up" into a question you can answer.

Timeouts protect the caller#

A Link waits for an answer until its timeout. Before the Timeouts and retries research, every Link waits 30 seconds and never retries. After it, you set each Link's timeout and retry policy.

A timeout is a trade. On a runtime that holds a slot per request, a caller holds its slot until the call answers or times out. A short timeout frees the slot sooner, so one slow Database cannot fill every App Server. But the slow calls now fail instead of finishing late. And the work the callee already did for them is wasted, because nobody uses the answer.

Retries multiply load#

A retry turns one failure into another attempt. With naive retries, a Link retries at once, up to three times. If a share f of calls fails, each call becomes this many attempts:

At a 50% failure rate, that is nearly twice the load on the callee. The callee was failing because it was overloaded, and now it gets almost double the work.

How a retry storm outlives its cause#

Retries create a feedback loop. Overload causes failures. Failures cause retries. Retries cause more overload. The game carries each retrying Link's failure share into the next Step, so the loop keeps going on its own.

In the Retry storm incident, the Database slows down for a short time. With naive retries, the load stays above its capacity after the slowdown ends, and it queues what it cannot take. The incident ends only when that backlog drains. This is a metastable failure: the system has two stable states, and the bad one keeps itself going.

Three tools break the loop:

• Backoff with jitter waits before each retry, at a random delay, and keeps extra calls inside a small retry budget. Most failures are not retried during a storm.

• Circuit breakers stop calls on a Link when most of them fail. The breaker fails calls at once for a few seconds, then tries again.

• Priority load shedding drops the lowest-value kinds on an overloaded App Server, so the requests it keeps stay fast.

For a step-by-step fix during play, see Stop a retry storm.

Connections: why more servers can break the Database#

The Database runs one process per connection, so it accepts only a limited number of them. Every App Server process holds its own pool of connections. The total need is the sum of every process's pool.

So adding App Servers raises the need. Past the limit, the extra queries fail as refused, and the fix you made for CPU breaks the Database instead. The Connection exhaustion incident makes it worse by doubling every pool.

A Connection Pooler fixes this. In transaction mode, many app connections share a few server connections. The price is that code which keeps session state breaks, until you ship the task that removes it. See connection pooling.

Retries and money#

Some calls must not run twice. A checkout sends a charge to the Payment Provider. If the charge times out and the Link retries, the first attempt may have succeeded already. The customer pays twice, and the request fails as duplicate.

The Idempotency keys research gives each charge a key, so a retry never charges twice. Without it, set the charge Link's retries to none. See idempotency and the Double charge incident.