Errores y reintentos
Por qué fallan las peticiones, cómo los tiempos de espera y los reintentos pueden convertir una ralentización breve en una caída larga, y por qué más Servidores de aplicaciones pueden romper la Base de datos.
A failed request earns nothing and counts as a frustrated user. Every failure has one cause, and each cause points to a different fix. Some failures also create more load, which is how a small problem becomes an outage.
Every failure has a cause#
Until you study Structured logs, the screen shows errors as one number. The research splits that number by cause:
stranded: no Node on the path accepts the kind.dropped: the Node was over capacity and had no room.refused: a limit said no, such as the Database's connection limit.down: the Node is in an outage.timeout: the caller stopped waiting.oom: the process ran out of memory and was killed.crash: a bug or a crash took the request down.stale: a read returned old data where the kind needs fresh data.blocked: the WAF blocked a real user by mistake.upstream: a managed service, such as the Payment Provider, failed.throttled: a rate limit rejected the request.duplicate: a retried checkout charged a card twice.
With the split, "errors are up" becomes a question you can answer.
Timeouts protect the caller#
A Link waits for an answer until its timeout. Before the Timeouts and retries research, every Link waits 30 seconds and never retries. After it, you set each Link's timeout and retry policy.
A timeout is a trade. On a runtime that holds a slot per request, a caller holds its slot until the call answers or times out. A short timeout frees the slot sooner, so one slow Database cannot fill every App Server. The price is that slow calls fail instead of finishing late, and the work the callee already did for them is wasted, because nobody uses the answer.
Retries multiply load#
A retry turns one failure into another attempt. With naive retries, a Link retries at once, up to
three times. If a share f of calls fails, each call becomes this many attempts:
At a 50% failure rate, that is nearly twice the load on the callee. It was failing because it was overloaded, so retries hand it almost double the work.
A Link retries at once, up to three times. Half of its calls fail. About how much load does the callee now get, compared with no retries?
How a retry storm outlives its cause#
Retries form a feedback loop: overload causes failures, failures cause retries, and retries cause more overload. The Simulation carries each retrying Link's failure share into the next Step, so the loop keeps going on its own.
In the Retry storm incident, the Database slows down for a short time. With naive retries, the load stays above its capacity after the slowdown ends, and the Database queues what it cannot take. The incident ends only when that backlog drains. This is a metastable failure: the system has two stable states, and the bad one keeps itself going.
SandboxTormenta de reintentosMantén el Apdex por encima de 0.9 durante 90 segundos.Three tools break the loop:
Backoff with jitter waits before each retry, at a random delay, and keeps extra calls inside a small retry budget. Most failures are not retried during a storm.
Circuit breakers stop calls on a Link when most of them fail. The breaker fails calls at once for a few seconds, then tries again.
Priority load shedding drops the lowest-value kinds on an overloaded App Server, so the requests it keeps stay fast.
For a step-by-step fix during play, see Stop a retry storm.
Connections: why more servers can break the Database#
The Database runs one process per connection, so it accepts only a limited number of them. Every App Server process holds its own pool of connections, and the Database needs the sum of every pool.
So adding App Servers raises the need. Past the limit, the extra queries fail as refused, and the
fix you made for CPU breaks the Database instead. The Connection exhaustion
incident makes it worse by doubling every pool.
A Connection Pooler fixes this. In transaction mode, many app connections share a few server connections. The price is that code which keeps session state breaks, until you ship the task that removes it. See connection pooling.
SandboxLímite de conexionesAtiende todas las peticiones durante 90 segundos.Retries and money#
A checkout must not run twice. It sends a charge to the Payment Provider, and if the charge times out
and the Link retries, the first attempt may have succeeded already. The customer pays twice, and the
request fails as duplicate.
The Idempotency keys research and the Idempotency keys engineering task that puts it in the code give each charge a key, so a retry never charges twice. Without them, set the charge Link's retries to none. See idempotency and the Double charge incident.