Cómo funcionan los incidentes
Cómo el director de incidentes elige qué se rompe, por qué apunta a tu punto más débil y cómo los postmortems, los presupuestos de error y los parches de seguridad conectan los incidentes con tu Diseño.
An incident is a test of your Design, and it is graded on what you built before it started. Each one is a real class of outage. A well-built Design survives most of them with no action from you, and a fragile one shows you exactly where it is fragile.
When incidents start#
The Garage has no incidents. They begin in Pre-seed, shortly after you raise or decline the round. After that, the director waits a random time around the Era's interval before it starts the next one, and only one incident runs at a time.
The clock stops when the Company is closed, so no incident starts while you are away. When you open it again, the director continues from the same Step.
The director aims at your weaknesses#
The director favors the incident that would hurt your Design most. Each incident type gets a weight from how hard it would hit your current Design:
Weakness is 0 when the Design has nothing the incident can hit. Otherwise it runs from 0.2 to 3. An Instance failure weighs 3 on a Node with no second copy, and much less on Nodes with one. A Retry storm weighs most when Links retry into a Database with no backoff.
Tech debt makes every incident likelier.
Repeat. The type that ran last is less likely to run again.
Open action items double an incident's weight (see below).
So the director works like a teacher. It finds the weak point in your Design and stops hitting it once you fix it. Four incidents skip this weighting: a Bad deploy comes only from a deploy, a Data breach comes only from a security patch you missed, a DR drill comes only when you run one, and a Credential leak comes only when you act on a phishing Mail.
Severity and escalation#
Each incident has a severity:
SEV3 hits one Node or one kind.
SEV2 hits a service or a zone.
SEV1 hits the whole Company or a region.
Each Era caps the worst severity an incident can start at, so no incident starts at SEV1 before Series B.
An incident can grow. At its halfway point, if it has hurt users for more than half of its time so far, it steps up one severity. Its remaining time grows by half, and a Traffic spike grows by half too. Acting early keeps an incident small.
How long an incident lasts#
An unmitigated incident ends on its own after a set time. From Series A, incident times follow the calendar, because the SLO does: at most 5 calendar minutes at SEV3, 20 at SEV2, and 60 at SEV1. One sim second is one calendar minute.
Many incidents end sooner when you fix the cause. Replace a failed Node, add the missing index, restart a leaking server, or roll back a bad deploy. A Retry storm is the exception. It has no clock, and it ends only when the backlog on the slow Database drains. Each incident page lists its fix and its prevention.
Postmortems and action items#
When an incident ends, the Simulation files a blameless postmortem. It shows the timeline, the users and revenue lost, the failed requests, and the MTTR: the minutes in which more than 1% of requests failed. See MTTR.
A postmortem pays no Insight. If it did, a weak Design that failed often would research faster than a strong one. It opens one action item instead: the first Research on the incident's prevention list that you do not own and can study now. Until you own it, the director picks that incident twice as often, because an unfixed root cause comes back. The War stories Perk softens this.
Failures still teach you something. Failed and frustrated requests fill the Failure Insight pool, which pays for Reliability Research.
SLOs and the error budget#
From Series A, each Era has an availability SLO, such as 99.9%. The Simulation counts your users' requests and failures over the last calendar week. The failures the SLO allows are your error budget:
Bots do not count, and neither do real users the WAF blocked, because a block is a client error and not a failure to answer. The budget counts failed requests, not clock time, so a partial outage spends a partial budget.
After Error budgets research, the budget changes your team's speed. With more than half left, teams work 10% faster. When it is spent, they work at half speed until it recovers. SLO dashboards put the budget on the HUD. See SLO, SLA, and SLI.
Security patches#
From Pre-seed, a vulnerability in software your Design runs is disclosed every few minutes. Each one puts a patch ticket at the end of your backlog, with a deadline. A higher score means a shorter deadline and a bigger patch. Move the ticket up or rush it with pooled Dev Hours, because if it stays overdue past a short grace, the vulnerability is exploited. A Data breach starts, and 15% of your users leave at once.
Related pages#
Your first incident walks you through one.
Stop a retry storm and Survive a traffic spike handle the hardest ones.
Errors and retries explains why retries make overload worse.