11. Incidents and reliability

An Incident is an event that stresses the design, and incidents are the game's tests. Each one checks whether the design survives a real failure mode that its Era makes likely.

Design rules#

1. Attack the design, not the player. The incident director prefers incidents that hit a real weakness in the current design: a single point of failure, a missing timeout, an unbounded queue. A well-built design survives most incidents with no action. This applies the Factorio lesson that problems should come from the player's own choices (Influences).

2. Readable in seconds. Every incident has a visible signal on the board within 5 seconds.

3. Real. Every incident type is a real class of outage, and most cite a real event.

4. Survivable. No incident ends a company. The worst case is lost users, lost Cash, and a lower valuation.

5. No incidents offline. The game does not punish the player for leaving (Simulation).

6. One at a time, early. Before Series A, at most one incident runs at a time. From Series A a cascade is possible, because cascades are what real large outages look like: an incident nobody answers starts a second one when it ends (Cascades). Even then only one runs at a time.

The incident director#

The director picks the next incident and its time.

There is no per-type base weight: every incident the Era allows starts from the same weight, before weakness.

A Venture can add a signature incident of its own (Exit and Legacy). Each Venture meets one: Big tenant hot shard for B2B SaaS, Hot key for Consumer social, Double charge for Fintech, Premiere spike for Streaming media, Launch day for Game backend, GPU shortage for AI startup, Flash sale for E-commerce, Retry storm for Developer platform, Audit finding for Healthcare, Latency regression for Ad tech, and Deadline day for Government contract. Bootstrapped indie has none: its row names Runway, a figure rather than an incident.

A signature incident weighs four times as much for its Venture and keeps a weakness of 2, so it fires even on a solid design. Six are signatures only: Latency regression, Deadline day, Big tenant hot shard, Premiere spike, Launch day and Flash sale. Another Venture's weakness never starts them, and they weigh 0 for every other Company. The other five are general incidents that their Venture meets four times as often.

weakness(type, design) is 0.2 to 3.0. For example, "Instance failure" has weakness 3.0 when a critical path has a single Node and 0.5 when every Node on it has a redundant copy in another zone; 0.2 is only the floor that every possible incident keeps. The director works as a teacher, because it keeps showing the player their weakest point.

Severity. SEV3 affects one Node or one kind. SEV2 affects a service or a zone. SEV1 affects the whole company or a region. An arrow in the catalog, such as "3→2", means the incident escalates once, halfway through, if it has hurt users for more than half its run so far. It steps to the severity the arrow names, its remaining time is multiplied by 1.5, and a traffic spike's demand is multiplied by 1.5 too. An incident with no arrow has no second severity. The director never starts an incident above the Era's highest starting severity.

Duration. Incident durations are in calendar time, because the SLO is. One sim second is one calendar minute. Before Series A, each incident runs for the playtest duration in its catalog entry, between 30 and 300 sim seconds. From Series A, when the error budget counts, an unmitigated incident lasts at most its starting severity's calendar duration:

An SRE team with an engineer on it restores service 40% faster, so every duration in this document, and the recovery it drives, runs 60% as long (Team).

Only failed requests spend the error budget, so a design with spare capacity or redundancy loses little or nothing.

The incident loop#

An Incident records the sim second its first signal lands and the sim second a player first acts on it. Its Postmortem, its card and the Incidents panel show MTTD, MTTA and MTTR beside each other. MTTD is measured from the start and MTTA from the signal. Either may be unknown: a run that never signalled, and one nobody answered, show a dash. A Game saved without the fields decodes them as unknown, so an old save opens with dashes and loads.

The signal is the pager when the Company has researched alerting: the Step a rule's threshold is crossed. Without it the board is the first signal, and the board shows trouble only when requests fail, at the same 1% the errors rule watches. The pager also catches an incident that has not failed a request yet but has pushed p99 or saturation past its threshold, which is where alerting earns its Insight.

The acknowledgement is the player's first action while the Incident runs. Any action counts, because it means the player is at the board. With on-call a rotation answers the page instead, five sim seconds (five calendar minutes) after it lands, so a Company with a rotation logs a five-minute MTTA whether or not the player was watching. anomaly-detection halves the hold before the pager fires, so the page lands earlier.

Who it pages, and what a page costs. An Incident pages the team that owns the Service whose code failed (Team). Every page is recorded against the engineers on that team and takes a little of their mood. That is how a bad quarter on call turns into a resignation, and why the three answers to an unhappy engineer include time off (Mail). A Company with no engineer on the owning team pages nobody, so the Garage founds alone.

Catalog#

The six kinds#

Ten kinds in the catalog above need more detail than a row holds, so this table gives the Era, severity, duration and weakness of each. The heading keeps its old anchor, which other pages link to.

The JVM warmup lasts 30 s, which is also Tomcat's warmup (06 "Java"). The other durations are 150, 90, 120, 120 and 60 sim seconds. A supply-chain compromise and a restore with no good backup drive away 10% of the Users, and a clock jump adds 1,000 ms to every hop and times out 10% of answers.

A restore with no good backup loses users because the rows in the deleted table are gone (07 "Backups"): pitr and restore-drills are what make the restore succeed. Restarting the Nodes a clock jump hits is one action, because the jump is the region's, not one Node's.

Cascades#

From Series A, an incident that is left alone can start a second one. Left alone means nobody answered it, neither the player's first action nor an on-call rotation, and it ran its whole clock rather than ending early. A response, a drain, a runbook, or an answer of any kind ends that possibility, because the incident was handled.

A cascade starts after its cause, never beside it: the director keeps one incident at a time, so the cause files its postmortem and the second incident begins the same Step. It holds the director's clock too, so the next incident waits for the cascade to end and the usual gap after it. The second one records its cause, and its postmortem does too, so the pager names the chain and the postmortem card says what it followed.

Each pair is a failure that the flagship incidents already describe:

A chain stops after one hop: a cascaded incident carries its cause, and one that carries a cause never starts another. The second incident must also have something to hit, so a Design with none of the second kind's weakness ends the chain there. A kind the director never starts on its own (a Data breach, a Credential leak, a DR drill) is never a cascade either, because only the player starts one. Before Series A, and in a Sandbox, no cascade happens at all.

Flagship incidents#

Retry storm#

The incident that teaches metastable failure.

1. A trigger: a 20-second slowdown on one data Node.

2. Callers time out and retry at once (naive retries, up to 3).

3. Load on the slow Node rises to nearly 4× while it recovers.

4. The trigger ends, but the Node stays overloaded, because retries keep it full.

5. Timeouts upstream cascade to other services.

The player can stop it by opening circuit breakers, shedding load, or temporarily removing traffic until queues drain. The lesson is that the stable state of an overloaded system with retries can be "overloaded" (Simulation).

Cache stampede#

1. A cache Node restarts or a hot key expires.

2. Hit ratio falls from 90% to 0%.

3. The database receives 10× its normal reads and saturates.

4. Requests that refill the cache time out, so the cache stays cold.

Mitigation: limit traffic until the cache fills. Prevention: stampede protection and cache warming (Data tier).

Bad deploy#

Bad-deploy chance per deploy:

The blast radius depends on the deploy batch (Components): a bad deploy fails 8% of App Server requests for each Feature in the batch it shipped. The ceiling on that share is the CI/CD Pipeline's deploy strategy: 40% without a Pipeline and on Rolling, 100% on Recreate, 10% on Blue-green and 5% on Canary. A rolling Cluster deploy stops after its first batch of pods, so pods there take a quarter of that.

Whose traffic it breaks. A bad deploy belongs to the Service that shipped it (Services). Each Service's deploy ships only its own Features, so the batch that breaks is that Service's last deploy, and the kinds its Features put on the wire fail while every other Service keeps serving. A Company with one Service breaks every kind, which is also how an old save loads. The incident and its postmortem name the Service and the team that owns it.

Feature Flags set a ceiling of their own (Components): a Feature only breaks the share of users its flag has reached, so a batch of thirteen Features at 5% breaks 5.2% of requests under Recreate, Rolling and Blue-green, and Canary's 5% holds it to 5. A Feature with no flag is live for every user, as an old save opens. The rollout does not scale the Feature's load, and a Feature's code is never removed.

Mitigation toolbox#

Actions the player can take during an incident. Most need research.

Restarting a Node is for a Node that is running with a bad state. It goes down for its restart time (an App Server boots its runtime, at least one second; any other Node, two seconds) and comes back by the one restart rule every other restart uses: cold, with leaked memory freed, a Cache emptied (Redis reloads part of its data), and a Realtime Hub's clients dropped so that they reconnect (Reconnect storm). Any box or process of the player's restarts: an App Server, a Worker, a Database, a Hub, and so on. A Managed or edge Component does not, because the provider runs it. A Node that is already restarting is refused, and a Stopped Node (the host under it died) is replaced instead, for Cash. Restarting the Node an incident hit ends that incident when the incident is Stale OPcache, Library incompatibility, Runtime compatibility, Split brain (the restart rejoins a Distributed Erlang cluster) or a GC pause storm. A leap second is the region's, so its Restart acts on every Node there at once.

Failing over needs multi-az, and has two forms. On a Node with a standby that is down (a zone outage or a failed Node), it promotes the standby at once instead of after the usual failover time (a sixth of the zone outage). The standby becomes the primary, in a zone the outage did not take, and no copy stands behind it any more, so its Burn halves until the player gives it a new standby. A Node with no standby, or one that is serving, has nothing to promote. During a Zone outage the banner promotes every standby in the zone together. During a Region outage or a DR drill the banner starts again, cold, in the nearest other open region, every stateless Node the outage took down, and the Design pays a replacement for each. Data that lives in one region does not move: only a store spread across regions survives, which is what the drill tests. With no second region open there is nowhere to fail over to.

Throttling connects is the Reconnect storm's second answer, beside waiting for the Hub to warm. The Hub admits the wave at the rate it can take, which is what jittered backoff does, so it refuses no client while it warms. It needs no research and works only on the Hub the storm hit.

Moving the kind is the Event loop block's answer. The service is the App Server behind a Load Balancer. An L7 Load Balancer in front of an App Server on an event loop stops sending it login and sends it to another App Server behind it on a runtime with no event loop. The incident ends once no login reaches an event loop. An L4 Load Balancer carries every kind, and a Design with no second service on a runtime without an event loop has nowhere to move it to, so both are refused.

Draining a Node takes it out of rotation at once, without waiting for a health check, and sends its traffic to the other Nodes on the path, which may be none of them: a drained Database with no replica strands every query sent to it. It finishes what it holds, and a Queue hands its backlog to its Workers while it is out. Ten sim seconds later, empty, it rejoins the rotation. A Node that is already down cannot be drained, because it has already stopped. Draining the Node an Incident is hitting cuts that Incident's run to the drain window: the traffic it was hurting users through has moved, so the Incident is over when the Node is empty.

Runbooks#

A runbook covers one incident kind and runs the known response by itself. That kind ends on the first Step of its next Incident with no player action, and the postmortem records it in its auto field. The Runbook achievement reads that record, so a written runbook earns it as a Design's own healing does.

Writing one takes three things. The team must have handled that kind by hand at least once, so automation is a reward for a lesson already learned (pillar 5): the Game keeps the list of kinds the player ended themselves, because the postmortem record only holds the last four. runbooks must be researched, and an SRE team must have an engineer on it, because someone has to keep the runbook current. The task costs 10 Dev Hours and names its kind in the Add work menu.

The Design's own responses come for free, before any of that: a Cluster that reschedules a pod, a standby that fails over, a Spot Node that comes back, and a retry storm that drains behind backoff all end their incident with no player action. Those, too, are recorded as auto.

Postmortems#

When an incident resolves, a postmortem card appears. It names the incident, the timeline, and the cost (users lost, revenue lost). It is blameless, like the practice it models.

A postmortem pays no Insight. A flat reward made incidents a source of income, so a weak design that failed often researched faster than a strong one. Instead, the postmortem opens one action item: the first research in the incident's prevention list that the player does not own and can study now. The card can put its Study first in the queue. Filing without it leaves the item open, and while any prevention of an incident is open and not owned, the director picks that incident twice as often. An unfixed root cause comes back. A player short on Insight is never blocked; the item stays open until its Study completes. When the design already has every prevention, the card opens no action item.

SLOs and error budgets#

From the Series A Era, the company has an SLO: an availability target per calendar week (Progression).

This is the Google SRE error budget policy. It turns reliability into a resource that the player spends on speed. The budget's hold on velocity needs research error-budgets, after slo-dashboards (Progression): from Series A the budget is always measured, but without that research it changes no velocity.

The budget counts failed requests, not clock time, so a partial failure spends a partial budget. A SEV1 incident that fails 100% of requests for its full 60 calendar minutes spends six times a 99.9% budget. A design that fails 2% of requests for the same hour spends about 12% of it.

SLA credits. B2B SaaS is the Venture that signs an SLA, from Series B (Ventures): its availability target is one level above the Era's. While the trailing week sits below that target, the Company pays credits back to its customers, at 10% of its subscription revenue, and the Cash falls with the Burn. The credits need enterprise-sales, which is what signs the contract. A Venture with no SLA, one that has not reached the Era it signs in, a week that met the target, or a Company without that research pays nothing.

The credit is 10% of a week's subscription revenue, paid as a rate while the week is short. In a steady week that is the same money, and it reads more simply on the Cash series: the player sees the line fall, and the fix is the same as for any other SLO miss.