12. Observability and Insight

A Hacker News comment on Factorio says: "The only thing preventing you from doing this to your application is the lack of visibility into the problems." Quorum makes visibility a progression track. The player starts with almost nothing, as an early startup does, and earns better tools. It is fog of war for systems.

Observability does two jobs:

1. It produces Insight, the research currency, in one pool per request kind.

2. It decides what the player can see.

Earning Insight#

Analytics is a tap (Components). Each Node linked into it sends a sample of its traffic.

Insight has a request kind. A sampled read gives read Insight, a sampled buy gives buy Insight, and so on. The Company holds one pool for each of the 17 request kinds and one Failure pool. Research costs Insight from named pools (Progression), so a new Era's research needs that Era's traffic. This is the Factorio science pack pattern, with request kinds as the packs (Influences).

A Service Mesh adds +20% (Components). Samples from a Node with a sidecar earn a fifth more, under the same square root, so a Design whose sampled Nodes are all meshed earns 20% more. tail-sampling doubles the whole Failure Insight formula above, outside the square root.

Analytics has a capacity in samples per second. Samples beyond it are dropped.

The rules behind the formula:

• The square root rewards a broad Design. Insight grows with the square root of the traffic, so four times the traffic gives two times the Insight. Each Component type and each request kind is its own term, so a Design that samples a read at the Load Balancer, the App Server, the Cache, and the Database learns four times as much as one that samples it at one Node.

• The sum is per Component type, not per Node. Ten App Servers make one term. Splitting a Node into many Nodes teaches nothing new.

• bot Insight is real. Bots teach how bots behave. Only security research asks for it.

• Failure Insight comes from failures. Reliability research asks for it, so the player learns reliability by seeing failures, as real teams do. Failures also cost users, revenue, and error budget, so failing on purpose is a bad trade. Postmortems pay no Insight (Incidents).

The cost of seeing#

Sampling is not free, and real observability bills surprise teams in the same way.

Ten times the samples gives about three times the Insight, and it costs capacity and Apdex while it is on. A high sample rate buys faster learning at a known price.

Retention#

Analytics keeps traces for a window: 15 minutes (default), 1 hour, or 6 hours of sim time. A longer window multiplies the tap's Storage Burn by 1, 2, or 4.

Insight above a pool's cap is not added. The cap follows the live rate, so the player cannot save up a large balance and buy a whole Era when it opens. floor(pool) is the pool's share of the costliest item open in the Era, and at least 100: a pool always holds one Study, so the Failure Insight of an incident still pays for a Study after the incident ends. Grants ignore the cap.

Other Insight sources#

A Load test's readout (Team) puts every Node at twice the demand, and a DR drill (Exit) fails the home region for a minute. Both push the Design past its limits on purpose, so both pay Failure Insight for as long as they run, on top of whatever the failures they cause earn through Analytics. An experiment that Chaos engineering schedules pays for the 10 seconds it runs (Progression).

The visibility ladder#

What the player sees at each step: an example#

The same problem, a database at 95% utilization behind PHP-FPM servers, seen at three steps:

The player can solve the problem at step 0 by trusting the bottleneck marker. At step 7 they understand why, and they can predict the effect of a change before they make it.

Alerting#

With alerting, the player can set alert rules. The defaults:

Queue growth reads the age of the oldest task in any Queue. Budget burn reads the error budget's burn rate: the share of requests failing against the share the SLO allows, so 1 spends exactly the whole budget over the SLO window. Each rule's threshold has three settings, and a rule pages like the others. Budget burn counts from Series A, when the budget starts.

Alerts that fire too often lower their own value: after five false alarms in a calendar day, the banner shows "Ignored" and the player must tune the rule. That models alert fatigue.

The Ledger and what is hidden#

The Ledger always holds every number. Visibility research changes what the UI shows, not what the simulation computes. A player who has not researched percentiles still loses revenue to slow requests. They only cannot see the p99.

The bottleneck marker is always visible. It is the safety net that keeps the game playable at step 0.

The Churn breakdown#

The bottleneck marker tells the player what limits revenue. The Churn breakdown tells the player why users leave. Users leave when Apdex falls slightly under target, with no outage and no red Node (Simulation), so the player needs a direct answer. Like the bottleneck marker, the Churn breakdown is always available. Research adds detail to it, but does not unlock it.

It opens from the Users figure on the HUD and from the Users Node. It has one section per region, with the region that loses the most users per hour first:

Europe   Apdex 0.91 · target 0.92 · −1.8% users per hour · 1 day at this rate: 410k → 277k
  read     slow     ██████████   0.05   p99 310 ms, T 200 ms
  feed     slow     ████         0.02
  write    failed   ██           0.01   refused
  page     slow     ██           0.01

In the Series C Era, one point of Apdex under target costs a region about a third of its users in a calendar day, when half of them are steady. Nothing else on the screen says so, which is why the Churn breakdown exists.

• Each row is one Request kind and one cause. Its bar is the Apdex points that the row takes from the region: the share of the region's requests in that row, times 1 for a failure and ½ for a tolerating request.

• The rows add up to 1 − A_r. The header marks the target, so the player sees how many points they must win back and which rows hold them.

• The header shows the rate and a forecast for one calendar day at the current Apdex. The figures use the same formula as the Simulation, with no estimate.

• The cause is far for the round trip to a Design in another region, slow for time inside the Design, or failed. Each needed row names one fix, such as Open Asia-Pacific, Research GeoDNS, or Show the bottleneck Node, and shows it once per region.

• Without research, a failure reads failed. With logs, failed shows the error class (Errors). With percentiles, a slow row shows p99 against the kind's T. With tracing, a slow row also names the Node whose hop takes the most time on that kind's path, read from hopByKind. With rum, a slow or far row names the device with the longest round trip over the internet (The visibility ladder).

• Each row's fix button selects the Node it names, or opens the region or the research it needs.

When a region's users fall for one calendar hour, the Customers sender writes a Mail with the top two rows (Mail).

The Inspector#

Click a Node and the Inspector opens on its numbers. It also breaks the Node's load down, so the player can trace a spike to the operations and the upstream Nodes that caused it.

• Incoming operations. Offered traffic per request kind, largest first, as a rate and as a share of the Node's inbound traffic, with the change since the previous sample. Shares describe incoming traffic, not CPU usage.

• Sent by. The same rate, share and change for each Node linked into it, so the cause is one hop back.

• A recommendation only on a measured constraint. A callout appears when the Node is down, swapping, out of CPU credits, refusing connections, reading from disk, stalled, or saturated at its limit, and it names that limit. A large traffic share alone raises nothing.

The Inspector keeps a one-day history per Node in session state, not in the Game: 144 samples, one every 10 sim seconds. With metrics, Traffic & load draws it.

Click a point on the chart, or use Earlier sample and Later sample, to read a past sample. The panel labels it "Recorded sample", and every rate, share and delta is the one recorded then. Selecting a sample changes no Game state and no time, so the Board keeps running. Back to live returns to now. The selection clears when its sample expires or its Node is sold.

The Dashboard#

The Board shows the system now. The Dashboard shows it over time. It is a full-screen view of Charts, with the look of a Grafana dashboard, from the Monitoring sender Grafauna (Mail). It opens with D once metrics is researched, and each later visibility step adds Charts to it. It has no research of its own to buy. It grows as the player climbs the ladder.

The Dashboard and pillar 4#

Pillar 4 says that the Board is the dashboard. The Dashboard keeps that rule because it answers only questions about time and correlation, which the Board cannot answer:

• Did p99 go up after that deploy, or after that resize?

• What happened during the 03:00 spike while I was away?

• Which Node saturated first, and which ones followed?

• Is the error budget burning faster this week than last week?

Rules:

1. No number is only on the Dashboard. Every value on a Chart's last point is also on a Node card, the HUD, or a panel. The Dashboard adds history, not new facts.

2. The bottleneck stays on the Board. The Dashboard has no bottleneck marker. Test: during normal play, the player can still find the bottleneck without the Dashboard.

3. It shows only what the player has researched. It follows the Ledger and what is hidden: a Chart that needs a later step shows as a greyed tile that names the research, as in "No data. Research tracing."

4. It works while paused. The player can study the past while time stands still.

How real teams build dashboards#

The Dashboard copies the methods real teams use, so what the player learns transfers.

Charts by research#

Each visibility step adds a row. A row appears only when its research is owned.

A chaos experiment (Progression) writes its injected refused errors into the Errors row while it runs, so the experiment is visible on the same Chart as the failures it causes. The Team panel's Chaos switch names the Node the next experiment will hit, which is the Design's weakest.

The Dashboard has eight of these rows: Overview, Errors, Latency, Alerts, Services, Resources, Users and SLO. Two more rows read the Game, the Ledger and the Inbox rather than a signal: Company and Incidents. A Chart cannot say who is on the Team, how long the money lasts, or what the last Incident cost, so these rows exist.

Each adds one series drawn from history the Board cannot show, and no new figure. The Company's Money card draws Cash from the Samples' cash. The Incidents row draws each Postmortem's run from endedAt less its duration to endedAt, split at the first signal and the acknowledgement so the spans before each show, with the running Incident last. It reaches back only as far as the four Postmortems the Game keeps and the Chart history behind its axis, so a run older than either stays in the table alone. These rows are not visibility steps: they gather facts the player already has, not a new signal.

Component rows#

Each Component adds its own row of the numbers that matter for it, the way the community dashboards for Postgres, Redis, nginx, or the JVM do. A row appears the first time the Board holds that Component, after metrics, and each Chart reads a number the Ledger already holds.

The Cache row's evictions count the keys a second it pushes out because the hot data does not fit its memory. They stay at 0 while it fits.

Annotations#

Every time series shows vertical markers for the events that change a system. The variables bar carries a switch per group, pressed by default, so the player can hide the marks that sit in the way of a change; the switch shows each group in its colour.

The Deploy, Incident and Company marks come from the Inbox, which is the only record the client keeps of when an event happened: the Game holds no Deploy, Study or round time. That bounds them to the Threads the Inbox still keeps, and it is why a mark names its kind rather than its detail.

The running Incident is marked once, with the SEV it holds now. Every other Incident is marked where its Thread started. The marks show when an Incident began. Its band, with the spans before it was seen and answered, is on the Incidents row (Interface).

Research marks read the same Inbox, and they are the one group the Game does not date. A Study is { id, progress } while it runs. When it completes, advanceStudies takes it off studies and adds its id to researched, so no field holds the second it finished. The mark is the event:finished entry that names a Research, at the second the Inbox wrote it.

That Thread also carries a finished task's Mail, and a letter names its item rather than its kind. So the mark is read from the item name in the text: an entry that names a Research item becomes a Research mark. A per-item field on the Thread's entry would make the mark exact, and so would the time on the Game.

Era markers are in the Company group: the Game holds stageAt, the sim second the current Era began, and the Dashboard marks it with the Era's name.

Design changes matter most. The player changes the Board to fix a problem, and the marker shows whether the line moved after the change. This closes the loop that the ghost-simulation preview opens before the change.

Controls#

• Time range. Last 1 hour, 6 hours (the default), 24 hours, 7 days, and 30 days, in calendar time. At 1× speed, 1 calendar hour is 1 real minute.

• Compare. A switch that draws yesterday's line for the same time range as a faint dashed line under the current series, on the Traffic Chart (Served) and the Latency Chart (p99, or the mean before Percentiles). The player can then separate the daily cycle from a real change. Grafana calls this a time shift. It reads the Dashboard's own blocks, so a range under 10 hours compares against 15-minute means, and a row whose day the history no longer reaches leaves a gap rather than a guess. The line is drawn by default and the switch takes it away.

• Variables. A Request kind and a Region, as drop-downs at the top of the page. They narrow every panel that reads a per-kind or a per-region figure: the Request mix and the Services row by kind; the Users row, the Users stat and the Apdex gauge by region. The Charts over time read the Simulation's global Samples, which carry no kind and no region, so a filter cannot narrow them. Narrowing them would need a Sample history per Request kind and per Region. The bar keeps read-only chips for the Era and the SLO, and the region drop-down appears from Series C, when users live in more than one place.

• Service. A drop-down that appears with a second Service (Services). It narrows the per-kind panels to the kinds that Service's Features touch, and a Code row beside the Services row gives each Service's rate, errors and duration from the Ledger's per-Service figures.

• Crosshair and focus. Hover shows the same moment on every Chart. Clicking a Node's series selects that Node on the Board and closes the Dashboard.

• Table view. A switch that shows each time series' values as a table in place of its plot: one row per time, one column per series with yesterday's when the compare line is drawn, formatted the way the Chart formats them, and a caption naming the Chart and its unit. It is there for screen readers and for reading an exact number (Accessibility).

The player cannot edit queries, add Charts, or move rows. The Dashboard teaches what to look at; building dashboards is out of scope.

History and resolution#

The Simulation keeps only the last 600 one-second Samples of global numbers. The Dashboard needs more, per Node, per Request kind, and over days, so the web app keeps its own history.

Each point at a coarser resolution stores min, max, sum, and count, so a Chart can show the mean and the worst value of a 15-minute block. A spike shorter than the block still shows in the max.

The history is client state, in the same way as Mail (ADR 0006). It is not part of the Game schema, so it does not grow the Simulation's state or change replays (ADR 0004). The save keeps it, and a save without it starts with empty Charts. Offline progress has no per-minute history: the Dashboard shows the time away as one band labelled "Away", with the values from the away card.

The error budget is the one exception: the Simulation keeps it per calendar hour (BudgetHour), so the SLO row reads it from the Game.

The Dashboard redraws once per real second, not on every frame.