Quorum
Play
How-to guides

Find the bottleneck

Find the one Node that limits your Design, learn which of its resources is full, and pick the fix that moves the limit.

Use this guide when Apdex falls under its target, users leave, and you do not know which Node to fix.

A Design is only as fast as its slowest Node. That Node is the bottleneck. Adding capacity anywhere else costs Cash and changes nothing.

Find the Node#

  1. Look for the BOTTLENECK tag on the board. It sits above one Node, and that Node has a pulsing outline.

  2. Select that Node. The Inspector opens.

  3. Read the Load row. It shows the share of capacity in use, as in "85% of 120/s".

  4. Compare the row color with these limits:

    • Plain: under 70%. The Node is healthy.

    • Amber: 70% or more. Waiting time rises fast.

    • Red: 90% or more. The tail latency is several times the median.

The game finds the bottleneck again once per second. It tries each of the busiest Nodes with double the capacity and marks the one that earns the most revenue. Only Nodes above 60% load can get the marker.

If no Node has the tag, capacity does not limit your revenue. Go to "When there is no marker" below.

Find the full resource#

  1. In the Inspector, read the Resources section of the bottleneck Node.

  2. Find the gauge that is nearest to full. Use the table to choose the fix.

Full gaugeWhat it meansFix
CPUThe Node does too much work per request.Select Larger, or place a second copy behind a Load Balancer.
Workers busy, with CPU lowEach request holds a worker while it waits for a slower Node downstream.Fix the Node that the requests wait for. Or select More workers if RAM fits a larger pool.
RAMThe process is near its memory limit.Select Fewer workers, select Larger, or pick the Memory family.
CPU creditsA Burstable box spent its credits and runs at a fraction of its CPU.Pick the General or Compute family.
ConnectionsThe Database is at its connection limit.Place a Connection Pooler. See Scale the database.

A full "Workers busy" gauge with low CPU is the most common trap. The App Server looks full, but the slow part is the Database behind it. This is Little's law: a request that waits longer holds its worker longer.

Apply the fix and check it#

  1. Make one change only.

  2. Wait a few seconds of game time, so that the flow settles.

  3. Check that the Load row of the old bottleneck falls under 70%.

  4. Check that Apdex on the HUD rises toward its target.

  5. Look for the BOTTLENECK tag again. It usually moves to a new Node. Repeat from step 2.

If the tag stays on the same Node and its Load does not fall, you fixed the wrong resource. Go back to step 5.

When there is no marker#

The Design can lose users with no bottleneck. Check these places:

  • A red "Stranded" row on a Node card. No linked Node accepts a request kind. Draw a Link to a Node that accepts it.

  • Errors on the HUD. With Structured logs, select the Users Node and read Errors by cause.

  • Why users leave. Select Users on the HUD. Each row names a request kind and a cause. A slow row has a Show button that selects the bottleneck Node for you.

Other ways to find it#

  • Press D to open the Dashboard, once you own Metrics. The Saturation row shows each Node's load over time. The stat Hottest Node names the busiest Node.

  • With Percentiles, the Latency row of each card shows the p99. A Node with a high p99 and a low load points to tail events, not capacity. See Read p99 and tail latency.

Practice#

The Shared Box Sandbox starts with the Database and the App Server on one box. Find out which resource fills first, and fix it within the Burn limit.

SandboxShared BoxKeep Apdex above 0.85 for 90 seconds.
Next path: Read the numbersLatency and percentiles