Quorum
Play
How-to guides
Era 1 · Pre-seed

Read p99 and tail latency

Read the p99 on the HUD, the Users Node, and each card, find the request kind and the Node that make the tail, and pick the fix for its cause.

Use this guide when Apdex is under target but the mean latency looks fine, or when the p99 on the HUD is much higher than the mean.

The p99 is the time that 99 requests out of 100 beat. The slowest 1% sit above it. A mean hides that 1%, but your users still feel it and pay less for it. See Latency and percentiles for the reason.

Before you start#

You need the Percentiles Research. It opens in the Pre-seed Era and needs Metrics first. Without it, the game shows only the mean.

Read the p99#

  1. Look at the latency figure on the HUD. With Percentiles it reads p99, and the mean shows next to it after μ.

  2. Compare the two values.

    • If the p99 is a few times the mean, the tail is normal for a busy Design.

    • If the p99 is ten or more times the mean, a small share of requests is very slow. Continue with step 3.

  3. Select the Users Node. The Inspector opens.

  4. In the Requests section, read the line under each request kind. It shows the mean, p50, p95, and p99 of that kind.

  5. Find the kind whose figures show in amber. Amber means its p99 is above its Apdex target. A red figure means the kind also fails often.

Each kind has its own target. An API read must be fast. A report can take seconds. The Request mix of each Era lists the kinds you serve.

Find the Node that makes the tail#

  1. Read the Latency row on each Node card. With Percentiles it shows the mean and then the p99.

  2. Follow the path of the slow kind from the Users Node to the data tier.

  3. Find the first Node whose p99 jumps far above its mean.

Match the pattern to the cause#

  1. Use the table to name the cause, and apply its fix.

What you seeCauseFix
The Node is amber or red on Load, and every kind on it is slow.Queueing near full capacity.Add capacity. See Find the bottleneck.
A Node.js App Server has a high p99 on every kind, and logins are on it.One CPU-heavy request blocks the event loop.Research libuv thread pool, place an Auth Provider, or use Node.js cluster.
A Java or Go App Server has a normal p50 and a high p99.Garbage collection pauses.Select Larger for more RAM. For Java, pick ZGC in Collector (ZGC). For Go, change GOGC (Go GC tuning).
The p99 rises just after new Java copies start.JIT warmup: new copies run slowly at first.Keep copies warm with a higher minimum on the Autoscaler.
The p99 is high only on one Node that many requests call.The tail at scale: one slow backend slows the whole request.Fix that backend first.

  1. Wait a short time of game time after the change.

  2. Check that the amber figures under the Users Node return to plain.

  3. Check that Apdex on the HUD moves up.

If the p99 does not change, the cause is on another Node of the path. Go back to step 7.

See the tail over time#

  • Press D to open the Dashboard. The Latency chart draws the mean with the p50, p95, and p99 lines. Look for the moment when the p99 left the others.

  • Select Users on the HUD to open Why users leave. A slow row shows the kind's p99 against its target, and it names the share of Apdex that the kind costs you.

  • Research Alerting to get a pager banner when the p99 stays high.

Practice#

The Event Loop Stall Sandbox runs Node.js with logins on the same App Server. Keep the p99 under its limit when password hashing gets slower.

SandboxEvent Loop StallKeep p99 under 300 ms for 2 minutes.

For garbage collection, try the GC Pauses Sandbox.

NextApdex and revenue