Distributed tracing and tail sampling
Follow one request across every service as a trace, and decide after it finishes which traces to keep, so the slow and failed ones survive sampling.
Give each request an ID that every hop passes along, then keep the traces worth reading and drop the rest.
Use it when#
A request is slow, the dashboard shows every service as healthy, and nobody knows which hop spent the time.
A page calls many services and you need the slowest one, not the average one.
Tracing every request costs more than the problems it finds.
How it works#
Each service records a span: its name, start and duration, with the trace ID it received. The spans of one request, joined by that ID, form a trace. A waterfall of it shows which hop was slow and which ones ran in parallel.
Head sampling decides at the first service, say keep 1 request in 100. It is cheap, but it drops 99% of the slow ones too. Tail sampling buffers a trace until it ends, then keeps every error and every request slower than a threshold, plus a small share of the rest. OpenTelemetry's collector does this with a tail-sampling processor, and the buffer holds spans for a decision window of about 30 seconds.
With a 1% error rate and 1% head sampling, 1 in 10,000 requests is an error you kept. Tail sampling keeps all of them.
What it costs#
Tail sampling needs every span of a trace on one collector, so it needs routing by trace ID and memory for the buffer. A trace is also only as complete as the services that propagate the ID. One service that drops it cuts every trace in two.
In Quorum#
Link Nodes into an Analytics Node and it samples their traffic, as below.
Drag to pan. Hold Ctrl or Cmd and scroll to zoom.
| Where | Setting | Value | Why |
|---|---|---|---|
| App Server | Instance size | medium | Medium gives the app four vCPUs, so the trace tap has room to run. |
| Database | Instance size | medium | Medium keeps the slowest hop on the app, where the trace should find it. |
| Analytics | Sample rate | 0.05 | Keep one request in twenty. A full sample costs CPU and latency on every tapped Node. |
| Analytics | Retention | 3600 | An hour of traces is enough to compare a slow request with its neighbours. |
Research this Design needs: Load balancing, Percentiles, Distributed tracing, Tail-based sampling.
Distributed tracing names the slowest hop of each request kind, and Tail-based sampling makes sampled failures teach more Failure Insight. The game keeps no per-trace buffer, so tail sampling there changes the Insight earned, not what the tap records. The glossary defines distributed tracing, and Insight and observability covers the cost of a higher rate.