Trazabilidad distribuida y muestreo de cola
Sigue una petición a través de cada servicio como una traza, y decide cuando termina qué trazas conservar, para que las lentas y las fallidas sobrevivan al muestreo.
Give each request an ID that every hop passes along, then keep the traces worth reading and drop the rest.
Use it when#
A request is slow, the dashboard shows every service as healthy, and nobody knows which hop spent the time.
A page calls many services and you need the slowest one, not the average one.
Tracing every request costs more than the problems it finds.
How it works#
Each service records a span: its name, start and duration, with the trace ID it received. The spans of one request, joined by that ID, form a trace. A waterfall of it shows which hop was slow and which ones ran in parallel.
Head sampling decides at the first service, say keep 1 request in 100. It is cheap, but it drops 99% of the slow ones too. Tail sampling buffers a trace until it ends, then keeps every error and every request slower than a threshold, plus a small share of the rest. OpenTelemetry's collector does this with a tail-sampling processor, and the buffer holds spans for a decision window of about 30 seconds.
With a 1% error rate and 1% head sampling, 1 in 10,000 requests is an error you kept. Tail sampling keeps all of them.
What it costs#
Tail sampling needs every span of a trace on one collector, so it needs routing by trace ID and memory for the buffer. A trace is also only as complete as the services that propagate the ID. One service that drops it cuts every trace in two.
In Quorum#
Link Nodes into an Analytics Node and it samples their traffic, as below.
Drag to pan. Hold Ctrl or Cmd and scroll to zoom.
| Where | Setting | Value | Why |
|---|---|---|---|
| App Server | Instance size | medium | Medium gives the app four vCPUs, so the trace tap has room to run. |
| Database | Instance size | medium | Medium keeps the slowest hop on the app, where the trace should find it. |
| Analytics | Sample rate | 0.05 | Keep one request in twenty. A full sample costs CPU and latency on every tapped Node. |
| Analytics | Retention | 3600 | An hour of traces is enough to compare a slow request with its neighbours. |
Research this Design needs: Load balancing, Percentiles, Distributed tracing, Tail-based sampling.
Distributed tracing names the slowest hop of each request kind, and Tail-based sampling makes sampled failures teach more Failure Insight. The game keeps no per-trace buffer, so tail sampling there changes the Insight earned, not what the tap records. The glossary defines distributed tracing, and Insight and observability covers the cost of a higher rate.