Distributed Tracing

A metric can tell you p99 latency went up. It cannot tell you which of the eleven services a request passed through added the extra 400ms. A trace can — it's the record of one request's actual path through the system, as a tree of timed spans, and it's the only signal built specifically to answer "where did the time go" across a distributed call graph.

That answer is only available if the trace is complete. A trace with a gap doesn't just lose detail about the missing service — it loses the ability to say anything conclusive about where the time went at all, because the missing piece could be where it all happened.

Tracing only works if every hop in that graph agrees to pass the same identifier forward. A trace is only as complete as its weakest propagation link — one service that drops the context breaks the chain for everyone downstream of it.

For a complete OpenTelemetry Collector configuration and sampling pipeline, read references/otel-collector-config.md.

1. Propagate context through every hop, not just the ones you own

A trace is stitched together from a shared trace ID and parent span ID passed along with the request — as an HTTP header, a gRPC metadata field, or a message queue attribute.

If any service in the path doesn't read and forward that context, the trace doesn't just lose that service's spans, it loses everything downstream of it too, because there's no ID left to attach them to. This is why third-party clients, message queues, and background job runners are the most common places tracing silently breaks: they weren't built with a context-propagation hook in mind, so someone has to add it explicitly.