Distributed Tracing

A metric can tell you p99 latency went up. It cannot tell you which of the eleven services a request passed through added the extra 400ms. A trace can — it's the record of one request's actual path through the system, as a tree of timed spans, and it's the only signal built specifically to answer "where did the time go" across a distributed call graph.

That answer is only available if the trace is complete. A trace with a gap doesn't just lose detail about the missing service — it loses the ability to say anything conclusive about where the time went at all, because the missing piece could be where it all happened.

Tracing only works if every hop in that graph agrees to pass the same identifier forward. A trace is only as complete as its weakest propagation link — one service that drops the context breaks the chain for everyone downstream of it.

1. Propagate context through every hop, not just the ones you own

A trace is stitched together from a shared trace ID and parent span ID passed along with the request — as an HTTP header, a gRPC metadata field, or a message queue attribute.

If any service in the path doesn't read and forward that context, the trace doesn't just lose that service's spans, it loses everything downstream of it too, because there's no ID left to attach them to. This is why third-party clients, message queues, and background job runners are the most common places tracing silently breaks: they weren't built with a context-propagation hook in mind, so someone has to add it explicitly.

• Synchronous HTTP and gRPC calls propagate through standard headers most tracing libraries handle automatically.
• Message queues and async workers need the trace context stored in the message payload explicitly, since there's no live header to carry it.
• Third-party or legacy clients that strip unknown headers are the most common silent break — verify propagation through them directly, don't assume.

Done when: a single request can be traced end to end through every service and queue it touches, with no unexplained gaps.