On this page
Distributed tracing and trace IDs
Tracking a single request as it travels across multiple network boundaries and microservices using injected context headers.
What it is
Distributed tracing is a method for profiling and monitoring applications built using a microservices architecture. It allows engineers to track the exact path of a single user request as it traverses across multiple discrete services, databases, and message queues.
The system relies on two primary concepts:
- Trace ID: A globally unique identifier generated at the very edge of the architecture (usually the API Gateway) when a request first enters the system.
- Span ID: An identifier for a specific unit of work (e.g., an HTTP call to a billing service, or a database query) that occurred during the request.
A complete Trace is simply a tree of Spans, all sharing the same Trace ID, mapping the exact sequence and duration of every step required to fulfill the user’s request.
Why it matters
In a monolith, if a request is slow, a developer can run a local profiler to see which function took the longest. In a distributed system, a single request to /checkout might involve the Cart Service, the Inventory Service, the Payment Gateway, and an Email queue.
If checkout takes 8 seconds, looking at the logs won’t help you determine which of those four services caused the delay. Distributed tracing provides a visual waterfall diagram showing that Cart took 0.1s, Payment took 0.4s, and Inventory waited 7.5s for a database lock. It removes guesswork from latency debugging.
How to implement
Tracing requires context propagation. The Trace ID must be passed from service to service over the network. Modern systems use the W3C traceparent HTTP header standard.
- Generation: A user hits the API gateway. The gateway generates a unique Trace ID (
4bf92f3577b34da6a3ce929d0e0e4736). It creates the first Span and forwards the request to Service A, injecting thetraceparentheader. - Propagation: Service A receives the header. It does its work, creating child Spans for local database queries. When Service A needs to call Service B, it injects the exact same Trace ID into the outgoing HTTP request headers.
- Emission: As each service finishes its work, it asynchronously sends its Span data (start time, end time, Trace ID, and metadata) to a centralized tracing backend (like Jaeger, Zipkin, or Honeycomb).
- Assembly: The tracing backend receives the disparate Spans from all over the cluster, groups them by Trace ID, and assembles them into the final waterfall visualization.
Common mistakes
Breaking the Context Chain
If a developer writes a custom HTTP client and forgets to copy the incoming traceparent header to the outgoing request, the chain is broken. The downstream service will assume it is receiving a brand new request, generate a new Trace ID, and the trace will be split in two. Traces are only useful if the context is propagated flawlessly across every single network boundary.
Tracing Without Logging Integration
Tracing tells you where the latency occurred, but not always why. If an engineer finds a failing span in the tracing UI, they need to see the logs for that exact moment. If the application’s logging library is not configured to automatically inject the current Trace ID into every JSON log line it emits, correlating the trace to the underlying error is a painful, manual timestamp-matching exercise.
Ignoring Asynchronous Boundaries
Tracing HTTP calls is easy. Tracing asynchronous message queues (like Kafka or RabbitMQ) is difficult. If Service A puts a message on a queue, and Service B reads it ten minutes later, the context is often lost. The Trace ID must be explicitly embedded into the message payload or message headers so the worker service can resume the trace when it processes the job.
Verification
To verify tracing is functioning, make a request to a public-facing endpoint using curl and inspect the HTTP response headers. A properly instrumented API gateway will often return the generated Trace ID to the client for support purposes.
curl -i https://api.example.com/checkoutOpen your application logs. Ensure that every single log line contains the Trace ID field:
{"level":"info", "msg":"Querying inventory", "trace_id":"4bf92f3577b34da6a3ce929d0e0e4736"}
{"level":"error", "msg":"Database timeout", "trace_id":"4bf92f3577b34da6a3ce929d0e0e4736"}Finally, open the tracing backend UI (e.g., Jaeger) and search for that exact Trace ID. You should see a contiguous waterfall spanning multiple services, confirming that context propagation is working across network boundaries.