Distributed Tracing: Spans, Traces, & OpenTelemetry (OTel)

In modern microservice architectures, a single user interaction (e.g., clicking "Place Order") can fan out into tens of synchronous REST/gRPC calls, message queue publishes, and database queries across different physical nodes.

When an endpoint takes 2.5 seconds to respond, traditional isolated logs and metrics cannot explain which specific network hop or internal function was slow.

Distributed Tracing models the end-to-end execution of a request as a Directed Acyclic Graph (DAG) of Spans:

• Trace: The entire end-to-end journey of a single user transaction across all distributed boundaries. A Trace is identified by a globally unique Trace ID (16-byte hex string, e.g., 4bf92f3577b34da6a3ce929d0e0e4736).
• Span: A single named, timed unit of contiguous work within a service (e.g., an HTTP handler, a database query, or a Redis lookup). Every span contains:
  • Span ID: 8-byte hex string (e.g., 00f067aa0ba902b7).
  • Parent Span ID: The Span ID of the caller (or null for the Root Span).
  • Timestamps: High-precision start and end times.
  • Span Attributes (Key-Value pairs): http.status_code = 200, db.statement = "SELECT * FROM users", user.id = "usr_8821".
  • Span Events (Structured Logs within a span): Point-in-time annotations (e.g., cache_miss, retry_attempt_2).
  • Span Status: Unset, Ok, or Error (with error message and stack trace).

To connect individual spans generated across different servers into a single coherent trace graph, services must pass metadata across network boundaries. This is known as Context Propagation.

The W3C TraceContext specification standardizes the exact HTTP headers and binary gRPC metadata formats:

The traceparent Header Format

text
Copy
traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
              │  └──────────────┬──────────────┘ └───────┬──────┘ └─┬┘
           Version          Trace ID                  Parent Span ID Trace Flags

• Version (00): The specification version.
• Trace ID (32 hex characters): Unique identifier for the entire distributed trace.
• Parent Span ID (16 hex characters): The ID of the span making the outbound request.
• Trace Flags (01): 8-bit field where 01 indicates the trace was recorded/sampled.

Context Propagation in Asynchronous Pipelines

When sending messages through Apache Kafka, RabbitMQ, or AWS SQS, the OpenTelemetry SDK injects the traceparent header directly into the message record headers. When the background consumer receives the message, it extracts the header and creates a child span with a FOLLOWS_FROM relationship, preserving the trace across asynchronous boundaries.

Formed by the merger of OpenTracing and OpenCensus under the Cloud Native Computing Foundation (CNCF), OpenTelemetry (OTel) is the vendor-neutral industry standard for telemetry:

1. OTel API: The abstract interfaces used in application code to instrument spans and metrics. It contains zero operational dependencies and compile-time no-op implementations.
2. OTel SDK: The implementation of the API that manages in-memory buffering, batching, context propagation, and exporting.
3. OTel Collector: A vendor-agnostic proxy daemon that can be deployed as a local sidecar, node DaemonSet, or centralized cluster:
   • Receivers: Ingests telemetry via OTLP (OpenTelemetry Protocol) over gRPC/HTTP, Jaeger, Zipkin, or Prometheus formats.
   • Processors: Batches spans, enforces memory limits, scrubs sensitive PII, and performs Tail Sampling.
   • Exporters: Translates and forwards traces to backend storage systems (Jaeger, Grafana Tempo, AWS X-Ray, Honeycomb, or Datadog).

In high-throughput systems processing 500,000 requests per second, storing 100% of distributed traces is economically impossible (generating petabytes of tracing data per day).

1. Head-Based Sampling (Probabilistic Sampling)

• Mechanism: The sampling decision is made at the Root Span (at the API Gateway) before the request begins execution.
• Example: The gateway flips a biased coin to sample 1\% of all incoming requests. All downstream services inherit this decision via the traceparent sampled flag.
• Pros: Extremely low CPU and memory overhead; no buffering needed.
• Cons: Blind to rare errors! If an unhandled exception occurs on a non-sampled request, the trace is dropped and engineers have zero tracing data for that production crash.

2. Tail-Based Sampling (Intelligent Collector Buffering)

• Mechanism: The decision to keep or drop a trace is delayed until the entire request finishes.
• Architecture: The OpenTelemetry Collector cluster buffers all spans belonging to a TraceID in memory for 10–30 seconds.
• Sampling Rules:
  • If any span in the trace contains an Error (Status = ERROR or HTTP 5xx) → Keep 100%.
  • If total trace latency exceeds 500ms (p99 latency breach) → Keep 100%.
  • For fast, successful requests (< 200ms) → Sample only 0.1%.
• Pros: 100% capture of all production outages and performance anomalies while reducing total storage costs by 95%.
• Cons: Requires a stateful routing tier of OTel Collectors with sufficient RAM to buffer in-flight spans.

Observability reaches maximum power when the three pillars are bidirectionally linked:

1. Prometheus Exemplars → Traces: When Prometheus records a high latency bucket in a histogram, it attaches an Exemplar containing the specific TraceID. In Grafana, clicking on a spike in a latency graph opens the exact Jaeger waterfall trace in one click.
2. Traces → Structured Logs: OpenTelemetry automatically injects trace_id and span_id into the thread-local Mapped Diagnostic Context (MDC). When an application logs a message, these IDs appear in the JSON output, allowing engineers to query OpenSearch for all logs emitted across 15 microservices for that exact request.