Microservices Distributed Tracing: Complete Guide

In microservices architectures, a single request traverses multiple services. Without distributed tracing, debugging is impossible—”the request failed” tells you nothing about which service failed or why. This guide covers tracing fundamentals, OpenTelemetry implementation, and visualization with Jaeger and Zipkin.

Tracing Concepts

flowchart LR
    subgraph Trace ["Trace (Request ID: abc-123)"]
        Span1["Span: API Gateway"] --> Span2["Span: Order Service"]
        Span2 --> Span3["Span: Payment Service"]
        Span2 --> Span4["Span: Inventory Service"]
    end
    
    style Span2 fill:#FFF3E0,stroke:#E65100

• Trace: End-to-end request journey
• Span: Single operation (HTTP call, database query)
• Context: Trace ID + Span ID propagated between services

OpenTelemetry .NET

var builder = WebApplication.CreateBuilder(args);

builder.Services.AddOpenTelemetry()
    .WithTracing(tracing => tracing
        .SetResourceBuilder(ResourceBuilder.CreateDefault()
            .AddService("order-service"))
        .AddAspNetCoreInstrumentation()
        .AddHttpClientInstrumentation()
        .AddSqlClientInstrumentation()
        .AddSource("OrderService.Activities")
        .AddOtlpExporter(options =>
        {
            options.Endpoint = new Uri("http://jaeger:4317");
        }));

// Custom span
using var activity = MyActivitySource.StartActivity("ProcessOrder");
activity?.SetTag("order.id", orderId);
activity?.SetTag("customer.id", customerId);
// ... process order
activity?.SetStatus(ActivityStatusCode.Ok);

Context Propagation

OpenTelemetry automatically propagates context via W3C Trace Context headers:

traceparent: 00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01
tracestate: congo=ucfJifl5GOE

Sampling Strategies

• Always On: 100% of traces (development only)
• Probabilistic: Sample 10% of traces
• Rate Limiting: 100 traces/second max
• Adaptive: Increase sampling on errors

Key Takeaways

• OpenTelemetry is the standard for distributed tracing
• Traces link spans across service boundaries
• Instrument HTTP, database, and custom operations
• Use sampling to manage data volume
• Jaeger/Zipkin for visualization, or cloud-native (AWS X-Ray, Azure Monitor)