Bird
Raised Fist0

Distributed Tracing System - Trace Context Propagation Across Services

HLDmediumInterview freq: medium
Choose your preparation mode4 modes available
🏗️
Distributed Tracing System - Trace Context Propagation Across Services
Design a scalable distributed tracing system that propagates trace context across microservices to enable end-to-end latency and error analysis.
mediumMonitoring and ObservabilityFreq: medium
⚙️
Functional Requirements
  • Propagate trace context (trace ID, span ID, sampling flags) across all microservice calls synchronously and asynchronously.
  • Capture and store trace spans with metadata (timestamps, service name, operation name, tags) for each service invocation.
  • Support distributed context propagation over HTTP headers (e.g., W3C Trace Context) and messaging systems (Kafka, RabbitMQ).
  • Provide APIs to query traces by trace ID, service, time range, and error status.
  • Support sampling strategies (probabilistic, rate-limiting) to control data volume.
  • Visualize end-to-end traces with span timelines and dependency graphs.
📊
Non-Functional Requirements
  • Handle 2 billion daily active users generating 40 trace spans per user per day.
  • Ensure 99.99% availability for trace ingestion and query APIs.
  • Support ingestion peak write QPS of approximately 925K messages per second.
  • Provide trace query latency P99 under 100ms for recent traces (last 1 hour).
  • Data retention for trace data of 30 days with automatic expiration.
❓
Clarifying Questions to Ask
  • What is the expected average and peak QPS for trace span ingestion?
  • What trace context propagation standards or protocols must be supported (e.g., W3C Trace Context)?
  • Is asynchronous context propagation over messaging systems required?
  • What is the expected trace data retention period and storage budget?
  • What sampling strategies are preferred or required?
  • Are there any SLAs for trace query latency and availability?
Out of Scope
✗ Instrumentation libraries and SDKs for automatic trace context injection/extraction.
✗ Detailed UI/UX design of trace visualization dashboards.
✗ Alerting and anomaly detection based on trace data.
✗ Integration with external APM or monitoring tools.
📌
Assumptions
  • 2 billion daily active users (DAU).
  • Each user generates 40 trace spans per day on average.
  • Average trace span size is 100 bytes (including metadata and context).
  • Retention period is 30 days.
  • Replication factor for storage is 3.
⚡
QPS Calculation
write qps2B users × 40 spans/user/day ÷ 86400 seconds/day ≈ 925,926 spans/sec
peak write qpsAssuming 3× peak factor during traffic spikes: 925,926 × 3 ≈ 2.78M spans/sec
read qpsAssuming 10% of write QPS for trace queries: 925,926 × 0.1 ≈ 92,593 queries/sec
💾
Storage Estimate
daily storage gb(2B × 40 × 100 bytes) ÷ (1024³) ≈ 7.45 TB/day
retention storage gb7.45 TB/day × 30 days ≈ 223.5 TB
with replication gb223.5 TB × 3 (replication) ≈ 670.5 TB
🖥️
Server Estimate
ingestion clusterKafka cluster with 100 brokers to handle 2.78M peak writes/sec (each broker ~30K msg/sec throughput)
storage clusterCassandra cluster with 200 nodes to store 670.5 TB replicated data with 10K writes/sec per node capacity
🏗️
Architecture Diagram
Architecture Diagram
Client Service A
    |
    v
Trace Context Injector (SDK)
    |
    v
HTTP/gRPC Call with Trace Headers
    |
    v
Service B (Trace Context Extractor)
    |
    v
Span Collector (Kafka Producer)
    |
    v
Kafka Cluster (Topic: trace-spans)
    |
    v
Span Processor (Kafka Consumer)
    |
    v
Storage Cluster (Apache Cassandra)
    |
    v
Trace Query API (Cassandra Reader)
    |
    v
Trace Visualization UI
🧩
Components
Trace Context Injector (SDK)
Injects trace context headers into outgoing requests synchronously and asynchronously.
Why thisOpenTelemetry SDK supports W3C Trace Context standard and integrates with multiple languages for automatic context propagation.
Why notCustom header injection lacks standardization and interoperability across heterogeneous services.
Kafka Cluster
Durable, scalable message queue for ingesting trace spans asynchronously from multiple services.
Why thisKafka supports high throughput (millions of messages/sec), partitioning, and retention policies suitable for trace span ingestion.
Why notRabbitMQ does not scale linearly beyond hundreds of thousands of messages/sec and lacks native partitioning.
Span Processor
Consumes trace spans from Kafka, enriches and batches them before writing to storage.
Why thisKafka Consumer groups enable horizontal scaling and fault tolerance for processing high QPS trace data.
Why notDirect writes from services to storage cause backpressure and reduce ingestion reliability.
Storage Cluster (Apache Cassandra)
Stores trace spans with high write throughput and supports time-series data with TTL for automatic expiration.
Why thisCassandra's wide-column model and tunable consistency enable scalable writes at 925K+ QPS with 30-day TTL.
Why notPostgreSQL requires complex manual sharding at this scale; B-tree writes don't scale linearly.
Trace Query API
Provides low-latency queries for trace retrieval by trace ID, service, and time range.
Why thisStateless microservice querying Cassandra with secondary indexes and caching for sub-100ms P99 latency.
Why notElasticsearch indexing adds complexity and latency; not optimized for high cardinality trace IDs.
Trace Visualization UI
Displays end-to-end trace timelines and service dependency graphs for observability.
Why thisCustom frontend optimized for trace data visualization with interactive span timelines.
Why notGeneric dashboards lack trace-specific visualization features.
🔄
Data Flow
1
Client Service A → Service B
Inject trace context headers into outgoing HTTP/gRPC requests
Uses OpenTelemetry SDK to propagate trace ID and span ID
2
Service B → Span Collector
Extract trace context and create new span for incoming request
Maintains parent-child relationship for distributed tracing
3
Span Collector → Kafka Cluster
Publish serialized span data asynchronously to Kafka topic
Decouples ingestion from storage for high throughput
4
Span Processor → Storage Cluster
Consume spans from Kafka, enrich and batch them, then write to Cassandra
Enables scalable, fault-tolerant storage with TTL for automatic expiration
5
Trace Query API → Storage Cluster
Query trace spans by trace ID, service, and time range
Provides low-latency access to recent trace data
6
Trace Visualization UI → Trace Query API
Fetch trace data for visualization
Displays end-to-end trace timelines and dependency graphs
🗄️
trace_spans
PK: trace_id
trace_id UUIDspan_id UUIDparent_span_id UUIDservice_name textoperation_name textstart_timestamp timestampend_timestamp timestamptags map<text,text>sampling_flags int
Stores individual trace spans partitioned by trace ID and clustered by start time for efficient retrieval.
🗄️
service_traces
PK: service_name
service_name texttrace_id UUIDspan_id UUIDoperation_name textstart_timestamp timestampend_timestamp timestamperror_flag boolean
Indexes traces by service and time to support queries filtering by service and time range.
🔌
GET
GetTraceById
Request
{
  "trace_id": "UUID"
}
Response
{
  "trace": "List of spans with metadata"
}
Fetches all spans for a given trace ID.
🔌
POST
QueryTraces
Request
{
  "service_name": "string",
  "start_time": "timestamp",
  "end_time": "timestamp",
  "error_only": "boolean (optional)"
}
Response
{
  "traces": "List of trace summaries matching criteria"
}
Supports querying traces by service, time range, and error status.
🔌
GET
GetTraceDependencies
Request
{
  "start_time": "timestamp",
  "end_time": "timestamp"
}
Response
{
  "dependencies": "Service dependency graph data"
}
Returns service dependency graphs for visualization.

Practice

(1/5)
1. A distributed system requires generating unique IDs across multiple services without coordination delays. Which component best ensures strictly increasing, collision-free IDs with minimal latency?
easy
A. Use timestamps combined with random bits generated locally
B. Use a UUID generator on each service independently
C. Use a distributed consensus algorithm like Raft for ID assignment
D. Use a centralized Ticket Server issuing sequential IDs

Solution

  1. Step 1: Identify uniqueness and ordering requirements

    UUIDs (A) are unique but not strictly increasing; timestamps with randomness (D) risk collisions under clock skew; consensus (C) adds coordination overhead and latency.
  2. Step 2: Recognize Ticket Server role

    The Ticket Server (B) centrally issues sequential IDs ensuring strict ordering and uniqueness with minimal latency due to lightweight allocation.
  3. Final Answer:

    Option D -> Option D
  4. Quick Check:

    Centralized Ticket Server -> strict, collision-free IDs with low latency [check]
Hint: Central server issues ordered IDs, avoiding coordination delays [check]
Common Mistakes:
2. In a horizontally scaled web service with a load balancer distributing requests to stateless app servers, at step 3 the load balancer receives a request from a client with a session cookie. What happens if the load balancer uses round-robin without sticky sessions?
easy
A. The client session cookie is ignored and a new session is created on each request
B. The request is always routed to the same app server that holds the session state
C. The load balancer caches the session state and forwards it to the app server
D. The request may be routed to a different app server, causing session state loss if servers are stateless

Solution

  1. Step 1: Understand load balancer behavior with round-robin and no sticky sessions

    Requests are distributed evenly but not consistently to the same server.
  2. Step 2: Analyze session state handling

    Stateless servers do not hold session state, so routing to different servers does not cause loss; but if state is stored in-memory, session loss occurs.
  3. Final Answer:

    Option D -> Option D
  4. Quick Check:

    Round-robin without sticky sessions routes requests to different servers, risking session loss [check]
Hint: Round-robin without sticky sessions breaks session affinity [check]
Common Mistakes:
3. For the restaurant discovery feature, you must support millions of users searching by location with low latency. Which architecture is best suited given the constraint that restaurant data updates happen frequently (every few minutes)?
medium
A. Use a relational database with geospatial indexes and query directly for each user search
B. Use a distributed cache with geospatial indexing that asynchronously syncs updates from the primary database
C. Use a batch ETL pipeline to rebuild a search index every hour and serve queries from it
D. Use client-side filtering by downloading all restaurants in the city and filtering locally

Solution

  1. Step 1: Analyze read/write pattern for restaurant discovery

    High read volume with frequent writes requires low-latency reads and near-real-time updates.
  2. Step 2: Evaluate architectures

    Relational DB (A) can't handle query volume with low latency; batch ETL (C) is too stale; client filtering (D) is impractical for large data.
  3. Final Answer:

    Option B -> Option B
  4. Quick Check:

    Distributed cache with async sync -> low latency + fresh data [check]
Hint: Async cache sync balances freshness and query speed [check]
Common Mistakes:
4. If the Booking Service loses connectivity to the Calendar Availability Service during booking confirmation, what is the most likely failure mode experienced by the user?
medium
A. The booking is queued locally and confirmed later without user notification
B. The booking request times out and the user sees an error without confirmation
C. The booking is confirmed but the calendar is not updated, causing double bookings
D. The system automatically retries and confirms booking without user delay

Solution

  1. Step 1: Trace failure impact on booking flow

    Loss of connectivity prevents immediate calendar update, risking inconsistency.
  2. Step 2: Analyze failure handling

    Immediate confirmation without calendar update (A) risks double bookings; timeout with error (B) hurts UX; automatic retry without delay (D) is unrealistic under network failure. Local queuing (C) allows eventual consistency with delayed confirmation.
  3. Final Answer:

    Option A -> Option A
  4. Quick Check:

    Local queuing delays confirmation but prevents data loss [check]
Hint: Booking queues locally on calendar service failure, delaying confirmation [check]
Common Mistakes:
5. If the Ticket Server crashes after allocating an ID block but before the client acknowledges, what is the exact failure observed in the system?
medium
A. Clients receive duplicate IDs because the block is reissued without tracking
B. Some ID blocks are lost, causing gaps in the global ID sequence but no duplicates
C. Clients block indefinitely waiting for the Ticket Server response, halting ID generation
D. Clients generate IDs locally without coordination, risking collisions

Solution

  1. Step 1: Understand Ticket Server crash impact

    If the server crashes post-allocation but pre-acknowledgment, it cannot confirm the block was assigned.
  2. Step 2: Trace ID block state

    The allocated block is effectively lost, causing gaps in the global ID sequence, but clients do not reuse lost blocks, preventing duplicates.
  3. Final Answer:

    Option B -> Option B
  4. Quick Check:

    Crash after allocation -> lost ID blocks, no duplicates [check]
Hint: Crash post-allocation loses blocks, no duplicate IDs generated [check]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard