Bird
Raised Fist0

Distributed Tracing System - Trace Context Propagation Across Services

HLDmediumInterview freq: medium
Choose your preparation mode4 modes available
ArchitectureTraceChallenge
🎯
Distributed Tracing System - Trace Context Propagation Across Services
45 min · senior
senior

You are designing a distributed tracing system for a startup with 50 million daily active users (DAU) running a microservices architecture. The system must propagate trace context across services to enable end-to-end latency tracking and debugging. The startup requires trace context propagation with encryption in transit, minimal overhead to service latency (P99 < 200ms), and offline delivery support for asynchronous services. The system must handle high throughput and maintain trace context integrity across heterogeneous services written in multiple languages.

Constraints
  • Trace context must be encrypted in transit to comply with security policies.
  • End-to-end latency overhead for trace propagation must be under 200ms P99.
  • Support offline delivery for asynchronous or batch processing services.
  • Trace context must be propagated reliably across heterogeneous services (polyglot environment).
  • The system must scale to handle 50M DAU generating millions of trace spans per second.
📋
Guided Steps
1/5
Step 1
Trace Context Propagation Format
What format and protocol will you use to propagate trace context across services?
💡 Hint
Consider industry standards that support cross-language compatibility and encryption.
Key Decision
Use W3C Trace Context standard with HTTP headers (traceparent and tracestate) combined with TLS 1.3 for encryption in transit.
Interviewer Expectation
Mid: Names W3C Trace Context and HTTP headers. Senior: Explains why W3C standard ensures interoperability and how TLS 1.3 encrypts headers without custom encryption overhead.
⚠️Common Wrong Answer
Custom binary trace context format without standardization - leads to incompatibility and increased maintenance overhead.
Step 2
Trace Context Propagation Mechanism
How will services propagate trace context synchronously and asynchronously while maintaining low latency?
💡 Hint
Think about in-process propagation and cross-process propagation for async calls.
Key Decision
Use OpenTelemetry SDK for in-process context propagation and inject/extract trace context into HTTP/gRPC headers for synchronous calls; for async, embed trace context in Kafka message headers.
Interviewer Expectation
Mid: Mentions OpenTelemetry SDK and HTTP headers. Senior: Explains Kafka headers for async propagation and how this avoids context loss in offline delivery.
⚠️Common Wrong Answer
Passing trace context only via HTTP headers - breaks async trace continuity and offline delivery.
Step 3
Trace Span Collection and Storage
How will you collect and store trace spans at scale for 50M DAU with millions of spans per second?
💡 Hint
Consider scalable ingestion pipelines and storage optimized for time-series and querying.
Key Decision
Use Kafka as the ingestion buffer with topic partitioned by trace_id for ordering, and store spans in Cassandra with partition key=trace_id for efficient range queries and linear write scalability.
Interviewer Expectation
Mid: Names Kafka and Cassandra. Senior: Explains Kafka partitioning by trace_id ensures ordered span ingestion, Cassandra LSM-tree writes scale linearly, and range queries enable efficient trace reconstruction.
⚠️Common Wrong Answer
Using relational DB for span storage - causes write bottlenecks and poor scalability at high throughput.
Step 4
Trace Context Encryption and Security
How will you ensure trace context is encrypted and secure without adding significant latency?
💡 Hint
Focus on transport layer encryption and minimal cryptographic overhead.
Key Decision
Use TLS 1.3 for all inter-service communication to encrypt trace context headers; avoid application-level encryption to minimize latency and complexity.
Interviewer Expectation
Mid: Mentions TLS encryption. Senior: Explains TLS 1.3 reduces handshake latency and offloads encryption to network layer, preserving P99 latency SLA.
⚠️Common Wrong Answer
Encrypting trace context at application layer with custom crypto - adds CPU overhead and increases latency beyond 200ms P99.
Step 5
API Design for Trace Context Propagation
Design 3 core API endpoints for trace context propagation and span reporting.
💡 Hint
Consider endpoints for context injection, extraction, and span submission.
Key Decision
Define: (1) InjectTraceContext(service_call) - injects traceparent header; (2) ExtractTraceContext(inbound_call) - extracts trace context from headers; (3) SubmitSpan(span_data) - asynchronously submits completed spans to backend via Kafka producer.
Interviewer Expectation
Mid: Lists endpoints with basic functionality. Senior: Details payload formats, asynchronous span submission via Kafka producer, and error handling for context extraction failures.
⚠️Common Wrong Answer
No separation between context propagation and span reporting APIs - leads to tight coupling and poor scalability.
🔌
API Design Challenge
Sketch 3 core API endpoints for the distributed tracing system focusing on trace context propagation.
Hints
  • How does a service send trace context when making an outbound call?
  • How does a service receive and extract trace context from an inbound call?
  • How does a service report completed trace spans back to the tracing backend?
📊
Scoring Rubric
▼ Show
Must Have - 60%
  • Use W3C Trace Context standard with TLS 1.3 for secure propagation.
  • Use OpenTelemetry SDK for in-process context management.
  • Kafka with trace_id partitioning for scalable span ingestion.
  • Cassandra with trace_id partition key for efficient storage and querying.
  • API design separating context propagation and span reporting.
Good to Have - 30%
  • Explain async trace context propagation using Kafka headers.
  • Discuss TLS 1.3 benefits for latency and security.
  • Describe error handling for missing or corrupted trace context.
Bonus - 10%
  • Propose sampling strategies to reduce overhead at extreme scale.
  • Suggest integration with existing monitoring and alerting tools.
Anti-patterns to Avoid
Custom binary trace context format causes incompatibility and maintenance burden due to lack of standardization.
Passing trace context only via HTTP headers without async support breaks trace continuity in asynchronous workflows and offline delivery.
Using relational databases for span storage leads to write bottlenecks and poor scalability at high throughput, causing system slowdowns.
Application-level encryption of trace context adds CPU overhead and violates latency SLA, increasing P99 latency beyond acceptable limits.
Combining trace context propagation and span reporting in a single API leads to tight coupling and poor scalability, making maintenance and evolution harder.
✅
Model Answer
▼ Reveal

This design uses the W3C Trace Context standard with HTTP headers and TLS 1.3 to securely propagate trace context across heterogeneous microservices. OpenTelemetry SDK manages in-process context propagation. Kafka partitions trace spans by trace_id for ordered ingestion, and Cassandra stores spans efficiently with trace_id as partition key. Asynchronous trace context propagation uses Kafka message headers to maintain continuity. The API design cleanly separates context injection/extraction from span reporting to ensure scalability and maintainability.

Key Decisions
  • Adopt W3C Trace Context standard with HTTP headers (traceparent, tracestate) for interoperability
  • Use TLS 1.3 to encrypt all inter-service communication including trace context headers
  • Leverage OpenTelemetry SDK for in-process context propagation and standard injection/extraction APIs
  • Partition Kafka topics by trace_id to guarantee ordered span ingestion at scale
  • Store spans in Cassandra with partition key=trace_id for linear write scalability and efficient range queries
  • Embed trace context in Kafka message headers for asynchronous propagation to support offline delivery
  • Design APIs to separate synchronous context propagation (InjectTraceContext, ExtractTraceContext) from asynchronous span reporting (SubmitSpan)
  • Avoid application-level encryption to meet strict P99 latency SLA
  • W3C Trace Context standard with HTTP headers ensures cross-language compatibility and industry adoption.
  • OpenTelemetry SDK provides in-process context propagation and standard APIs for injection/extraction.
  • Kafka as ingestion buffer with trace_id partitioning guarantees ordered span delivery and scales to millions of spans/sec.
  • Cassandra with partition key=trace_id supports linear write scalability and efficient range queries for trace reconstruction.
  • TLS 1.3 encrypts all inter-service communication including trace context headers with minimal latency overhead.
  • Kafka headers used for asynchronous trace context propagation in message queues to maintain trace continuity.
  • API endpoints separate concerns: context injection/extraction for synchronous calls and asynchronous span submission.
  • Avoid application-level encryption of trace context to meet strict P99 latency SLA.

Practice

(1/5)
1. A microservice calls a downstream service that occasionally fails or responds slowly. To prevent cascading failures and allow quick recovery, which component should be introduced?
easy
A. Implement a circuit breaker to stop calls after failure threshold is reached
B. Use a retry mechanism with exponential backoff on every request
C. Add a load balancer to distribute requests evenly across instances
D. Use a message queue to buffer all requests asynchronously

Solution

  1. Step 1: Identify failure containment mechanism

    Retries alone (A) can worsen cascading failures by increasing load; load balancers (C) distribute load but don't prevent failures; message queues (D) buffer but don't stop calls to failing services.

  2. Step 2: Recognize circuit breaker role

    Circuit breakers (B) detect failure thresholds and open to stop calls, preventing cascading failures and allowing recovery.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Circuit breaker stops calls after failures to prevent cascade [check]
Hint: Circuit breaker stops calls after failures to prevent cascade [OK]
Common Mistakes:
2. In a token bucket rate limiter, at step 4 the Request Handler checks the token count before processing a request. What happens if the bucket is empty at this step?
easy
A. The token count is reset to maximum and the request proceeds
B. The request is immediately processed and the token count goes negative
C. The request is rejected or delayed until tokens are refilled
D. The request is queued indefinitely without token consumption

Solution

  1. Step 1: Trace Request Handler token check at Step 4

    If the token bucket is empty, no tokens are available to allow immediate processing.

  2. Step 2: Determine request fate

    The request must be rejected or delayed until tokens refill; processing without tokens violates rate limits.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Empty bucket -> reject or delay request [check]
Hint: No tokens? Delay or reject request [OK]
Common Mistakes:
3. If the Kafka cluster used for transaction ingestion fails completely, what is the immediate failure mode experienced by the fraud detection system?
medium
A. The scoring service continues with stale transactions, causing false negatives
B. The ingestion service blocks on Kafka produce calls, causing transaction ingestion to stall
C. The rule engine stops evaluating rules due to missing transaction data
D. Transactions are lost silently because the ingestion service does not retry

Solution

  1. Step 1: Trace ingestion failure impact

    Kafka failure causes ingestion produce calls to fail or block, but ingestion service typically retries or buffers (A and B partially true but not immediate silent loss or indefinite blocking).

  2. Step 2: Trace downstream impact

    Scoring service consumes from Kafka; if Kafka is down, no new transactions arrive, so scoring uses stale data, increasing false negatives.

  3. Step 3: Rule engine impact

    Rule engine depends on scoring output, not directly on Kafka; it continues evaluating existing data.

  4. Final Answer:

    Option A -> Option A

  5. Quick Check:

    Kafka down -> no new data -> scoring uses stale data -> false negatives [check]
Hint: Kafka failure stalls ingestion; scoring uses stale data causing misses [check]
Common Mistakes:
4. For a high-read, low-write system, which caching strategy best balances freshness and performance?
medium
A. Use write-through cache to update cache synchronously on every write
B. Disable caching and rely on database read replicas for freshness
C. Use write-back cache to batch writes and update cache asynchronously
D. Use cache-aside pattern where application loads cache on demand and invalidates on writes

Solution

  1. Step 1: Analyze write frequency and freshness needs

    High-read, low-write favors lazy loading and invalidation to avoid cache update overhead.

  2. Step 2: Evaluate caching strategies

    Write-through adds write latency; write-back risks stale data; cache-aside balances freshness and performance.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Cache-aside suits high-read, low-write with freshness [check]
Hint: Cache-aside balances freshness and performance [check]
Common Mistakes:
5. At 10x traffic scale, which component in a Layer 7 load balancing architecture is most likely to become the bottleneck first, and why?
hard
A. Layer 7 load balancer CPU, due to expensive HTTP header parsing and SSL termination
B. Network switches, because increased packet forwarding saturates bandwidth
C. Backend servers, because they cannot handle increased HTTP request volume
D. DNS servers, because client-side resolution delays increase exponentially

Solution

  1. Step 1: Identify Layer 7 load balancer workload at scale

    Layer 7 load balancers perform CPU-intensive tasks like HTTP header parsing and SSL termination, which scale poorly with traffic.

  2. Step 2: Compare with other components

    Backend servers can be scaled horizontally; network switches handle packet forwarding efficiently; DNS servers are less impacted by traffic volume.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Layer 7 LB CPU bottleneck due to HTTP parsing and SSL termination [check]
Hint: Layer 7 LB CPU chokes first at high traffic scale [check]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard