Bird
Raised Fist0

Distributed Tracing System - Trace Context Propagation Across Services

HLDmediumInterview freq: medium
Choose your preparation mode4 modes available
ArchitectureTraceChallenge
▶
Request Trace
Step 1/6
Trace Context Injector (SDK)

Inject trace context into outgoing request headers

Adds trace ID and span ID to HTTP headers before sending

Protocol:HTTP/REST
BeforeRequest has no trace context attached
AfterRequest headers contain trace context identifiers
Kafka Cluster

Receive trace spans from services and enqueue them

Trace spans are published to Kafka topic partitioned by trace ID

Protocol:Kafka Protocol
BeforeSpan data is generated but not yet durably stored
AfterSpan data is durably committed to Kafka topic
Span Processor

Consume spans from Kafka and enrich with metadata

Adds timing, tags, and error info before storage

Protocol:Kafka Protocol (consume) and CQL (write)
BeforeRaw spans are in Kafka, unprocessed
AfterEnriched spans are ready for storage in Cassandra
Storage Cluster (Apache Cassandra)

Store enriched spans in distributed database

Writes spans partitioned by trace ID for efficient queries

Protocol:CQL
BeforeEnriched spans are in memory at Span Processor
AfterSpans are durably stored and indexed in Cassandra
Trace Query API

Query stored spans based on trace ID or filters

Fetches spans from Cassandra and formats response

Protocol:HTTP/REST
BeforeTrace data is stored but not yet queried
AfterTrace data is retrieved and ready for UI consumption
Trace Visualization UI

Render trace data for user visualization

Displays spans, timing, and service dependencies

Protocol:HTTP/REST
BeforeTrace data is available from API response
AfterUser sees detailed trace visualization in UI
🏗️
Architecture Diagram
Architecture Diagram
┌───────────────────────────────┐
│ Trace Context Injector (SDK)  │
└───────────────┬───────────────┘
                │ HTTP/REST (Inject trace ctx)
                ▼
┌───────────────────────────────┐
│ Kafka Cluster                 │
│ (Distributed Msg Queue)       │
└───────────────┬───────────────┘
                │ Kafka Protocol
                ▼
┌───────────────────────────────┐
│ Span Processor                │
│ (Processes and enriches spans)│
└───────────────┬───────────────┘
                │ CQL (Cassandra Query Lang)
                ▼
┌───────────────────────────────┐
│ Storage Cluster               │
│ (Apache Cassandra DB)         │
└───────────────┬───────────────┘
                │ HTTP/REST (Query spans)
                ▼
┌───────────────────────────────┐
│ Trace Query API               │
│ (Serves trace data to UI)     │
└───────────────┬───────────────┘
                │ HTTP/REST
                ▼
┌───────────────────────────────┐
│ Trace Visualization UI       │
│ (Displays trace data)         │
└───────────────────────────────┘
Design Rationale

This distributed tracing system propagates trace context via the Trace Context Injector SDK, which attaches trace identifiers to outgoing requests. These spans are asynchronously sent to the Kafka Cluster, a durable distributed message queue that partitions spans by trace ID to maintain ordering. The Span Processor consumes spans from Kafka, enriches them with metadata such as timing and error tags, and writes them to the Storage Cluster backed by Apache Cassandra for scalable, fault-tolerant storage. The Trace Query API serves stored trace data to the Trace Visualization UI, enabling users to explore detailed traces and service dependencies. This vertical pipeline ensures trace context is preserved end-to-end, enabling reliable and scalable trace collection and querying.

Primary Focus

✓

Practice

(1/5)
1. If the event ingestion pipeline fails to persist incoming metric data, what is the exact failure observed in the alerting system?
medium
A. Threshold and anomaly detectors receive incomplete data, causing missed or false alerts
B. Alerts are generated with delayed timestamps but eventually delivered
C. Composite alerts continue to function normally using cached inputs
D. The system switches to a backup data source automatically without alert impact

Solution

  1. Step 1: Role of event ingestion pipeline

    This pipeline persists raw metric data for downstream alerting components.

  2. Step 2: Consequence of pipeline failure

    Without persistence, threshold and anomaly detectors get incomplete or missing data, leading to missed alerts or false positives.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Missing data -> detectors miss or misfire alerts [check]
Hint: No data persistence -> detectors get incomplete data, miss alerts [check]
Common Mistakes:
2. For a CDN serving frequently updated dynamic content, which origin content delivery strategy is best to ensure freshness without overloading the origin server?
medium
A. Pull model where edge nodes fetch content from origin on cache miss with TTL-based caching
B. Push model where origin proactively sends all content updates to edge nodes
C. Push model with edge nodes polling origin periodically for updates
D. Pull model with edge nodes never caching content and always fetching from origin

Solution

  1. Step 1: Analyze push vs pull for dynamic content freshness

    Push model can overload origin with frequent updates; polling adds unnecessary load.

  2. Step 2: Evaluate pull with TTL caching

    Pull with TTL balances freshness and origin load by caching briefly and refreshing on expiry.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Pull with TTL caching balances freshness and origin load [check]
Hint: Pull with TTL balances freshness and load [check]
Common Mistakes:
3. For delivering notifications to mobile devices at 10M+ scale, which architecture better handles intermittent connectivity and battery constraints?
medium
A. Use push notification services (APNs, FCM) with message TTL and retries managed by the platform
B. Use persistent gRPC streaming connections from server to device for real-time delivery
C. Use long polling HTTP connections from device to server to fetch notifications
D. Use SMS as the primary delivery channel for all notifications

Solution

  1. Step 1: Evaluate persistent connection approaches

    gRPC streaming (A) is unreliable on mobile due to network changes and battery constraints; long polling (C) wastes battery and bandwidth.

  2. Step 2: Consider platform push services and SMS

    Push services (B) handle retries and offline buffering efficiently; SMS (D) is costly and not suitable for all notifications.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Platform push services manage retries and offline buffering [check]
Hint: Use platform push services, not persistent streams on mobile [check]
Common Mistakes:
4. For a high-traffic API with strict memory limits and tolerance for bursty traffic, which rate limiting approach is best and why?
medium
A. Leaky bucket, because it enforces a steady output rate regardless of bursts
B. Sliding window counter, because it tracks exact request times for precise limits
C. Token bucket, because it smooths bursts and uses fixed memory for tokens
D. Fixed window counter, because it uses constant memory and handles bursts at window edges

Solution

  1. Step 1: Analyze memory constraints and burst tolerance

    Sliding window counters require storing timestamps, which grows with request volume, unsuitable for strict memory limits.

  2. Step 2: Fixed window counters use constant memory by counting requests per fixed interval, allowing bursts at window edges but fitting memory constraints

    Token and leaky buckets smooth bursts but require token state management and may not tolerate bursts as well.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Fixed window uses constant memory, tolerates bursts, fits strict memory limits [check]
Hint: Fixed window uses constant memory, sliding window stores timestamps [check]
Common Mistakes:
5. At 10x traffic scale during a flash sale, which component is most likely to fail first and why?
hard
A. The relational database because it cannot handle 10x write throughput without sharding
B. The API Gateway because it maintains stateful user sessions and cannot horizontally scale easily
C. The cache layer because TTL expiration floods the database with cache misses
D. The inventory service because distributed locking causes contention and latency spikes

Solution

  1. Step 1: Identify stateful components under load

    API Gateway maintains user session state (e.g., WebSocket connections), which are sticky and stateful, limiting horizontal scaling.

  2. Step 2: Compare other components

    Relational DB (A) can be sharded; cache (C) TTL floods are mitigated by pre-warming; inventory locking (D) is optimized with partitioning. API Gateway statefulness causes connection limits and bottlenecks first.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    API Gateway stateful sessions limit horizontal scaling [check]
Hint: Stateful API Gateway limits scaling under 10x load [check]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard