Distributed tracing in HLD - Scalability & System Analysis

Scalability Analysis - Distributed tracing

Growth Table: Distributed Tracing at Different Scales

Users/Requests	Trace Volume	Storage Needs	Processing Load	Visualization Complexity
100 users	Low (few traces/sec)	Small (MBs/day)	Single collector handles	Simple UI, few traces
10K users	Moderate (hundreds traces/sec)	GBs/day	Multiple collectors needed	UI supports filtering, sampling
1M users	High (thousands traces/sec)	TBs/month	Distributed collectors, storage clusters	Advanced querying, sampling, aggregation
100M users	Very High (hundreds K traces/sec)	Petabytes/year	Highly distributed, sharded storage, autoscaling	AI-assisted analysis, anomaly detection

First Bottleneck

The first bottleneck is the trace data ingestion and storage. As user requests grow, the volume of trace data increases rapidly. A single collector or storage node cannot handle the write throughput and storage demands. This causes delays or data loss.

Scaling Solutions

• Horizontal scaling: Add more trace collectors and storage nodes to distribute load.
• Sampling: Collect only a subset of traces to reduce volume.
• Aggregation: Summarize trace data to reduce storage and processing.
• Sharding: Partition trace data by service or time to scale storage.
• Compression: Compress trace data before storage to save space.
• CDN/Edge: Use edge collectors near services to reduce network load.
• Asynchronous processing: Buffer and batch trace data to smooth spikes.

Back-of-Envelope Cost Analysis

Assuming 1M users generating 1000 traces/sec, each trace ~10KB:

• Data ingestion: 1000 traces/sec * 10KB = ~10MB/sec (~80Mbps)
• Storage per day: 10MB/sec * 86400 sec = ~864GB/day
• Storage per month: ~25TB
• Collector servers: Each handles ~5000 traces/sec, so 1 server can handle ingestion load.
• Storage cluster: Needs to support ~10K writes/sec with replication.

Interview Tip

Start by explaining what distributed tracing is and why it matters. Then discuss how trace volume grows with users and requests. Identify the bottleneck in ingestion and storage. Propose scaling solutions like sampling and horizontal scaling. Mention trade-offs like data loss vs cost. Finally, discuss monitoring and alerting for trace system health.

Self Check Question

Your trace database handles 1000 queries per second (QPS). Traffic grows 10x to 10,000 QPS. What do you do first and why?

Answer: Implement sampling to reduce trace volume or add read replicas and shard storage to distribute load. This prevents overload and maintains performance.

Key Result

Distributed tracing scales by managing trace data volume through sampling, horizontal scaling of collectors and storage, and data partitioning to handle increasing user traffic without loss or delay.

Practice

1. What is the main purpose of distributed tracing in a system?
TraceID and spans are used to:

easy

A. Balance load between servers automatically

B. Store user data securely in a database

C. Track requests as they move through multiple services

D. Encrypt communication between clients and servers

Solution

1. Step 1: Understand distributed tracing basics

   Distributed tracing helps track how a request flows through different parts of a system.

2. Step 2: Identify the role of TraceID and spans

   TraceID links all parts of a request, and spans represent each step or service involved.

3. Final Answer:

   Track requests as they move through multiple services -> Option C

4. Quick Check:

   Distributed tracing = Track requests [OK]

Hint: TraceID connects steps in request flow [OK]

Common Mistakes:

• Confusing tracing with data storage
• Mixing tracing with load balancing
• Thinking tracing encrypts data

2. Which of the following is the correct way to represent a span in distributed tracing data?

easy

A. "traceId:abc123;spanId:xyz789;operation:db_query"

B. { "traceId": "abc123", "spanId": "xyz789", "operation": "db_query" }

C. [ "traceId", "spanId", "operation" ]

D. abc123xyz789

Solution

1. Step 1: Identify correct data format for spans

   Spans are usually represented as structured data like JSON objects with keys and values.

2. Step 2: Compare options

   { "traceId": "abc123", "spanId": "xyz789", "operation": "db_query" } shows a JSON object with clear keys for traceId, spanId, and operation, which is standard.

3. Final Answer:

   { "traceId": "abc123", "spanId": "xyz789", "operation": "db_query" } -> Option B

4. Quick Check:

   Span = JSON object with keys [OK]

Hint: Spans use JSON key-value pairs [OK]

Common Mistakes:

• Using arrays instead of objects
• Using plain strings without structure
• Using XML tags incorrectly

3. Given the following trace data snippet:

{"traceId": "t1", "spans": [{"spanId": "s1", "parentId": null, "operation": "frontend"}, {"spanId": "s2", "parentId": "s1", "operation": "backend"}, {"spanId": "s3", "parentId": "s2", "operation": "db_query"}]}

What is the correct order of operations for this request?

medium

A. frontend -> backend -> db_query

B. db_query -> backend -> frontend

C. backend -> frontend -> db_query

D. frontend -> db_query -> backend

Solution

1. Step 1: Analyze parent-child relationships

   The span with parentId null is the root: frontend (s1). Backend (s2) is child of frontend, and db_query (s3) is child of backend.

2. Step 2: Determine operation order

   Request flows from frontend to backend to db_query following parent-child links.

3. Final Answer:

   frontend -> backend -> db_query -> Option A

4. Quick Check:

   Parent-child order = frontend to db_query [OK]

Hint: Follow parentId links from root span [OK]

Common Mistakes:

• Ignoring parentId null as root
• Reversing operation order
• Mixing span order in array with execution order

4. You notice that some spans in your distributed tracing system have missing traceId values. What is the most likely cause?

medium

A. The tracing library failed to propagate the traceId across services

B. The database storing traces is full

C. The network is too slow to send trace data

D. The user did not authenticate properly

Solution

1. Step 1: Understand traceId propagation

   TraceId must be passed along all services handling the request to link spans correctly.

2. Step 2: Identify cause of missing traceId

   If traceId is missing, it usually means the tracing library or instrumentation did not forward it properly.

3. Final Answer:

   The tracing library failed to propagate the traceId across services -> Option A

4. Quick Check:

   Missing traceId = propagation failure [OK]

Hint: Missing traceId means propagation error [OK]

Common Mistakes:

• Blaming storage or network issues
• Confusing authentication with tracing
• Assuming traceId is optional

5. You want to design a distributed tracing system for a microservices architecture with hundreds of services. Which approach best ensures scalability and minimal overhead?

hard

A. Store all trace data locally on each service without aggregation

B. Trace every request fully and store all spans in a single centralized database

C. Send trace data synchronously during each request to avoid data loss

D. Use sampling to trace only a subset of requests and store spans in a scalable backend

Solution

1. Step 1: Consider scalability challenges

   Tracing every request fully creates huge data and overhead, which is not scalable for hundreds of services.

2. Step 2: Evaluate best practices

   Sampling reduces data volume by tracing only some requests, and scalable backends handle storage efficiently.

3. Step 3: Assess other options

   Synchronous sending blocks requests, and local storage prevents global analysis.

4. Final Answer:

   Use sampling to trace only a subset of requests and store spans in a scalable backend -> Option D

5. Quick Check:

   Sampling + scalable storage = scalable tracing [OK]

Hint: Sample requests to reduce tracing load [OK]

Common Mistakes:

• Tracing all requests causing overload
• Synchronous sending blocking services
• Not aggregating trace data centrally