Distributed tracing in HLD - Architecture Diagram

System Overview - Distributed tracing

Distributed tracing helps track a user's request as it moves through many services in a system. It shows where delays happen and helps find errors quickly. The system must collect trace data from services, store it, and let users search and view traces easily.

Architecture Diagram

User
  |
  v
Load Balancer
  |
  v
API Gateway
  |
  v
+-------------------+       +----------------+
|   Service A       |<----->| Trace Collector |
+-------------------+       +----------------+
          |
          v
+-------------------+
|   Service B       |
+-------------------+
          |
          v
+-------------------+
|   Service C       |
+-------------------+
          |
          v
+-------------------+
| Distributed Storage|
+-------------------+
          |
          v
+-------------------+
| Trace Query Engine |
+-------------------+
          |
          v
User Interface (UI)

Components

User

user

Initiates requests and views trace results

Load Balancer

load_balancer

Distributes incoming requests to API Gateway

API Gateway

api_gateway

Entry point for requests, routes to services

Service A

service

Handles part of the request and generates trace data

Service B

service

Processes request further and generates trace data

Service C

service

Final service in chain, generates trace data

Trace Collector

service

Receives trace data from services and forwards to storage

Distributed Storage

database

Stores trace data for querying and analysis

Trace Query Engine

service

Processes user queries to retrieve trace data

User Interface (UI)

frontend

Allows users to search and view distributed traces

Request Flow - 12 Hops

User → Load Balancer

Load Balancer → API Gateway

API Gateway → Service A

Service A → Trace Collector

Service A → Service B

Service B → Trace Collector

Service B → Service C

Service C → Trace Collector

Trace Collector → Distributed Storage

User → Trace Query Engine

Trace Query Engine → Distributed Storage

Trace Query Engine → User Interface (UI)

Failure Scenario

Component Fails:Trace Collector

Impact:Trace data from services is lost or delayed, making tracing incomplete or unavailable

Mitigation:Use multiple Trace Collector instances with load balancing and buffering at services to retry sending trace data

Architecture Quiz - 3 Questions

Which component is responsible for storing the collected trace data?

A Distributed Storage

B Trace Collector

C API Gateway

D Load Balancer

Design Principle

This architecture shows how distributed tracing collects data from multiple services asynchronously to avoid slowing down user requests. It uses a centralized storage and query engine to let users explore traces, helping find performance issues across complex systems.

Practice

1. What is the main purpose of distributed tracing in a system?
TraceID and spans are used to:

easy

A. Balance load between servers automatically

B. Store user data securely in a database

C. Track requests as they move through multiple services

D. Encrypt communication between clients and servers

Solution

1. Step 1: Understand distributed tracing basics

   Distributed tracing helps track how a request flows through different parts of a system.

2. Step 2: Identify the role of TraceID and spans

   TraceID links all parts of a request, and spans represent each step or service involved.

3. Final Answer:

   Track requests as they move through multiple services -> Option C

4. Quick Check:

   Distributed tracing = Track requests [OK]

Hint: TraceID connects steps in request flow [OK]

Common Mistakes:

• Confusing tracing with data storage
• Mixing tracing with load balancing
• Thinking tracing encrypts data

2. Which of the following is the correct way to represent a span in distributed tracing data?

easy

A. "traceId:abc123;spanId:xyz789;operation:db_query"

B. { "traceId": "abc123", "spanId": "xyz789", "operation": "db_query" }

C. [ "traceId", "spanId", "operation" ]

D. abc123xyz789

Solution

1. Step 1: Identify correct data format for spans

   Spans are usually represented as structured data like JSON objects with keys and values.

2. Step 2: Compare options

   { "traceId": "abc123", "spanId": "xyz789", "operation": "db_query" } shows a JSON object with clear keys for traceId, spanId, and operation, which is standard.

3. Final Answer:

   { "traceId": "abc123", "spanId": "xyz789", "operation": "db_query" } -> Option B

4. Quick Check:

   Span = JSON object with keys [OK]

Hint: Spans use JSON key-value pairs [OK]

Common Mistakes:

• Using arrays instead of objects
• Using plain strings without structure
• Using XML tags incorrectly

3. Given the following trace data snippet:

{"traceId": "t1", "spans": [{"spanId": "s1", "parentId": null, "operation": "frontend"}, {"spanId": "s2", "parentId": "s1", "operation": "backend"}, {"spanId": "s3", "parentId": "s2", "operation": "db_query"}]}

What is the correct order of operations for this request?

medium

A. frontend -> backend -> db_query

B. db_query -> backend -> frontend

C. backend -> frontend -> db_query

D. frontend -> db_query -> backend

Solution

1. Step 1: Analyze parent-child relationships

   The span with parentId null is the root: frontend (s1). Backend (s2) is child of frontend, and db_query (s3) is child of backend.

2. Step 2: Determine operation order

   Request flows from frontend to backend to db_query following parent-child links.

3. Final Answer:

   frontend -> backend -> db_query -> Option A

4. Quick Check:

   Parent-child order = frontend to db_query [OK]

Hint: Follow parentId links from root span [OK]

Common Mistakes:

• Ignoring parentId null as root
• Reversing operation order
• Mixing span order in array with execution order

4. You notice that some spans in your distributed tracing system have missing traceId values. What is the most likely cause?

medium

A. The tracing library failed to propagate the traceId across services

B. The database storing traces is full

C. The network is too slow to send trace data

D. The user did not authenticate properly

Solution

1. Step 1: Understand traceId propagation

   TraceId must be passed along all services handling the request to link spans correctly.

2. Step 2: Identify cause of missing traceId

   If traceId is missing, it usually means the tracing library or instrumentation did not forward it properly.

3. Final Answer:

   The tracing library failed to propagate the traceId across services -> Option A

4. Quick Check:

   Missing traceId = propagation failure [OK]

Hint: Missing traceId means propagation error [OK]

Common Mistakes:

• Blaming storage or network issues
• Confusing authentication with tracing
• Assuming traceId is optional

5. You want to design a distributed tracing system for a microservices architecture with hundreds of services. Which approach best ensures scalability and minimal overhead?

hard

A. Store all trace data locally on each service without aggregation

B. Trace every request fully and store all spans in a single centralized database

C. Send trace data synchronously during each request to avoid data loss

D. Use sampling to trace only a subset of requests and store spans in a scalable backend

Solution

1. Step 1: Consider scalability challenges

   Tracing every request fully creates huge data and overhead, which is not scalable for hundreds of services.

2. Step 2: Evaluate best practices

   Sampling reduces data volume by tracing only some requests, and scalable backends handle storage efficiently.

3. Step 3: Assess other options

   Synchronous sending blocks requests, and local storage prevents global analysis.

4. Final Answer:

   Use sampling to trace only a subset of requests and store spans in a scalable backend -> Option D

5. Quick Check:

   Sampling + scalable storage = scalable tracing [OK]

Hint: Sample requests to reduce tracing load [OK]

Common Mistakes:

• Tracing all requests causing overload
• Synchronous sending blocking services
• Not aggregating trace data centrally