Correlation IDs in Microservices - System Design Exercise

Design: Correlation ID Tracking in Microservices

Design the mechanism to generate, propagate, and log correlation IDs across microservices. Out of scope: detailed implementation of each microservice business logic.

Functional Requirements

FR1: Assign a unique correlation ID to each client request entering the system

FR2: Propagate the correlation ID through all microservices involved in processing the request

FR3: Log the correlation ID with all service logs related to the request

FR4: Allow tracing and debugging of requests across multiple services using the correlation ID

FR5: Support asynchronous communication patterns (e.g., message queues) with correlation ID propagation

FR6: Ensure minimal performance overhead when generating and passing correlation IDs

Non-Functional Requirements

NFR1: System must handle 10,000 concurrent requests with correlation tracking

NFR2: Correlation ID propagation latency should not add more than 5ms per service hop

NFR3: Logging with correlation IDs must not degrade overall system availability below 99.9%

NFR4: Correlation IDs must be globally unique and collision-resistant

NFR5: Support both synchronous HTTP calls and asynchronous messaging

Think Before You Design

Questions to Ask

❓ Question 1

❓ Question 2

❓ Question 3

❓ Question 4

❓ Question 5

❓ Question 6

Key Components

API Gateway or Edge Service for initial correlation ID generation

Middleware or Interceptors in each microservice to propagate and log correlation IDs

Logging system that supports structured logs with correlation ID fields

Message brokers or queues with headers to carry correlation IDs

Tracing tools or distributed tracing systems (optional)

Design Patterns

Request Context Propagation

Distributed Tracing

Correlation ID Injection via Middleware

Log Enrichment with Correlation IDs

Message Header Propagation

Reference Architecture

Client
  |
  v
API Gateway (generates correlation ID)
  |
  v
Microservice A <--> Microservice B <--> Microservice C
  |            |            |
  v            v            v
Logging System (logs with correlation ID)

Message Queue (with correlation ID in headers)
  ^
  |
Microservice D (consumes messages with correlation ID)

Components

API Gateway

Nginx/Envoy/Custom Gateway

Generate unique correlation IDs for incoming requests and inject them into request headers

Microservice Middleware

HTTP Middleware / gRPC Interceptors

Extract, propagate, and log correlation IDs for each request or message

Logging System

ELK Stack / Fluentd / Cloud Logging

Store structured logs enriched with correlation IDs for traceability

Message Broker

Kafka / RabbitMQ / AWS SQS

Carry correlation IDs in message headers for asynchronous propagation

Request Flow

1. Client sends request to API Gateway

2. API Gateway generates a unique correlation ID if missing and adds it to request headers

3. Request forwarded to Microservice A with correlation ID in headers

4. Microservice A middleware extracts correlation ID, logs it, and forwards request to Microservice B

5. Microservice B repeats extraction, logging, and propagation to Microservice C

6. If Microservice C sends an asynchronous message, it includes the correlation ID in message headers

7. Microservice D consumes the message, extracts correlation ID, and logs processing with the same ID

8. All logs from services include the correlation ID, enabling tracing of the full request path

Database Schema

No specific database schema required for correlation IDs as they are transient metadata passed in headers and logs. However, logging storage schema should include a 'correlation_id' field indexed for fast querying.

Scaling Discussion

Bottlenecks

High volume of logs with correlation IDs can overwhelm logging infrastructure

Propagation overhead may increase latency if correlation ID handling is inefficient

Message brokers may drop or lose correlation ID headers if not configured properly

Inconsistent correlation ID formats can cause tracing failures

Solutions

Use efficient, asynchronous logging with batching and compression to handle log volume

Implement lightweight middleware for correlation ID handling to minimize latency

Configure message brokers to preserve headers and validate correlation ID presence

Standardize correlation ID format (e.g., UUID v4) and validate on entry points

Interview Tips

Time: Spend 10 minutes clarifying requirements and constraints, 20 minutes designing the architecture and data flow, 10 minutes discussing scaling and trade-offs, and 5 minutes summarizing.

Explain the importance of correlation IDs for debugging and tracing in microservices

Describe how and where correlation IDs are generated and propagated

Discuss synchronous and asynchronous propagation challenges

Highlight integration with logging and monitoring systems

Address performance and scalability considerations

Mention industry best practices and standards for correlation IDs

Practice

1. What is the primary purpose of a Correlation ID in microservices?

easy

A. To balance load between servers

B. To encrypt data between services

C. To track a single request across multiple services for easier debugging

D. To store user session information

Solution

1. Step 1: Understand the role of Correlation ID

   A Correlation ID is a unique identifier attached to a request that travels through multiple services.

2. Step 2: Identify its main use

   This ID helps developers trace and debug the flow of that request across distributed systems.

3. Final Answer:

   To track a single request across multiple services for easier debugging -> Option C

4. Quick Check:

   Correlation ID = request tracking [OK]

Hint: Correlation ID links logs of one request across services [OK]

Common Mistakes:

• Confusing Correlation ID with encryption keys
• Thinking it balances load
• Assuming it stores user data

2. Which of the following is the correct way to pass a Correlation ID between microservices?

easy

A. Add it as a custom HTTP header in the request

B. Include it as a query parameter in the URL

C. Store it in a database before each call

D. Embed it inside the response body

Solution

1. Step 1: Review common practices for passing metadata

   Metadata like Correlation IDs are typically passed in HTTP headers to keep requests clean and consistent.

2. Step 2: Evaluate options

   Query parameters can be altered or logged insecurely; storing in DB is inefficient; response body is too late for tracking.

3. Final Answer:

   Add it as a custom HTTP header in the request -> Option A

4. Quick Check:

   Correlation ID in headers = best practice [OK]

Hint: Use HTTP headers to pass Correlation IDs [OK]

Common Mistakes:

• Using query parameters which can be insecure
• Storing IDs in database for each call
• Embedding IDs in response body instead of request

3. Consider this simplified code snippet in a microservice receiving an HTTP request:

def handle_request(request):
    correlation_id = request.headers.get('X-Correlation-ID')
    log(f"Start processing request {correlation_id}")
    # ... process ...
    log(f"End processing request {correlation_id}")

What will be logged if the incoming request has header X-Correlation-ID: abc123?

medium

A. No logs will be generated

B. Start processing request abc123
End processing request abc123

C. Start processing request X-Correlation-ID
End processing request X-Correlation-ID

D. Start processing request None
End processing request None

Solution

1. Step 1: Extract Correlation ID from headers

   The code uses request.headers.get('X-Correlation-ID') which returns the header value if present.

2. Step 2: Check the header value in the request

   The request has X-Correlation-ID: abc123, so correlation_id will be 'abc123'.

3. Final Answer:

   Start processing request abc123 End processing request abc123 -> Option B

4. Quick Check:

   Header value read correctly = logs with abc123 [OK]

Hint: Headers.get returns value or None; here value exists [OK]

Common Mistakes:

• Assuming header key is logged instead of value
• Thinking None is logged when header exists
• Believing no logs are generated

4. A developer notices that Correlation IDs are missing in logs of downstream services. Which is the most likely cause?

medium

A. The Correlation ID header is not forwarded in outgoing requests

B. The Correlation ID is too long and gets truncated

C. The logging system does not support string messages

D. The services are using different programming languages

Solution

1. Step 1: Understand Correlation ID propagation

   Correlation IDs must be passed along with every outgoing request to maintain traceability.

2. Step 2: Identify common propagation mistake

   If downstream logs miss the ID, it usually means the header was not forwarded properly.

3. Final Answer:

   The Correlation ID header is not forwarded in outgoing requests -> Option A

4. Quick Check:

   Missing ID in logs = header not forwarded [OK]

Hint: Always forward Correlation ID header downstream [OK]

Common Mistakes:

• Blaming length truncation without evidence
• Assuming logging system limitation
• Thinking language differences cause missing IDs

5. You design a microservices system where each service generates its own Correlation ID if none is provided. What is a potential problem with this approach?

hard

A. It ensures better security by hiding original IDs

B. It improves performance by reducing header size

C. It simplifies logging by having unique IDs per service

D. It breaks the traceability because multiple IDs exist for the same request

Solution

1. Step 1: Analyze the effect of generating new IDs per service

   If each service creates a new Correlation ID, the original request's trace is lost.

2. Step 2: Understand impact on traceability

   Multiple different IDs for one request make it impossible to follow the full request path across services.

3. Final Answer:

   It breaks the traceability because multiple IDs exist for the same request -> Option D

4. Quick Check:

   Multiple IDs = broken traceability [OK]

Hint: One Correlation ID per request across services only [OK]

Common Mistakes:

• Thinking multiple IDs improve security
• Assuming performance improves with new IDs
• Believing unique IDs per service simplify logs