Three pillars (metrics, logs, traces) in Microservices - System Design Guide

Problem Statement

When microservices fail or behave unexpectedly, teams struggle to find the root cause quickly because data about system behavior is scattered or incomplete. Without a clear way to observe system health and diagnose issues, outages last longer and degrade user experience.

Solution

The three pillars—metrics, logs, and traces—work together to provide a complete picture of system health and behavior. Metrics give numeric summaries of system performance over time, logs record detailed events and errors, and traces show the path of requests across services. Together, they enable fast detection, diagnosis, and resolution of problems.

Architecture

Microservice

Instance

→Metrics DB

↓

Tracing

Instrumentation

→Trace Storage

This diagram shows how microservices emit metrics, logs, and traces to their respective storage systems. Dashboards and alerting tools consume this data to provide observability.

Trade-offs

✓ Pros

→

Provides comprehensive observability by combining numeric data, detailed events, and request flows.

→

Enables faster root cause analysis by correlating metrics spikes with logs and traces.

→

Supports proactive alerting and capacity planning through metrics.

→

Improves understanding of distributed system behavior with traces.

✗ Cons

→

Requires additional infrastructure and storage for three different data types.

→

Increases complexity in data collection, processing, and correlation.

→

Needs careful design to avoid high overhead and data noise.

Use when operating distributed microservices at scale with complex interactions and the need for fast incident response and performance monitoring.

Avoid when running simple, monolithic applications with low traffic where the overhead of collecting and managing all three data types outweighs benefits.

Real World Examples

Netflix

Uses metrics to monitor streaming quality, logs for error details, and distributed tracing to track user requests across microservices for quick troubleshooting.

Uber

Combines metrics, logs, and traces to monitor ride requests and driver matching services, enabling rapid detection and resolution of latency issues.

Amazon

Employs the three pillars to maintain high availability of its e-commerce platform by correlating system health metrics with logs and traces from thousands of microservices.

Alternatives

Single pillar monitoring

Focuses on only one type of observability data, such as logs only or metrics only.

Use when: Use only metrics or logs when system complexity is low and full observability is not required.

Event-driven monitoring

Relies primarily on events and alerts rather than continuous metrics and traces.

Use when: Choose when system events are rare but critical, and detailed tracing is unnecessary.

Summary

The three pillars of observability are metrics, logs, and traces, each providing unique insights into system behavior.

Together, they enable fast detection and diagnosis of issues in complex microservices environments.

Implementing all three requires careful design to balance observability benefits with system overhead.

Practice

1. Which of the following best describes the role of metrics in microservices monitoring?

easy

A. They track the path of a request through multiple services.

B. They record detailed events and errors in the system.

C. They provide numerical data about system performance over time.

D. They store configuration settings for microservices.

Solution

1. Step 1: Understand what metrics represent

   Metrics are numerical measurements like CPU usage, request counts, or latency that show system health over time.

2. Step 2: Differentiate metrics from logs and traces

   Logs record events, traces follow request paths, but metrics summarize performance data.

3. Final Answer:

   They provide numerical data about system performance over time. -> Option C

4. Quick Check:

   Metrics = numerical performance data [OK]

Hint: Metrics = numbers about performance, not events or paths [OK]

Common Mistakes:

• Confusing metrics with logs as event records
• Thinking traces are numerical data
• Assuming metrics store configurations

2. Which syntax correctly represents a log entry in a microservice system?

easy

A. [2024-06-01 12:00:00] ERROR Failed to connect

B. {"timestamp": "2024-06-01T12:00:00Z", "level": "ERROR", "message": "Failed to connect"}

C. Failed to connect

D. ERROR 2024-06-01T12:00:00Z Failed to connect

Solution

1. Step 1: Identify standard log formats

   JSON format is widely used for structured logs in microservices for easy parsing and querying.

2. Step 2: Compare options for correctness

   {"timestamp": "2024-06-01T12:00:00Z", "level": "ERROR", "message": "Failed to connect"} is a valid JSON log entry with timestamp, level, and message fields. Others are less structured or not JSON.

3. Final Answer:

   {"timestamp": "2024-06-01T12:00:00Z", "level": "ERROR", "message": "Failed to connect"} -> Option B

4. Quick Check:

   Structured JSON logs = {"timestamp": "2024-06-01T12:00:00Z", "level": "ERROR", "message": "Failed to connect"} [OK]

Hint: Logs are best as structured JSON for easy use [OK]

Common Mistakes:

• Using unstructured plain text logs
• Confusing XML-like logs with JSON
• Ignoring timestamp or level fields

3. Given this trace data snippet for a request through three microservices, what is the total time spent processing the request?

{
  "traceId": "abc123",
  "spans": [
    {"service": "A", "duration_ms": 50},
    {"service": "B", "duration_ms": 30},
    {"service": "C", "duration_ms": 20}
  ]
}

medium

A. 100 ms

B. 50 ms

C. 30 ms

D. 20 ms

Solution

1. Step 1: Understand trace spans and durations

   Each span shows time spent in a service. Total time is sum if services are sequential.

2. Step 2: Sum durations of all spans

   50 ms + 30 ms + 20 ms = 100 ms total processing time.

3. Final Answer:

   100 ms -> Option A

4. Quick Check:

   Sum spans durations = 100 ms [OK]

Hint: Add all span durations for total trace time [OK]

Common Mistakes:

• Taking only the longest span as total time
• Ignoring some spans in calculation
• Confusing traceId with duration

4. A developer notices that logs are missing trace IDs in a microservices system. What is the most likely cause?

medium

A. Services are using different programming languages.

B. Metrics collection is disabled.

C. Logs are stored in a different database.

D. Trace context is not propagated between services.

Solution

1. Step 1: Understand trace ID propagation

   Trace IDs must be passed along service calls to link logs and traces.

2. Step 2: Identify cause of missing trace IDs

   If trace context is not propagated, logs won't have trace IDs, breaking trace-log correlation.

3. Final Answer:

   Trace context is not propagated between services. -> Option D

4. Quick Check:

   Missing trace IDs = missing context propagation [OK]

Hint: Trace IDs must flow between services to appear in logs [OK]

Common Mistakes:

• Confusing metrics with trace IDs
• Assuming storage location causes missing IDs
• Blaming programming language differences

5. You are designing a microservices system and want to implement the three pillars: metrics, logs, and traces. Which approach best ensures scalability and effective monitoring?

hard

A. Use a centralized monitoring system that collects metrics via Prometheus, logs via ELK stack, and traces via OpenTelemetry.

B. Store all logs and traces locally on each service to reduce network overhead.

C. Only collect metrics and ignore logs and traces to save storage space.

D. Send all raw logs and traces directly to the client application for analysis.

Solution

1. Step 1: Identify best practices for scalable monitoring

   Centralized systems like Prometheus for metrics, ELK for logs, and OpenTelemetry for traces are industry standards for scalability and analysis.

2. Step 2: Evaluate options for scalability and effectiveness

   Local storage limits analysis and scalability; ignoring logs/traces loses insights; sending raw data to clients is inefficient and insecure.

3. Final Answer:

   Use a centralized monitoring system that collects metrics via Prometheus, logs via ELK stack, and traces via OpenTelemetry. -> Option A

4. Quick Check:

   Centralized, specialized tools = scalable monitoring [OK]

Hint: Centralize collection with proven tools for all three pillars [OK]

Common Mistakes:

• Storing logs/traces locally only
• Ignoring logs or traces
• Sending raw data directly to clients