Bird
Raised Fist0
Microservicessystem_design~12 mins

Three pillars (metrics, logs, traces) in Microservices - Architecture Diagram

Choose your learning style10 modes available
LearnWhyDeepArchPracticeChallengeDesignRecallScale

Start learning this pattern below

Jump into concepts and practice - no test required

or
Recommended
Test this pattern10 questions across easy, medium, and hard to know if this pattern is strong
System Overview - Three pillars (metrics, logs, traces)

This system collects and analyzes three key types of data from microservices: metrics, logs, and traces. These data pillars help monitor service health, diagnose issues, and understand request flows across distributed services.

Key requirements include real-time data collection, scalable storage, and easy querying for troubleshooting and performance optimization.

Architecture Diagram
User
  |
  v
Load Balancer
  |
  v
API Gateway
  |
  v
+-------------------+       +------------------+       +------------------+
| Microservice A     |<----->| Metrics Collector |<----->| Metrics Database |
| (Logs, Traces)     |       +------------------+       +------------------+
|                   |               ^                          ^
+-------------------+               |                          |
        |                           |                          |
        v                           |                          |
+-------------------+       +------------------+       +------------------+
| Microservice B     |<----->| Logs Collector   |<----->| Logs Storage     |
| (Logs, Traces)     |       +------------------+       +------------------+
|                   |               ^                          ^
+-------------------+               |                          |
        |                           |                          |
        v                           |                          |
+-------------------+       +------------------+       +------------------+
| Microservice C     |<----->| Traces Collector |<----->| Traces Storage   |
| (Logs, Traces)     |       +------------------+       +------------------+
Components
User
user
Initiates requests to microservices
Load Balancer
load_balancer
Distributes incoming requests evenly across microservices
API Gateway
api_gateway
Routes requests to appropriate microservices and enforces security
Microservice A
service
Handles business logic and generates logs, metrics, and traces
Microservice B
service
Handles business logic and generates logs, metrics, and traces
Microservice C
service
Handles business logic and generates logs, metrics, and traces
Metrics Collector
collector
Aggregates metrics data from microservices
Metrics Database
database
Stores time-series metrics data for analysis
Logs Collector
collector
Aggregates logs from microservices
Logs Storage
database
Stores logs for search and troubleshooting
Traces Collector
collector
Aggregates distributed traces from microservices
Traces Storage
database
Stores trace data to analyze request flows
Request Flow - 12 Hops
User → Load Balancer
Load Balancer → API Gateway
API Gateway → Microservice A
Microservice A → Metrics Collector
Microservice A → Logs Collector
Microservice A → Traces Collector
Metrics Collector → Metrics Database
Logs Collector → Logs Storage
Traces Collector → Traces Storage
Microservice A → API Gateway
API Gateway → Load Balancer
Load Balancer → User
Failure Scenario
Component Fails:Metrics Database
Impact:Metrics data cannot be stored, so monitoring dashboards may show stale or missing data. Logs and traces remain unaffected.
Mitigation:Use database replication and failover to a standby metrics database. Buffer metrics temporarily in the collector to retry writes.
Architecture Quiz - 3 Questions
Test your understanding
Which component collects logs from microservices?
AMetrics Collector
BAPI Gateway
CLogs Collector
DTraces Storage
Design Principle
This architecture cleanly separates the three pillars of observability—metrics, logs, and traces—using dedicated collectors and storage. This separation allows scalable, specialized processing and querying of each data type, improving monitoring and troubleshooting in microservices.

Practice

(1/5)
1. Which of the following best describes the role of metrics in microservices monitoring?
easy
A. They track the path of a request through multiple services.
B. They record detailed events and errors in the system.
C. They provide numerical data about system performance over time.
D. They store configuration settings for microservices.

Solution

  1. Step 1: Understand what metrics represent

    Metrics are numerical measurements like CPU usage, request counts, or latency that show system health over time.

  2. Step 2: Differentiate metrics from logs and traces

    Logs record events, traces follow request paths, but metrics summarize performance data.

  3. Final Answer:

    They provide numerical data about system performance over time. -> Option C

  4. Quick Check:

    Metrics = numerical performance data [OK]
Hint: Metrics = numbers about performance, not events or paths [OK]
Common Mistakes:
  • Confusing metrics with logs as event records
  • Thinking traces are numerical data
  • Assuming metrics store configurations
2. Which syntax correctly represents a log entry in a microservice system?
easy
A. [2024-06-01 12:00:00] ERROR Failed to connect
B. {"timestamp": "2024-06-01T12:00:00Z", "level": "ERROR", "message": "Failed to connect"}
C. Failed to connect
D. ERROR 2024-06-01T12:00:00Z Failed to connect

Solution

  1. Step 1: Identify standard log formats

    JSON format is widely used for structured logs in microservices for easy parsing and querying.

  2. Step 2: Compare options for correctness

    {"timestamp": "2024-06-01T12:00:00Z", "level": "ERROR", "message": "Failed to connect"} is a valid JSON log entry with timestamp, level, and message fields. Others are less structured or not JSON.

  3. Final Answer:

    {"timestamp": "2024-06-01T12:00:00Z", "level": "ERROR", "message": "Failed to connect"} -> Option B

  4. Quick Check:

    Structured JSON logs = {"timestamp": "2024-06-01T12:00:00Z", "level": "ERROR", "message": "Failed to connect"} [OK]
Hint: Logs are best as structured JSON for easy use [OK]
Common Mistakes:
  • Using unstructured plain text logs
  • Confusing XML-like logs with JSON
  • Ignoring timestamp or level fields
3. Given this trace data snippet for a request through three microservices, what is the total time spent processing the request?
{
  "traceId": "abc123",
  "spans": [
    {"service": "A", "duration_ms": 50},
    {"service": "B", "duration_ms": 30},
    {"service": "C", "duration_ms": 20}
  ]
}
medium
A. 100 ms
B. 50 ms
C. 30 ms
D. 20 ms

Solution

  1. Step 1: Understand trace spans and durations

    Each span shows time spent in a service. Total time is sum if services are sequential.

  2. Step 2: Sum durations of all spans

    50 ms + 30 ms + 20 ms = 100 ms total processing time.

  3. Final Answer:

    100 ms -> Option A

  4. Quick Check:

    Sum spans durations = 100 ms [OK]
Hint: Add all span durations for total trace time [OK]
Common Mistakes:
  • Taking only the longest span as total time
  • Ignoring some spans in calculation
  • Confusing traceId with duration
4. A developer notices that logs are missing trace IDs in a microservices system. What is the most likely cause?
medium
A. Services are using different programming languages.
B. Metrics collection is disabled.
C. Logs are stored in a different database.
D. Trace context is not propagated between services.

Solution

  1. Step 1: Understand trace ID propagation

    Trace IDs must be passed along service calls to link logs and traces.

  2. Step 2: Identify cause of missing trace IDs

    If trace context is not propagated, logs won't have trace IDs, breaking trace-log correlation.

  3. Final Answer:

    Trace context is not propagated between services. -> Option D

  4. Quick Check:

    Missing trace IDs = missing context propagation [OK]
Hint: Trace IDs must flow between services to appear in logs [OK]
Common Mistakes:
  • Confusing metrics with trace IDs
  • Assuming storage location causes missing IDs
  • Blaming programming language differences
5. You are designing a microservices system and want to implement the three pillars: metrics, logs, and traces. Which approach best ensures scalability and effective monitoring?
hard
A. Use a centralized monitoring system that collects metrics via Prometheus, logs via ELK stack, and traces via OpenTelemetry.
B. Store all logs and traces locally on each service to reduce network overhead.
C. Only collect metrics and ignore logs and traces to save storage space.
D. Send all raw logs and traces directly to the client application for analysis.

Solution

  1. Step 1: Identify best practices for scalable monitoring

    Centralized systems like Prometheus for metrics, ELK for logs, and OpenTelemetry for traces are industry standards for scalability and analysis.

  2. Step 2: Evaluate options for scalability and effectiveness

    Local storage limits analysis and scalability; ignoring logs/traces loses insights; sending raw data to clients is inefficient and insecure.

  3. Final Answer:

    Use a centralized monitoring system that collects metrics via Prometheus, logs via ELK stack, and traces via OpenTelemetry. -> Option A

  4. Quick Check:

    Centralized, specialized tools = scalable monitoring [OK]
Hint: Centralize collection with proven tools for all three pillars [OK]
Common Mistakes:
  • Storing logs/traces locally only
  • Ignoring logs or traces
  • Sending raw data directly to clients