Three pillars (metrics, logs, traces) in Microservices - Deep Dive

Overview - Three pillars (metrics, logs, traces)

What is it?

The three pillars are metrics, logs, and traces. They are ways to collect information about how software systems work. Metrics are numbers that show system health. Logs are detailed records of events. Traces show the path of requests through the system.

Why it matters

Without these pillars, it is very hard to understand what is happening inside complex software systems. Problems would be difficult to find and fix. This would cause slow responses, unhappy users, and lost business. These pillars help teams keep systems reliable and fast.

Where it fits

Before learning this, you should know basic software and microservices concepts. After this, you can learn about monitoring tools, alerting, and incident response. This topic fits into the bigger picture of system observability and reliability engineering.

Mental Model

Core Idea

Metrics, logs, and traces together give a full picture of system behavior from numbers, events, and request journeys.

Think of it like...

Imagine a car dashboard: metrics are the speedometer and fuel gauge, logs are the detailed trip diary, and traces are the GPS route showing where the car traveled.

┌─────────────┐   ┌─────────────┐   ┌─────────────┐
│   Metrics   │   │    Logs     │   │   Traces    │
│ (numbers)   │   │ (events)    │   │ (journeys)  │
└─────┬───────┘   └─────┬───────┘   └─────┬───────┘
      │                 │                 │       
      │                 │                 │       
      └───────┬─────────┴─────────┬───────┘       
              │                   │               
        System Health        System Behavior     
              │                   │               
              └───────────┬───────┘               
                          │                       
                   Full Observability            

Build-Up - 6 Steps

1

Foundation

Understanding Metrics Basics

Concept: Metrics are simple numbers that measure system performance or health over time.

Metrics include counts, rates, and gauges. For example, number of requests per second or CPU usage percentage. They are collected regularly and stored in time series databases.

Result

You can see trends like increasing load or errors rising before problems happen.

Understanding metrics helps you spot system health changes quickly and predict issues before users notice.

2

Foundation

Introduction to Logs

Concept: Logs are detailed records of events happening inside the system, often with timestamps and context.

Logs capture what happened, when, and sometimes why. For example, a user login event or an error message. Logs are usually text and can be searched or filtered.

Result

You get detailed clues to diagnose problems or understand system behavior after the fact.

Knowing logs lets you investigate specific incidents deeply and understand exact system actions.

3

Intermediate

Tracing Request Journeys

🤔Before reading on: do you think traces show only errors or the full path of requests? Commit to your answer.

Concept: Traces follow a request as it moves through different services, showing timing and dependencies.

Each step in a request is recorded with start and end times, and linked to the next step. This helps find slow or failing parts in complex systems.

Result

You can see exactly where delays or errors happen in multi-service workflows.

Understanding traces reveals hidden bottlenecks and helps optimize system flow end-to-end.

4

Intermediate

How Metrics, Logs, and Traces Work Together

🤔Before reading on: do you think one pillar alone is enough to fully understand system issues? Commit to yes or no.

Concept: The three pillars complement each other to provide full observability.

Metrics give quick health signals, logs provide detailed event context, and traces show request paths. Together, they help detect, diagnose, and fix problems faster.

Result

Teams can respond to incidents with confidence and reduce downtime.

Knowing how these pillars combine prevents blind spots in monitoring and speeds up troubleshooting.

5

Advanced

Implementing Pillars in Microservices

🤔Before reading on: do you think collecting all three pillars in microservices is simple or complex? Commit to your answer.

Concept: Microservices require distributed collection and correlation of metrics, logs, and traces.

Each service emits its own data. Tools aggregate and correlate them using unique request IDs. This allows end-to-end visibility despite system complexity.

Result

You get a unified view of system health and behavior across many independent services.

Understanding distributed collection and correlation is key to effective observability in microservices.

6

Expert

Challenges and Tradeoffs in Observability

🤔Before reading on: do you think collecting more data always improves observability? Commit yes or no.

Concept: Collecting metrics, logs, and traces involves tradeoffs in cost, performance, and data volume.

Too much data can overwhelm storage and slow systems. Sampling, aggregation, and filtering balance detail with efficiency. Choosing what to collect depends on system needs and risks.

Result

You achieve practical observability that scales and stays useful over time.

Knowing these tradeoffs helps design observability that is both effective and sustainable.

Under the Hood

Metrics are collected by counters and gauges updated in code or agents, then sent to time-series databases. Logs are generated as text entries with timestamps and context, stored in log management systems. Traces are created by instrumenting code to record spans with start/end times and metadata, linked by trace IDs. Correlation across services uses unique IDs passed through requests.

Why designed this way?

These pillars evolved to address different monitoring needs: metrics for quick health checks, logs for detailed event records, and traces for understanding complex request flows. Separating concerns allows specialized tools and efficient data handling. Alternatives like only logs or only metrics proved insufficient for modern distributed systems.

┌───────────────┐       ┌───────────────┐       ┌───────────────┐
│   Metrics     │──────▶│ Time-Series   │       │               │
│ (Counters)    │       │   Database    │       │               │
└───────────────┘       └───────────────┘       │               │
                                              │               │
┌───────────────┐       ┌───────────────┐       │ Observability │
│    Logs       │──────▶│ Log Storage   │──────▶│    System     │
│ (Text Events) │       │   & Search    │       │               │
└───────────────┘       └───────────────┘       │               │
                                              │               │
┌───────────────┐       ┌───────────────┐       │               │
│   Traces      │──────▶│ Trace Storage │──────▶│               │
│ (Spans)      │        │ & Analysis    │       └───────────────┘
└───────────────┘       └───────────────┘

Myth Busters - 4 Common Misconceptions

Quick: Do you think logs alone can give you full system health at a glance? Commit yes or no.

Common Belief: Logs alone are enough to monitor system health.

Reality: Logs provide detailed events but are not designed for quick health metrics or trend analysis.

Why it matters: Relying only on logs can delay detecting problems because logs are harder to summarize and analyze quickly.

Quick: Do you think traces only help when errors occur? Commit yes or no.

Common Belief: Traces are only useful for debugging errors.

Reality: Traces also help identify performance bottlenecks and understand normal request flows.

Why it matters: Ignoring traces limits your ability to optimize system speed and user experience.

Quick: Do you think collecting all data without limits is always best? Commit yes or no.

Common Belief: More data collection always improves observability.

Reality: Excessive data can overwhelm storage and slow down systems, making observability less effective.

Why it matters: Without careful data management, observability tools become costly and hard to use.

Quick: Do you think metrics, logs, and traces are independent and unrelated? Commit yes or no.

Common Belief: The three pillars work separately and do not need to be connected.

Reality: They are most powerful when correlated using IDs and timestamps to provide a unified view.

Why it matters: Treating them separately can cause blind spots and slow problem resolution.

Expert Zone

1

Metrics aggregation intervals affect alert sensitivity and must balance noise and delay.

2

Log verbosity levels should be tuned per environment to avoid missing info or flooding storage.

3

Trace sampling strategies impact visibility and system overhead; adaptive sampling is often best.

When NOT to use

In very simple or single-service applications, full tracing may be overkill; lightweight metrics and logs suffice. For extremely high-throughput systems, custom aggregation or specialized monitoring may be needed instead of standard pillars.

Production Patterns

Teams use centralized platforms like Prometheus for metrics, ELK stack or Loki for logs, and Jaeger or Zipkin for traces. They implement correlation IDs in requests to link data. Alerting rules trigger on metrics, while logs and traces support deep investigation.

Connections

Incident Response

Builds-on

Understanding observability pillars enables faster detection and diagnosis during incidents, improving recovery times.

Supply Chain Management

Similar pattern

Just like tracing parts through a supply chain reveals bottlenecks, tracing requests through microservices reveals system bottlenecks.

Human Body Health Monitoring

Analogy in different field

Metrics are like vital signs, logs like medical history, and traces like tracking a patient's movement through hospital departments, all combining to diagnose health.

Common Pitfalls

#1 Collecting logs without timestamps or context.

Wrong approach:

User login successful
Error connecting to DB
Request processed

Correct approach:

[2024-06-01T12:00:00Z] INFO User login successful userId=123
[2024-06-01T12:00:01Z] ERROR Error connecting to DB timeout=30s
[2024-06-01T12:00:02Z] INFO Request processed requestId=abc123

Root cause:

Not including timestamps and context makes logs hard to search, correlate, and understand.

#2 Using metrics without labels or dimensions.

Wrong approach:

http_requests_total = 1000

Correct approach:

http_requests_total{method="GET",status="200"} = 800
http_requests_total{method="POST",status="500"} = 200

Root cause:

Without labels, metrics lack detail needed to pinpoint issues by method, status, or service.

#3 Not propagating trace IDs across services.

Wrong approach:

Service A generates trace ID, but Service B starts a new unrelated trace ID.

Correct approach:

Service A generates trace ID and passes it in request headers; Service B continues the same trace ID.

Root cause:

Missing trace ID propagation breaks request journey visibility across services.

Key Takeaways

Metrics, logs, and traces are three essential ways to observe and understand software systems.

Each pillar provides unique insights: metrics for health, logs for events, and traces for request paths.

Together, they enable fast detection, diagnosis, and resolution of system problems.

Effective observability requires collecting, correlating, and managing data carefully to balance detail and performance.

Mastering these pillars is critical for running reliable, scalable microservices in production.

Practice

1. Which of the following best describes the role of metrics in microservices monitoring?

easy

A. They track the path of a request through multiple services.

B. They record detailed events and errors in the system.

C. They provide numerical data about system performance over time.

D. They store configuration settings for microservices.

Solution

1. Step 1: Understand what metrics represent

   Metrics are numerical measurements like CPU usage, request counts, or latency that show system health over time.

2. Step 2: Differentiate metrics from logs and traces

   Logs record events, traces follow request paths, but metrics summarize performance data.

3. Final Answer:

   They provide numerical data about system performance over time. -> Option C

4. Quick Check:

   Metrics = numerical performance data [OK]

Hint: Metrics = numbers about performance, not events or paths [OK]

Common Mistakes:

• Confusing metrics with logs as event records
• Thinking traces are numerical data
• Assuming metrics store configurations

2. Which syntax correctly represents a log entry in a microservice system?

easy

A. [2024-06-01 12:00:00] ERROR Failed to connect

B. {"timestamp": "2024-06-01T12:00:00Z", "level": "ERROR", "message": "Failed to connect"}

C. Failed to connect

D. ERROR 2024-06-01T12:00:00Z Failed to connect

Solution

1. Step 1: Identify standard log formats

   JSON format is widely used for structured logs in microservices for easy parsing and querying.

2. Step 2: Compare options for correctness

   {"timestamp": "2024-06-01T12:00:00Z", "level": "ERROR", "message": "Failed to connect"} is a valid JSON log entry with timestamp, level, and message fields. Others are less structured or not JSON.

3. Final Answer:

   {"timestamp": "2024-06-01T12:00:00Z", "level": "ERROR", "message": "Failed to connect"} -> Option B

4. Quick Check:

   Structured JSON logs = {"timestamp": "2024-06-01T12:00:00Z", "level": "ERROR", "message": "Failed to connect"} [OK]

Hint: Logs are best as structured JSON for easy use [OK]

Common Mistakes:

• Using unstructured plain text logs
• Confusing XML-like logs with JSON
• Ignoring timestamp or level fields

3. Given this trace data snippet for a request through three microservices, what is the total time spent processing the request?

{
  "traceId": "abc123",
  "spans": [
    {"service": "A", "duration_ms": 50},
    {"service": "B", "duration_ms": 30},
    {"service": "C", "duration_ms": 20}
  ]
}

medium

A. 100 ms

B. 50 ms

C. 30 ms

D. 20 ms

Solution

1. Step 1: Understand trace spans and durations

   Each span shows time spent in a service. Total time is sum if services are sequential.

2. Step 2: Sum durations of all spans

   50 ms + 30 ms + 20 ms = 100 ms total processing time.

3. Final Answer:

   100 ms -> Option A

4. Quick Check:

   Sum spans durations = 100 ms [OK]

Hint: Add all span durations for total trace time [OK]

Common Mistakes:

• Taking only the longest span as total time
• Ignoring some spans in calculation
• Confusing traceId with duration

4. A developer notices that logs are missing trace IDs in a microservices system. What is the most likely cause?

medium

A. Services are using different programming languages.

B. Metrics collection is disabled.

C. Logs are stored in a different database.

D. Trace context is not propagated between services.

Solution

1. Step 1: Understand trace ID propagation

   Trace IDs must be passed along service calls to link logs and traces.

2. Step 2: Identify cause of missing trace IDs

   If trace context is not propagated, logs won't have trace IDs, breaking trace-log correlation.

3. Final Answer:

   Trace context is not propagated between services. -> Option D

4. Quick Check:

   Missing trace IDs = missing context propagation [OK]

Hint: Trace IDs must flow between services to appear in logs [OK]

Common Mistakes:

• Confusing metrics with trace IDs
• Assuming storage location causes missing IDs
• Blaming programming language differences

5. You are designing a microservices system and want to implement the three pillars: metrics, logs, and traces. Which approach best ensures scalability and effective monitoring?

hard

A. Use a centralized monitoring system that collects metrics via Prometheus, logs via ELK stack, and traces via OpenTelemetry.

B. Store all logs and traces locally on each service to reduce network overhead.

C. Only collect metrics and ignore logs and traces to save storage space.

D. Send all raw logs and traces directly to the client application for analysis.

Solution

1. Step 1: Identify best practices for scalable monitoring

   Centralized systems like Prometheus for metrics, ELK for logs, and OpenTelemetry for traces are industry standards for scalability and analysis.

2. Step 2: Evaluate options for scalability and effectiveness

   Local storage limits analysis and scalability; ignoring logs/traces loses insights; sending raw data to clients is inefficient and insecure.

3. Final Answer:

   Use a centralized monitoring system that collects metrics via Prometheus, logs via ELK stack, and traces via OpenTelemetry. -> Option A

4. Quick Check:

   Centralized, specialized tools = scalable monitoring [OK]

Hint: Centralize collection with proven tools for all three pillars [OK]

Common Mistakes:

• Storing logs/traces locally only
• Ignoring logs or traces
• Sending raw data directly to clients