Log-based metrics in GCP - Deep Dive

Overview - Log-based metrics

What is it?

Log-based metrics are measurements created from log entries in cloud systems. They count or summarize specific events recorded in logs to help monitor and understand system behavior. Instead of relying only on predefined metrics, log-based metrics let you create custom insights from detailed logs. This helps track unique or complex events that standard metrics might miss.

Why it matters

Without log-based metrics, you would only see general system data and miss important details hidden in logs. This makes it hard to detect unusual problems or track specific user actions. Log-based metrics turn raw logs into clear numbers you can watch and alert on, improving system reliability and response time. They help teams catch issues early and understand system health deeply.

Where it fits

Before learning log-based metrics, you should understand basic logging and standard metrics in cloud monitoring. After mastering log-based metrics, you can explore alerting based on these metrics and advanced monitoring dashboards. This topic fits in the monitoring and observability part of cloud infrastructure management.

Mental Model

Core Idea

Log-based metrics transform detailed log messages into meaningful numbers that reveal system behavior and issues.

Think of it like...

Imagine logs as a diary full of detailed daily notes, and log-based metrics as the summary charts that count how many times certain events happened, like how many times you went jogging or cooked dinner.

┌───────────────┐      ┌───────────────┐      ┌───────────────┐
│   Raw Logs    │─────▶│ Log-based     │─────▶│ Metrics       │
│ (detailed     │      │ Metrics       │      │ (counts, sums)│
│  messages)    │      │ (filters &    │      │               │
│               │      │  aggregations)│      │               │
└───────────────┘      └───────────────┘      └───────────────┘

Build-Up - 7 Steps

1

Foundation

Understanding Logs and Their Role

Concept: Logs are detailed records of events happening in a system, capturing what occurred and when.

Logs are like a diary for your system. They record every action, error, or event with details like time, source, and message. For example, a web server log might record each page visit or error. Logs help you understand exactly what happened inside your system.

Result

You know that logs are detailed event records, but they can be very large and hard to analyze directly.

Understanding logs as detailed event records is essential because it shows why raw logs alone are too complex for quick monitoring.

2

Foundation

What Are Metrics in Cloud Monitoring

Concept: Metrics are numbers that summarize system behavior over time, like counts or averages.

Metrics simplify monitoring by turning many events into numbers you can watch. For example, a metric might count how many requests a server gets per minute. Cloud platforms provide standard metrics like CPU usage or network traffic, which help track system health at a glance.

Result

You see how metrics give a clear, simple view of system performance compared to raw logs.

Knowing metrics summarize system data helps you appreciate why turning logs into metrics is powerful for monitoring.

3

Intermediate

Creating Custom Metrics from Logs

🤔Before reading on: do you think you can create metrics only from predefined system data or also from your own logs? Commit to your answer.

Concept: Log-based metrics let you define custom metrics by filtering and counting specific log entries.

Instead of only using standard metrics, you can create your own by selecting log entries that match certain patterns. For example, count how many times a specific error message appears in logs. This custom metric updates as new logs arrive, giving you tailored insights.

Result

You can monitor unique events that standard metrics don’t cover, improving system visibility.

Understanding that logs can be filtered and counted to create custom metrics unlocks flexible monitoring tailored to your needs.

4

Intermediate

Types of Log-based Metrics

🤔Before reading on: do you think log-based metrics only count events, or can they also measure other things like sums or distributions? Commit to your answer.

Concept: Log-based metrics can be counters, distributions, or gauges, each summarizing logs differently.

Counters count how many times an event happens, like error occurrences. Distributions measure values over time, like response times from logs. Gauges track the latest value, such as current queue size from logs. Choosing the right type helps capture the right insight.

Result

You understand how different metric types provide varied views of system behavior from logs.

Knowing metric types helps you pick the best way to summarize logs for your monitoring goals.

5

Intermediate

Filtering and Aggregating Logs for Metrics

🤔Before reading on: do you think log-based metrics use all logs or only selected ones? Commit to your answer.

Concept: You define filters to select relevant logs and aggregation rules to summarize them into metrics.

Filters use conditions like text matching or fields to pick logs of interest, e.g., only errors or specific services. Aggregations then count or summarize these filtered logs over time windows. This process turns noisy logs into focused, meaningful metrics.

Result

You can create precise metrics that reflect only the events you care about.

Understanding filtering and aggregation is key to making log-based metrics useful and efficient.

6

Advanced

Using Log-based Metrics for Alerting and Dashboards

🤔Before reading on: do you think log-based metrics can trigger alerts and be visualized like standard metrics? Commit to your answer.

Concept: Log-based metrics integrate with monitoring tools to create alerts and dashboards for proactive system management.

Once created, log-based metrics appear like regular metrics in monitoring systems. You can set alerts to notify you when counts exceed thresholds, like too many errors. Dashboards can graph these metrics over time, helping spot trends or sudden issues.

Result

You gain proactive control over system health using custom insights from logs.

Knowing log-based metrics feed into alerting and visualization closes the loop from raw data to action.

7

Expert

Performance and Cost Considerations of Log-based Metrics

🤔Before reading on: do you think creating many log-based metrics has no impact on system cost or performance? Commit to your answer.

Concept: Log-based metrics consume resources and may increase costs; careful design balances insight and efficiency.

Each log-based metric requires processing logs in real time, which uses compute and storage. Creating too many or very complex metrics can slow down monitoring and increase cloud bills. Experts design metrics to focus on critical events and use sampling or aggregation to reduce load.

Result

You understand how to optimize log-based metrics for cost-effective, performant monitoring.

Knowing the tradeoffs of log-based metrics helps prevent unexpected costs and system slowdowns in production.

Under the Hood

Log-based metrics work by continuously scanning incoming log entries and applying user-defined filters to select relevant logs. These selected logs are then aggregated over time windows using counting, summing, or distribution calculations. The results are stored as time series data, which monitoring systems query like standard metrics. This process happens in near real-time using scalable cloud infrastructure.

Why designed this way?

This design allows flexible, custom insights without changing application code or relying only on predefined metrics. It leverages existing logs, which are rich in detail, to create metrics tailored to specific monitoring needs. Alternatives like instrumenting code for every metric are costly and less flexible, so log-based metrics provide a scalable, adaptable solution.

┌───────────────┐      ┌───────────────┐      ┌───────────────┐      ┌───────────────┐
│   Incoming    │─────▶│   Log Filter  │─────▶│ Aggregation   │─────▶│ Time Series   │
│    Logs      │      │ (select logs) │      │ (count, sum)  │      │   Storage     │
└───────────────┘      └───────────────┘      └───────────────┘      └───────────────┘

Myth Busters - 4 Common Misconceptions

Quick: do you think log-based metrics automatically include all logs without filtering? Commit to yes or no.

Common Belief: Log-based metrics use all logs by default and show complete system data.

Reality: Log-based metrics only include logs that match the filters you define; they do not automatically use all logs.

Why it matters: Assuming all logs are included can lead to missing important events or creating misleading metrics if filters are not set correctly.

Quick: do you think log-based metrics are free and have no impact on cloud costs? Commit to yes or no.

Common Belief: Creating many log-based metrics has no cost or performance impact.

Reality: Each log-based metric consumes processing and storage resources, which can increase cloud costs and affect performance.

Why it matters: Ignoring cost impact can lead to unexpectedly high bills and slower monitoring systems.

Quick: do you think log-based metrics can replace all standard metrics? Commit to yes or no.

Common Belief: Log-based metrics can fully replace standard system metrics.

Reality: Log-based metrics complement but do not replace standard metrics, which are often more efficient and reliable for core system data.

Why it matters: Relying only on log-based metrics may miss optimized, low-latency data provided by standard metrics.

Quick: do you think log-based metrics can only count events, not measure values like latency? Commit to yes or no.

Common Belief: Log-based metrics only count how many times something happens.

Reality: Log-based metrics can also measure distributions and gauges, capturing values like latency or current state from logs.

Why it matters: Underestimating metric types limits the ability to monitor complex system behaviors.

Expert Zone

1

Log-based metrics latency can vary; understanding ingestion delays helps set realistic alert thresholds.

2

Complex filters with regex or multiple conditions can slow metric processing; balancing complexity and performance is key.

3

Combining log-based metrics with trace data provides richer observability but requires careful correlation strategies.

When NOT to use

Avoid log-based metrics for high-frequency, low-latency core system metrics where standard metrics are optimized. Use log-based metrics mainly for custom, event-driven insights not covered by standard metrics.

Production Patterns

In production, teams create log-based metrics for error tracking, user behavior counts, and custom SLA monitoring. They integrate these metrics into alerting policies and dashboards, often combining them with standard metrics for comprehensive observability.

Connections

Event-driven architecture

Builds-on

Log-based metrics transform event logs into measurable data, similar to how event-driven systems react to events; understanding one helps grasp how systems respond to and measure events.

Data aggregation in databases

Same pattern

Both log-based metrics and database aggregation summarize large data sets into meaningful summaries, showing a common pattern of filtering and summarizing data for insight.

Human memory summarization

Analogy in cognition

Just as humans remember key points from many details, log-based metrics summarize vast logs into key numbers, illustrating how summarization aids understanding across fields.

Common Pitfalls

#1 Creating too many detailed log-based metrics without filtering.

Wrong approach:

Create a log-based metric that counts every log entry without any filter.

Correct approach:

Create log-based metrics with specific filters targeting relevant log entries only.

Root cause:

Misunderstanding that filtering is necessary to focus metrics and avoid overload.

#2 Expecting instant availability of log-based metrics after creation.

Wrong approach:

Immediately setting alerts on a newly created log-based metric without waiting for data to accumulate.

Correct approach:

Wait for sufficient log data to be processed before relying on the metric for alerts.

Root cause:

Not realizing log processing and metric aggregation have ingestion delays.

#3 Using log-based metrics for high-frequency system metrics like CPU usage.

Wrong approach:

Replacing standard CPU usage metrics with log-based metrics counting CPU logs.

Correct approach:

Use standard metrics for CPU usage and reserve log-based metrics for custom event counts.

Root cause:

Confusing the purpose and efficiency of standard vs. log-based metrics.

Key Takeaways

Log-based metrics turn detailed logs into meaningful numbers that reveal system events and behaviors.

They allow custom monitoring beyond standard metrics by filtering and aggregating specific log entries.

Choosing the right metric type and filters is crucial for effective and efficient monitoring.

Log-based metrics integrate with alerting and dashboards to enable proactive system management.

Understanding their cost and performance impact helps design scalable, cost-effective monitoring solutions.

Practice

1. What is the main purpose of a log-based metric in Google Cloud?

easy

A. To delete old logs automatically

B. To count specific log entries and turn them into numbers you can monitor

C. To store logs permanently without any processing

D. To encrypt logs for security

Solution

1. Step 1: Understand what log-based metrics do

   Log-based metrics count log entries that match a filter and convert them into numerical data.

2. Step 2: Compare options to this definition

   Only To count specific log entries and turn them into numbers you can monitor describes counting logs and turning them into numbers for monitoring.

3. Final Answer:

   To count specific log entries and turn them into numbers you can monitor -> Option B

4. Quick Check:

   Log-based metrics = counting logs for monitoring [OK]

Hint: Log-based metrics count logs to create numbers you watch [OK]

Common Mistakes:

• Confusing log storage with metrics
• Thinking metrics delete or encrypt logs
• Assuming metrics store logs permanently

2. Which of the following is the correct way to create a log-based metric filter to count error logs with severity ERROR?

easy

A. severity != "ERROR"

B. severity = "ERROR"

C. severity:ERROR

D. severity == "ERROR"

Solution

1. Step 1: Recall the correct filter syntax for log-based metrics

   Google Cloud uses the syntax severity == "ERROR" to filter logs with severity ERROR.

2. Step 2: Check each option

   Double equals (==) is the correct equality operator; single equals (=) is invalid. Colon (:) is for substring matching, not exact equality. != excludes the value.

3. Final Answer:

   severity == "ERROR" -> Option D

4. Quick Check:

   Filter syntax uses == for equality [OK]

Hint: Use double equals sign for filter equality [OK]

Common Mistakes:

• Using single equals instead of double
• Using colon instead of equals
• Using not equals when filtering for ERROR

3. Given this log-based metric filter: resource.type = "gce_instance" AND severity >= "ERROR", what logs will this metric count?

medium

A. All logs from GCE instances with severity ERROR or higher

B. Only logs with severity exactly ERROR from any resource

C. All logs from any resource with severity ERROR or higher

D. Only logs from GCE instances with severity lower than ERROR

Solution

1. Step 1: Analyze the filter conditions

   The filter requires resource.type to be "gce_instance" and severity to be ERROR or higher (like CRITICAL).

2. Step 2: Match options to filter meaning

   All logs from GCE instances with severity ERROR or higher correctly states logs from GCE instances with severity ERROR or higher. Others either ignore resource type or severity condition.

3. Final Answer:

   All logs from GCE instances with severity ERROR or higher -> Option A

4. Quick Check:

   Filter = GCE instance + severity >= ERROR [OK]

Hint: AND means both conditions must be true [OK]

Common Mistakes:

• Ignoring resource.type condition
• Confusing severity >= ERROR with exact match
• Assuming severity lower than ERROR is included

4. You created a log-based metric with filter severity = "ERROR" OR severity = "CRITICAL", but the metric shows zero counts. What is the likely problem?

medium

A. The filter syntax is invalid; OR is not supported in log-based metric filters

B. Severity levels ERROR and CRITICAL do not exist

C. The logs do not have severity fields

D. The metric needs at least 24 hours to show data

Solution

1. Step 1: Check if the filter syntax is valid

   The syntax severity = "ERROR" OR severity = "CRITICAL" is valid; OR is supported in log-based metric filters.

2. Step 2: Understand impact of zero counts

   A valid filter with zero counts indicates no matching logs, likely because the logs do not have severity fields or lack ERROR/CRITICAL values.

3. Final Answer:

   The logs do not have severity fields -> Option C

4. Quick Check:

   Valid filter + zero = no matching logs [OK]

Hint: Zero metric despite valid filter usually means no matching log entries [OK]

Common Mistakes:

• Assuming filter syntax invalid
• Thinking OR unsupported
• Waiting 24 hours unnecessarily

5. You want to create a log-based metric that counts how many times a specific error message appears in logs from Cloud Functions. The error message is in the field jsonPayload.message. Which filter correctly counts logs where the message contains the word "timeout"?

hard

A. resource.type = "cloud_function" AND jsonPayload.message : "timeout"

B. resource.type = "cloud_function" AND jsonPayload.message = "timeout"

C. resource.type = "cloud_function" AND jsonPayload.message >= "timeout"

D. resource.type = "cloud_function" AND jsonPayload.message != "timeout"

Solution

1. Step 1: Understand how to filter logs containing a word

   The colon operator (:) is used to check if a field contains a substring in log filters.

2. Step 2: Check resource type and filter syntax

   resource.type = "cloud_function" AND jsonPayload.message : "timeout" correctly filters logs from Cloud Functions where jsonPayload.message contains "timeout".

3. Final Answer:

   resource.type = "cloud_function" AND jsonPayload.message : "timeout" -> Option A

4. Quick Check:

   Use : to check substring in log filters [OK]

Hint: Use colon : to find substring in log fields [OK]

Common Mistakes:

• Using equals instead of colon for substring
• Using comparison operators like >= incorrectly
• Filtering for not equals when looking for matches