Prometheus for metrics collection in Kubernetes - Deep Dive

Overview - Prometheus for metrics collection

What is it?

Prometheus is a tool that collects and stores information about how software and systems are performing. It watches many parts of a system, like servers or applications, and records numbers called metrics. These metrics help people understand if everything is working well or if there are problems. Prometheus is often used with Kubernetes to keep track of containerized applications.

Why it matters

Without Prometheus, it would be very hard to know if your applications or servers are healthy or slow. Problems might go unnoticed until users complain. Prometheus solves this by giving real-time insights, so teams can fix issues quickly and keep systems running smoothly. This helps avoid downtime and improves user experience.

Where it fits

Before learning Prometheus, you should understand basic Kubernetes concepts like pods, services, and containers. After mastering Prometheus, you can learn about alerting systems like Alertmanager and visualization tools like Grafana to create dashboards from the collected metrics.

Mental Model

Core Idea

Prometheus continuously scrapes numeric data from targets to build a time-series database that helps monitor system health and performance.

Think of it like...

Imagine Prometheus as a diligent weather station that regularly checks temperature, wind, and rain at many locations, recording these numbers over time so you can see patterns and predict storms.

┌─────────────┐       ┌───────────────┐       ┌───────────────┐
│  Targets   │──────▶│  Prometheus   │──────▶│ Time-Series DB │
│ (Apps,    │       │  Server       │       │ (Metrics Data) │
│  Servers) │       │ (Scrapes &   │       └───────────────┘
└─────────────┘       │  Stores)     │
                      └───────────────┘
                             │
                             ▼
                      ┌───────────────┐
                      │  Query &      │
                      │  Alerting     │
                      └───────────────┘

Build-Up - 7 Steps

1

Foundation

What is Prometheus and Metrics

Concept: Introduce Prometheus as a tool that collects numeric data called metrics from software and systems.

Prometheus collects numbers that describe how well software or hardware is working. These numbers can be things like how many requests a server gets or how much memory it uses. Metrics are collected regularly to see changes over time.

Result

You understand that Prometheus gathers important numbers to help monitor systems.

Knowing that Prometheus focuses on numeric data helps you see why it is useful for tracking system health continuously.

2

Foundation

How Prometheus Collects Metrics

Concept: Explain the scraping method Prometheus uses to get metrics from targets.

Prometheus asks targets (like applications or servers) for their current metrics by sending HTTP requests to special endpoints. This process is called scraping. Targets expose their metrics in a format Prometheus understands.

Result

You see how Prometheus actively pulls data instead of waiting for it to be sent.

Understanding scraping clarifies how Prometheus stays updated with fresh data from many sources.

3

Intermediate

Prometheus Data Model and Time-Series

🤔Before reading on: do you think Prometheus stores raw logs or numeric time-stamped data? Commit to your answer.

Concept: Introduce the time-series data model where each metric has a name, labels, and values over time.

Prometheus stores metrics as time-series, which means each metric is recorded with a timestamp and optional labels that describe details like which server or region it came from. This allows detailed filtering and analysis.

Result

You understand that Prometheus organizes data to track changes and differences across many dimensions.

Knowing the time-series model explains how Prometheus can answer complex questions about system behavior over time.

4

Intermediate

Prometheus Query Language (PromQL)

🤔Before reading on: do you think PromQL is a simple keyword search or a powerful language for math and filtering? Commit to your answer.

Concept: Explain PromQL as a language to ask questions about metrics and get meaningful answers.

PromQL lets you select, filter, and calculate with metrics stored in Prometheus. For example, you can find the average CPU usage over the last 5 minutes or compare request rates between servers.

Result

You can write queries to explore and understand your metrics data.

Understanding PromQL unlocks the power of Prometheus to provide actionable insights from raw numbers.

5

Intermediate

Integrating Prometheus with Kubernetes

Concept: Show how Prometheus discovers and scrapes metrics from Kubernetes components automatically.

Prometheus can find Kubernetes pods and services by using Kubernetes APIs. It uses labels and annotations to know which targets to scrape. This automatic discovery means you don't have to manually list every target.

Result

Prometheus keeps up with dynamic Kubernetes environments without manual updates.

Knowing service discovery in Kubernetes explains how Prometheus scales and adapts to changing clusters.

6

Advanced

Alerting and Exporters in Prometheus

🤔Before reading on: do you think Prometheus sends alerts directly or uses another tool? Commit to your answer.

Concept: Introduce alerting rules and exporters that extend Prometheus's capabilities.

Prometheus can define alert rules that trigger when metrics cross thresholds. Alerts are sent to Alertmanager, which handles notifications. Exporters are programs that expose metrics from systems that don't natively support Prometheus, like databases or hardware.

Result

You understand how Prometheus fits into a larger monitoring and alerting system.

Knowing alerting and exporters shows how Prometheus monitors diverse systems and notifies teams proactively.

7

Expert

Scaling Prometheus and Handling High Cardinality

🤔Before reading on: do you think Prometheus handles unlimited unique metric labels easily? Commit to your answer.

Concept: Discuss challenges with many unique label combinations and strategies to scale Prometheus in large environments.

High cardinality means having many unique label sets, which can cause performance issues. Experts use techniques like relabeling to reduce labels, federation to split data across servers, and remote storage integrations to handle large scale.

Result

You grasp the limits of Prometheus and how to architect solutions for big systems.

Understanding scaling challenges prevents common pitfalls and helps design robust monitoring setups.

Under the Hood

Prometheus runs a server that periodically sends HTTP requests to configured targets' /metrics endpoints. Targets respond with plaintext data formatted in a specific way. Prometheus parses this data and stores it as time-stamped samples in a local time-series database. It indexes metrics by name and labels for fast querying. Alerting rules are evaluated on this data, and results can trigger notifications.

Why designed this way?

Prometheus was designed for reliability and simplicity. Pull-based scraping avoids losing data if targets go down temporarily. The time-series database is optimized for fast writes and queries of numeric data. Using labels allows flexible grouping without rigid schemas. Alternatives like push-based systems were rejected to keep the architecture simple and scalable.

┌───────────────┐       ┌───────────────┐       ┌───────────────┐
│   Targets     │──────▶│  Prometheus   │──────▶│  Local TSDB   │
│ (/metrics)    │       │  Scraper      │       │ (Time-Series) │
└───────────────┘       └───────────────┘       └───────────────┘
                             │
                             ▼
                      ┌───────────────┐
                      │  Query Engine │
                      └───────────────┘
                             │
                             ▼
                      ┌───────────────┐
                      │ Alertmanager  │
                      └───────────────┘

Myth Busters - 4 Common Misconceptions

Quick: Does Prometheus push metrics to the server or pull them by scraping? Commit to push or pull.

Common Belief: Prometheus pushes metrics from applications to the server like some other monitoring tools.

Reality: Prometheus pulls metrics by scraping HTTP endpoints exposed by targets at regular intervals.

Why it matters: Assuming push leads to wrong setup and missed metrics because Prometheus expects to scrape targets, not receive data pushed.

Quick: Can Prometheus store logs and traces as well as metrics? Commit yes or no.

Common Belief: Prometheus can store any kind of monitoring data including logs and traces.

Reality: Prometheus is designed only for numeric time-series metrics, not logs or distributed traces.

Why it matters: Trying to use Prometheus for logs wastes resources and misses the benefits of specialized tools like Loki or Jaeger.

Quick: Does adding many unique labels always improve monitoring detail without downsides? Commit yes or no.

Common Belief: More labels always mean better monitoring detail and no problems.

Reality: High cardinality from many unique labels can cause performance and storage issues in Prometheus.

Why it matters: Ignoring this leads to slow queries, crashes, and unreliable monitoring in production.

Quick: Is Prometheus a long-term storage solution for metrics? Commit yes or no.

Common Belief: Prometheus stores metrics indefinitely for historical analysis.

Reality: Prometheus stores data locally for a limited time (usually weeks); long-term storage requires external systems.

Why it matters: Expecting long-term storage causes data loss and surprises when old metrics disappear.

Expert Zone

1

Prometheus's pull model simplifies network security by requiring only the server to initiate connections, reducing firewall complexity.

2

Relabeling rules in Prometheus allow dynamic modification of target labels before scraping, enabling flexible and efficient monitoring setups.

3

Federation lets multiple Prometheus servers share data hierarchically, which is essential for scaling monitoring across global or multi-tenant environments.

When NOT to use

Prometheus is not suitable for collecting high-volume logs or distributed traces; use specialized tools like Loki for logs and Jaeger for traces. For very high cardinality metrics or long-term storage, consider remote storage solutions or other monitoring systems designed for those needs.

Production Patterns

In production, Prometheus is often paired with Alertmanager for alerting and Grafana for visualization. Exporters extend monitoring to databases, hardware, and cloud services. Operators use service discovery and relabeling to automate target management in Kubernetes clusters. Federation and remote write enable scaling and integration with long-term storage.

Connections

Time-Series Databases

Prometheus builds on the concept of time-series databases specialized for numeric data over time.

Understanding time-series databases helps grasp how Prometheus efficiently stores and queries metrics.

Event-Driven Alerting Systems

Prometheus integrates with alerting systems that react to metric thresholds by sending notifications.

Knowing alerting systems clarifies how monitoring data triggers real-world responses to issues.

Supply Chain Management

Both Prometheus monitoring and supply chain management track many moving parts continuously to detect problems early.

Seeing this connection highlights the universal need for real-time data collection and analysis in complex systems.

Common Pitfalls

#1 Trying to monitor all Kubernetes pods without filtering causes overload.

Wrong approach:

scrape_configs:
- job_name: 'kubernetes-pods'
kubernetes_sd_configs:
- role: pod

Correct approach:

scrape_configs:
- job_name: 'kubernetes-pods'
kubernetes_sd_configs:
- role: pod
relabel_configs:
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
action: keep
regex: true

Root cause:

Not filtering targets leads to scraping unnecessary pods, causing high load and wasted resources.

#2 Using too many unique labels on metrics without limits.

Wrong approach:

http_requests_total{method="GET",endpoint="/api",user_id="12345",session_id="abcde",region="us-east-1",extra_label="value"} 42

Correct approach:

http_requests_total{method="GET",endpoint="/api",region="us-east-1"} 42

Root cause:

Adding high-cardinality labels like user_id or session_id causes performance degradation.

#3 Expecting Prometheus to store data forever without external storage.

Wrong approach:

No remote_write configured; relying solely on local storage for months of data.

Correct approach:

remote_write:
- url: 'https://remote-storage.example.com/api/v1/write'

Root cause:

Local storage is limited in retention; without remote storage, old data is lost.

Key Takeaways

Prometheus collects numeric metrics by regularly scraping targets, building a time-series database for monitoring.

Its pull-based model and flexible labeling system allow dynamic and detailed tracking of complex systems like Kubernetes.

PromQL is a powerful language to query and analyze metrics, enabling deep insights and alerting.

Understanding Prometheus's limits with high cardinality and storage helps design scalable and reliable monitoring solutions.

In production, Prometheus works best with exporters, alerting tools, and visualization platforms to provide a full monitoring ecosystem.