Prometheus for metrics collection in Kubernetes - Step-by-Step Execution

Process Flow - Prometheus for metrics collection

Start Prometheus

↓

Scrape Targets Configured

↓

Send HTTP Requests to Targets

↓

Targets Respond with Metrics

↓

Prometheus Stores Metrics

↓

Query Metrics via PromQL

↓

Visualize or Alert Based on Data

Prometheus starts, scrapes configured targets by HTTP, stores metrics, and allows querying or alerting.

Execution Sample

Kubernetes

scrape_configs:
  - job_name: 'example'
    static_configs:
      - targets: ['localhost:9090']

This config tells Prometheus to scrape metrics from localhost on port 9090 under job 'example'.

Process Table

Step	Action	Target	Request Sent	Response Received	Metrics Stored
1	Start Prometheus	-	No	No	No
2	Load scrape configs	localhost:9090	No	No	No
3	Send HTTP request	localhost:9090	Yes	No	No
4	Receive metrics response	localhost:9090	Yes	Yes	No
5	Store metrics data	localhost:9090	Yes	Yes	Yes
6	Wait for next scrape interval	-	No	No	Yes

💡 Prometheus continuously scrapes targets at configured intervals; this trace shows one scrape cycle.

Status Tracker

Variable	Start	After Step 2	After Step 4	After Step 5	Final
Prometheus State	Stopped	Running	Running	Running	Running
Scrape Targets	None	localhost:9090	localhost:9090	localhost:9090	localhost:9090
Metrics Data	Empty	Empty	Received	Stored	Stored

Key Moments - 3 Insights

Why does Prometheus send HTTP requests to targets?

Prometheus scrapes metrics by sending HTTP requests to targets as shown in step 3 of the execution_table to collect their current metrics.

What happens if a target does not respond with metrics?

If no response is received (step 4), Prometheus cannot store metrics for that scrape cycle, so metrics data remains unchanged as shown in the execution_table.

Does Prometheus store metrics permanently after one scrape?

Prometheus stores metrics after each successful scrape (step 5), but it continuously updates data over time, not permanently after one scrape.

Visual Quiz - 3 Questions

Look at the execution_table, at which step does Prometheus receive metrics from the target?

A Step 2

B Step 3

C Step 4

D Step 5

💡 Hint

Check the 'Response Received' column in the execution_table to find when metrics arrive.

According to variable_tracker, what is the state of 'Metrics Data' after step 5?

A Stored

B Empty

C Received

D None

💡 Hint

Look at the 'Metrics Data' row under 'After Step 5' in variable_tracker.

If the target at localhost:9090 did not respond, which step in execution_table would show 'No' under 'Response Received'?

A Step 3

B Step 4

C Step 5

D Step 6

💡 Hint

The 'Response Received' column shows 'Yes' or 'No' at each step; check step 4.

Concept Snapshot

Prometheus scrapes metrics by sending HTTP requests to configured targets.
Targets expose metrics on HTTP endpoints.
Prometheus stores scraped metrics for querying and alerting.
Scrape configs define which targets and how often to scrape.
PromQL queries allow extracting insights from stored metrics.

Full Transcript

Prometheus is a tool that collects metrics by regularly sending HTTP requests to configured targets. It starts by loading its scrape configuration, which lists the targets to monitor. Then it sends HTTP requests to these targets to get their current metrics. When targets respond, Prometheus stores the metrics data. This process repeats continuously at set intervals. If a target does not respond, Prometheus skips storing new data for that cycle. The stored metrics can be queried or used for alerts. This visual trace shows one full scrape cycle from start to storing metrics.

Practice

1. What is the main purpose of Prometheus in a Kubernetes environment?

easy

A. To deploy applications automatically

B. To collect and store metrics data for monitoring

C. To manage Kubernetes cluster nodes

D. To provide a user interface for Kubernetes

Solution

1. Step 1: Understand Prometheus role

   Prometheus is designed to collect numerical data called metrics from applications and systems.

2. Step 2: Identify its main function in Kubernetes

   In Kubernetes, Prometheus collects metrics to monitor app health and performance.

3. Final Answer:

   To collect and store metrics data for monitoring -> Option B

4. Quick Check:

   Prometheus collects metrics = A [OK]

Hint: Prometheus = metrics collection tool [OK]

Common Mistakes:

• Confusing Prometheus with deployment tools
• Thinking Prometheus manages nodes
• Assuming Prometheus is a UI tool

2. Which Kubernetes resource is used to tell Prometheus which services to monitor?

easy

A. ServiceMonitor

B. PodMonitor

C. ConfigMap

D. Ingress

Solution

1. Step 1: Identify Prometheus monitoring resources

   Prometheus uses special Kubernetes custom resources to know what to watch.

2. Step 2: Recognize ServiceMonitor's role

   ServiceMonitor tells Prometheus which Kubernetes services to scrape metrics from.

3. Final Answer:

   ServiceMonitor -> Option A

4. Quick Check:

   ServiceMonitor selects services for Prometheus [OK]

Hint: ServiceMonitor = tells Prometheus what to watch [OK]

Common Mistakes:

• Confusing PodMonitor with ServiceMonitor
• Using ConfigMap for monitoring targets
• Thinking Ingress controls Prometheus scraping

3. Given this snippet of a ServiceMonitor YAML, what is the scrape interval Prometheus will use?

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: example-monitor
spec:
  endpoints:
  - port: web
    interval: 15s
  selector:
    matchLabels:
      app: example

medium

A. 5 seconds

B. 30 seconds

C. 15 seconds

D. 1 minute

Solution

1. Step 1: Locate the interval field in YAML

   The interval is set under endpoints as 'interval: 15s'.

2. Step 2: Understand interval meaning

   This means Prometheus scrapes metrics every 15 seconds from the specified port.

3. Final Answer:

   15 seconds -> Option C

4. Quick Check:

   interval: 15s means 15 seconds [OK]

Hint: Check 'interval' value under endpoints [OK]

Common Mistakes:

• Ignoring the interval field and guessing default
• Confusing seconds with minutes
• Assuming interval is global, not per endpoint

4. You created a ServiceMonitor but Prometheus is not scraping metrics from your service. Which of these is a likely cause?

medium

A. The ServiceMonitor selector labels do not match the service labels

B. The Prometheus server is not running on the cluster

C. The service port is not exposed in the ServiceMonitor endpoints

D. All of the above

Solution

1. Step 1: Check label matching

   If ServiceMonitor selector labels don't match service labels, Prometheus won't find the service.

2. Step 2: Verify Prometheus server status and endpoint config

   Prometheus must be running and the service port must be correctly specified in endpoints to scrape metrics.

3. Final Answer:

   All of the above -> Option D

4. Quick Check:

   Any mismatch or missing config stops scraping [OK]

Hint: Check labels, server status, and endpoints all match [OK]

Common Mistakes:

• Only checking one cause and ignoring others
• Assuming Prometheus always runs by default
• Forgetting to expose correct port in ServiceMonitor

5. You want Prometheus to scrape metrics from multiple services with different scrape intervals. How should you configure this in Kubernetes?

hard

A. Create separate ServiceMonitor resources for each service with their specific intervals

B. Set a global scrape interval in Prometheus config and ignore ServiceMonitor intervals

C. Create one ServiceMonitor with multiple endpoints, each having its own interval

D. Use a ConfigMap to list all services and intervals for Prometheus

Solution

1. Step 1: Understand ServiceMonitor scope

   Each ServiceMonitor targets services with specific scrape configs; intervals are per endpoint.

2. Step 2: Manage different intervals

   To have different intervals per service, create separate ServiceMonitors with their own intervals.

3. Step 3: Why not other options?

   One ServiceMonitor with multiple endpoints cannot set different intervals per service easily; global config overrides intervals; ConfigMap does not control scraping targets.

4. Final Answer:

   Create separate ServiceMonitor resources for each service with their specific intervals -> Option A

5. Quick Check:

   Separate ServiceMonitors allow different intervals [OK]

Hint: Use separate ServiceMonitors for different intervals [OK]

Common Mistakes:

• Trying to set different intervals in one ServiceMonitor
• Ignoring ServiceMonitor intervals in favor of global config
• Using ConfigMap incorrectly for scraping targets