Prometheus for metrics collection in Kubernetes - Mini Project: Build & Apply

Prometheus for Metrics Collection in Kubernetes

📖 Scenario: You are managing a Kubernetes cluster and want to monitor the health and performance of your applications. Prometheus is a popular tool that collects metrics from your cluster and applications to help you understand their behavior.In this project, you will set up a basic Prometheus configuration to scrape metrics from a sample application running in your Kubernetes cluster.

🎯 Goal: Build a simple Prometheus setup that collects metrics from a Kubernetes pod running a sample application. You will create the necessary Kubernetes resources and Prometheus configuration to enable metrics scraping.

📋 What You'll Learn

Create a Kubernetes ConfigMap with Prometheus scrape configuration

Deploy a sample application pod exposing metrics on port 8080

Create a Prometheus Pod that uses the ConfigMap to scrape metrics

Verify Prometheus successfully scrapes metrics from the sample application

💡 Why This Matters

🌍 Real World

Prometheus is widely used in real Kubernetes environments to monitor applications and infrastructure by collecting metrics automatically.

💼 Career

Understanding how to configure Prometheus in Kubernetes is a key skill for DevOps engineers and site reliability engineers to ensure system health and performance.

1

Create a sample application pod exposing metrics

Write a Kubernetes Pod manifest named sample-app-pod.yaml that creates a pod called sample-app in the default namespace. The pod should run the image prom/prometheus-example-app and expose container port 8080.

Kubernetes

# Write the YAML manifest for the sample-app pod
# Your code here

Hint

Use kind: Pod, set metadata.name to sample-app, and specify the container image prom/prometheus-example-app. Expose port 8080 in containerPort.

2

Create a Prometheus scrape configuration ConfigMap

Create a Kubernetes ConfigMap manifest named prometheus-config.yaml with the name prometheus-config in the default namespace. The ConfigMap should contain a key prometheus.yml with the following scrape config YAML content exactly:

scrape_configs:
  - job_name: 'sample-app'
    static_configs:
      - targets: ['sample-app:8080']

Kubernetes

apiVersion: v1
kind: Pod
metadata:
  name: sample-app
  namespace: default
spec:
  containers:
  - name: sample-app
    image: prom/prometheus-example-app
    ports:
    - containerPort: 8080

# Add the ConfigMap manifest for prometheus-config below
# Your code here

Hint

Use kind: ConfigMap with metadata.name: prometheus-config. The key prometheus.yml should contain the scrape_configs YAML exactly as shown.

3

Deploy a Prometheus pod using the ConfigMap

Add a Kubernetes Pod manifest named prometheus-pod.yaml to your existing YAML that creates a pod called prometheus in the default namespace. The pod should run the image prom/prometheus and mount the ConfigMap prometheus-config at /etc/prometheus/ as a volume. The container should use the command --config.file=/etc/prometheus/prometheus.yml to load the config.

Kubernetes

apiVersion: v1
kind: Pod
metadata:
  name: sample-app
  namespace: default
spec:
  containers:
  - name: sample-app
    image: prom/prometheus-example-app
    ports:
    - containerPort: 8080
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus-config
  namespace: default
data:
  prometheus.yml: |
    scrape_configs:
      - job_name: 'sample-app'
        static_configs:
          - targets: ['sample-app:8080']

# Add the Prometheus pod manifest below
# Your code here

Hint

Create a pod named prometheus using image prom/prometheus. Mount the ConfigMap prometheus-config at /etc/prometheus/ and pass --config.file=/etc/prometheus/prometheus.yml as an argument.

4

Verify Prometheus scrapes metrics from sample app

Run the command kubectl logs prometheus to check the Prometheus pod logs. Confirm that the logs contain the text Scrape succeeded indicating metrics were collected from sample-app. Write a print statement in Python that outputs exactly Scrape succeeded to simulate this verification.

Kubernetes

apiVersion: v1
kind: Pod
metadata:
  name: sample-app
  namespace: default
spec:
  containers:
  - name: sample-app
    image: prom/prometheus-example-app
    ports:
    - containerPort: 8080
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus-config
  namespace: default
data:
  prometheus.yml: |
    scrape_configs:
      - job_name: 'sample-app'
        static_configs:
          - targets: ['sample-app:8080']
---
apiVersion: v1
kind: Pod
metadata:
  name: prometheus
  namespace: default
spec:
  containers:
  - name: prometheus
    image: prom/prometheus
    args:
      - '--config.file=/etc/prometheus/prometheus.yml'
    volumeMounts:
      - name: config-volume
        mountPath: /etc/prometheus/
  volumes:
    - name: config-volume
      configMap:
        name: prometheus-config

# Write a print statement to output 'Scrape succeeded'
# Your code here

Hint

Use print("Scrape succeeded") exactly to simulate checking Prometheus logs.

Practice

1. What is the main purpose of Prometheus in a Kubernetes environment?

easy

A. To deploy applications automatically

B. To collect and store metrics data for monitoring

C. To manage Kubernetes cluster nodes

D. To provide a user interface for Kubernetes

Solution

1. Step 1: Understand Prometheus role

   Prometheus is designed to collect numerical data called metrics from applications and systems.

2. Step 2: Identify its main function in Kubernetes

   In Kubernetes, Prometheus collects metrics to monitor app health and performance.

3. Final Answer:

   To collect and store metrics data for monitoring -> Option B

4. Quick Check:

   Prometheus collects metrics = A [OK]

Hint: Prometheus = metrics collection tool [OK]

Common Mistakes:

• Confusing Prometheus with deployment tools
• Thinking Prometheus manages nodes
• Assuming Prometheus is a UI tool

2. Which Kubernetes resource is used to tell Prometheus which services to monitor?

easy

A. ServiceMonitor

B. PodMonitor

C. ConfigMap

D. Ingress

Solution

1. Step 1: Identify Prometheus monitoring resources

   Prometheus uses special Kubernetes custom resources to know what to watch.

2. Step 2: Recognize ServiceMonitor's role

   ServiceMonitor tells Prometheus which Kubernetes services to scrape metrics from.

3. Final Answer:

   ServiceMonitor -> Option A

4. Quick Check:

   ServiceMonitor selects services for Prometheus [OK]

Hint: ServiceMonitor = tells Prometheus what to watch [OK]

Common Mistakes:

• Confusing PodMonitor with ServiceMonitor
• Using ConfigMap for monitoring targets
• Thinking Ingress controls Prometheus scraping

3. Given this snippet of a ServiceMonitor YAML, what is the scrape interval Prometheus will use?

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: example-monitor
spec:
  endpoints:
  - port: web
    interval: 15s
  selector:
    matchLabels:
      app: example

medium

A. 5 seconds

B. 30 seconds

C. 15 seconds

D. 1 minute

Solution

1. Step 1: Locate the interval field in YAML

   The interval is set under endpoints as 'interval: 15s'.

2. Step 2: Understand interval meaning

   This means Prometheus scrapes metrics every 15 seconds from the specified port.

3. Final Answer:

   15 seconds -> Option C

4. Quick Check:

   interval: 15s means 15 seconds [OK]

Hint: Check 'interval' value under endpoints [OK]

Common Mistakes:

• Ignoring the interval field and guessing default
• Confusing seconds with minutes
• Assuming interval is global, not per endpoint

4. You created a ServiceMonitor but Prometheus is not scraping metrics from your service. Which of these is a likely cause?

medium

A. The ServiceMonitor selector labels do not match the service labels

B. The Prometheus server is not running on the cluster

C. The service port is not exposed in the ServiceMonitor endpoints

D. All of the above

Solution

1. Step 1: Check label matching

   If ServiceMonitor selector labels don't match service labels, Prometheus won't find the service.

2. Step 2: Verify Prometheus server status and endpoint config

   Prometheus must be running and the service port must be correctly specified in endpoints to scrape metrics.

3. Final Answer:

   All of the above -> Option D

4. Quick Check:

   Any mismatch or missing config stops scraping [OK]

Hint: Check labels, server status, and endpoints all match [OK]

Common Mistakes:

• Only checking one cause and ignoring others
• Assuming Prometheus always runs by default
• Forgetting to expose correct port in ServiceMonitor

5. You want Prometheus to scrape metrics from multiple services with different scrape intervals. How should you configure this in Kubernetes?

hard

A. Create separate ServiceMonitor resources for each service with their specific intervals

B. Set a global scrape interval in Prometheus config and ignore ServiceMonitor intervals

C. Create one ServiceMonitor with multiple endpoints, each having its own interval

D. Use a ConfigMap to list all services and intervals for Prometheus

Solution

1. Step 1: Understand ServiceMonitor scope

   Each ServiceMonitor targets services with specific scrape configs; intervals are per endpoint.

2. Step 2: Manage different intervals

   To have different intervals per service, create separate ServiceMonitors with their own intervals.

3. Step 3: Why not other options?

   One ServiceMonitor with multiple endpoints cannot set different intervals per service easily; global config overrides intervals; ConfigMap does not control scraping targets.

4. Final Answer:

   Create separate ServiceMonitor resources for each service with their specific intervals -> Option A

5. Quick Check:

   Separate ServiceMonitors allow different intervals [OK]

Hint: Use separate ServiceMonitors for different intervals [OK]

Common Mistakes:

• Trying to set different intervals in one ServiceMonitor
• Ignoring ServiceMonitor intervals in favor of global config
• Using ConfigMap incorrectly for scraping targets