Prometheus for metrics collection in Kubernetes - Practice Problems & Coding Challenges

Challenge - 5 Problems

💻 Command Output

intermediate

Prometheus Query Result Interpretation

You run the Prometheus query rate(http_requests_total[5m]) to monitor HTTP requests per second over the last 5 minutes. What type of output do you expect from this query?

A A single integer representing the total number of HTTP requests since the server started.

B A time series showing the per-second rate of HTTP requests averaged over the last 5 minutes.

C A list of all HTTP request logs collected in the last 5 minutes.

D A histogram showing the distribution of HTTP request sizes.

💡 Hint

Think about what the rate() function does in Prometheus queries.

The rate() function calculates the per-second average increase of a counter over the specified time window. Here, it returns a time series showing the average HTTP requests per second over the last 5 minutes.

❓ Configuration

intermediate

Prometheus Scrape Configuration

You want Prometheus to scrape metrics from a Kubernetes service named my-app in the default namespace every 15 seconds. Which scrape configuration snippet correctly achieves this?

A

scrape_configs:
  - job_name: 'my-app'
    file_sd_configs:
      - files: ['my-app.yaml']
    scrape_interval: 15s

B

scrape_configs:
  - job_name: 'my-app'
    static_configs:
      - targets: ['my-app.default.svc.cluster.local:9090']
    scrape_interval: 15s

C

scrape_configs:
  - job_name: 'my-app'
    kubernetes_sd_configs:
      - role: endpoints
    relabel_configs:
      - source_labels: [__meta_kubernetes_service_name]
        action: keep
        regex: my-app
    scrape_interval: 15s

D

scrape_configs:
  - job_name: 'my-app'
    kubernetes_sd_configs:
      - role: pod
    scrape_interval: 15s

💡 Hint

Use Kubernetes service discovery and filter by service name.

Option C uses Kubernetes service discovery with role: endpoints and filters targets by the service name my-app. This is the correct way to scrape metrics from a Kubernetes service.

🔀 Workflow

advanced

Prometheus Alerting Workflow

You want to create an alert that fires when the CPU usage of any pod exceeds 80% for 5 minutes. Which sequence of steps correctly describes the workflow to achieve this?

A

1. Create a Kubernetes pod annotation for CPU limit.
2. Restart Prometheus.
3. Configure Grafana dashboard.
4. Send email manually when CPU is high.

B

1. Configure Prometheus to scrape node metrics.
2. Write a query for disk usage.
3. Restart Kubernetes cluster.
4. Configure Alertmanager to silence alerts.

C

1. Write a Prometheus query to list pods.
2. Use kubectl to scale pods.
3. Restart Alertmanager.
4. Configure Prometheus scrape interval.

D

1. Write a Prometheus alerting rule with a query checking CPU usage > 80% for 5 minutes.
2. Add the rule to Prometheus alerting rules file.
3. Reload Prometheus configuration.
4. Configure Alertmanager to handle the alert notifications.

💡 Hint

Focus on alerting rules and notification setup.

To alert on CPU usage, you write an alerting rule in Prometheus, load it, and configure Alertmanager to send notifications. Other options do not follow the correct alerting workflow.

❓ Troubleshoot

advanced

Prometheus Metrics Missing from Target

You notice Prometheus is not scraping metrics from a pod exposing metrics on port 9100, but the pod is healthy and reachable. What is the most likely cause?

A The pod's metrics endpoint is not labeled or annotated correctly for Prometheus to discover it.

B Prometheus server is down and not running.

C The pod is running on a node without network connectivity.

D The pod's container image is outdated.

💡 Hint

Check how Prometheus discovers scrape targets in Kubernetes.

Prometheus relies on labels or annotations to discover pods for scraping. If missing, Prometheus won't scrape metrics even if the pod is healthy.

✅ Best Practice

expert

Efficient Prometheus Metrics Collection Strategy

Which approach is the best practice to minimize Prometheus load while collecting metrics from a large Kubernetes cluster?

A Use service discovery with relabeling to scrape only necessary pods and endpoints, and increase scrape intervals for less critical metrics.

B Scrape all pods every 5 seconds to ensure real-time data accuracy.

C Disable service discovery and manually list all pod IPs in static configs.

D Run multiple Prometheus servers scraping the same targets simultaneously without coordination.

💡 Hint

Think about selective scraping and scrape frequency.

Selective scraping using service discovery with relabeling and adjusting scrape intervals reduces load and improves performance. Other options cause unnecessary load or complexity.

Practice

1. What is the main purpose of Prometheus in a Kubernetes environment?

easy

A. To deploy applications automatically

B. To collect and store metrics data for monitoring

C. To manage Kubernetes cluster nodes

D. To provide a user interface for Kubernetes

Solution

1. Step 1: Understand Prometheus role

   Prometheus is designed to collect numerical data called metrics from applications and systems.

2. Step 2: Identify its main function in Kubernetes

   In Kubernetes, Prometheus collects metrics to monitor app health and performance.

3. Final Answer:

   To collect and store metrics data for monitoring -> Option B

4. Quick Check:

   Prometheus collects metrics = A [OK]

Hint: Prometheus = metrics collection tool [OK]

Common Mistakes:

• Confusing Prometheus with deployment tools
• Thinking Prometheus manages nodes
• Assuming Prometheus is a UI tool

2. Which Kubernetes resource is used to tell Prometheus which services to monitor?

easy

A. ServiceMonitor

B. PodMonitor

C. ConfigMap

D. Ingress

Solution

1. Step 1: Identify Prometheus monitoring resources

   Prometheus uses special Kubernetes custom resources to know what to watch.

2. Step 2: Recognize ServiceMonitor's role

   ServiceMonitor tells Prometheus which Kubernetes services to scrape metrics from.

3. Final Answer:

   ServiceMonitor -> Option A

4. Quick Check:

   ServiceMonitor selects services for Prometheus [OK]

Hint: ServiceMonitor = tells Prometheus what to watch [OK]

Common Mistakes:

• Confusing PodMonitor with ServiceMonitor
• Using ConfigMap for monitoring targets
• Thinking Ingress controls Prometheus scraping

3. Given this snippet of a ServiceMonitor YAML, what is the scrape interval Prometheus will use?

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: example-monitor
spec:
  endpoints:
  - port: web
    interval: 15s
  selector:
    matchLabels:
      app: example

medium

A. 5 seconds

B. 30 seconds

C. 15 seconds

D. 1 minute

Solution

1. Step 1: Locate the interval field in YAML

   The interval is set under endpoints as 'interval: 15s'.

2. Step 2: Understand interval meaning

   This means Prometheus scrapes metrics every 15 seconds from the specified port.

3. Final Answer:

   15 seconds -> Option C

4. Quick Check:

   interval: 15s means 15 seconds [OK]

Hint: Check 'interval' value under endpoints [OK]

Common Mistakes:

• Ignoring the interval field and guessing default
• Confusing seconds with minutes
• Assuming interval is global, not per endpoint

4. You created a ServiceMonitor but Prometheus is not scraping metrics from your service. Which of these is a likely cause?

medium

A. The ServiceMonitor selector labels do not match the service labels

B. The Prometheus server is not running on the cluster

C. The service port is not exposed in the ServiceMonitor endpoints

D. All of the above

Solution

1. Step 1: Check label matching

   If ServiceMonitor selector labels don't match service labels, Prometheus won't find the service.

2. Step 2: Verify Prometheus server status and endpoint config

   Prometheus must be running and the service port must be correctly specified in endpoints to scrape metrics.

3. Final Answer:

   All of the above -> Option D

4. Quick Check:

   Any mismatch or missing config stops scraping [OK]

Hint: Check labels, server status, and endpoints all match [OK]

Common Mistakes:

• Only checking one cause and ignoring others
• Assuming Prometheus always runs by default
• Forgetting to expose correct port in ServiceMonitor

5. You want Prometheus to scrape metrics from multiple services with different scrape intervals. How should you configure this in Kubernetes?

hard

A. Create separate ServiceMonitor resources for each service with their specific intervals

B. Set a global scrape interval in Prometheus config and ignore ServiceMonitor intervals

C. Create one ServiceMonitor with multiple endpoints, each having its own interval

D. Use a ConfigMap to list all services and intervals for Prometheus

Solution

1. Step 1: Understand ServiceMonitor scope

   Each ServiceMonitor targets services with specific scrape configs; intervals are per endpoint.

2. Step 2: Manage different intervals

   To have different intervals per service, create separate ServiceMonitors with their own intervals.

3. Step 3: Why not other options?

   One ServiceMonitor with multiple endpoints cannot set different intervals per service easily; global config overrides intervals; ConfigMap does not control scraping targets.

4. Final Answer:

   Create separate ServiceMonitor resources for each service with their specific intervals -> Option A

5. Quick Check:

   Separate ServiceMonitors allow different intervals [OK]

Hint: Use separate ServiceMonitors for different intervals [OK]

Common Mistakes:

• Trying to set different intervals in one ServiceMonitor
• Ignoring ServiceMonitor intervals in favor of global config
• Using ConfigMap incorrectly for scraping targets