Alerting with Prometheus Alertmanager in Kubernetes - Mini Project: Build & Apply

Alerting with Prometheus Alertmanager

📖 Scenario: You are managing a Kubernetes cluster and want to monitor the health of your applications. You will set up Prometheus Alertmanager to send alerts when your application metrics cross certain thresholds.

🎯 Goal: Build a simple Prometheus alerting rule and configure Alertmanager to send notifications when the alert fires.

📋 What You'll Learn

Create a Prometheus alerting rule for high CPU usage

Configure Alertmanager with a basic alert receiver

Connect Prometheus to Alertmanager

Test the alert and display the alert firing message

💡 Why This Matters

🌍 Real World

Monitoring applications in Kubernetes clusters is critical to detect issues early. Prometheus and Alertmanager help automate alerting to notify teams.

💼 Career

DevOps engineers and SREs use Prometheus and Alertmanager daily to maintain system reliability and respond quickly to problems.

1

Create Prometheus Alerting Rule

Create a file called cpu_alert_rule.yaml with a Prometheus alerting rule named HighCpuUsage that triggers when rate(container_cpu_usage_seconds_total[5m]) > 0.5 for 5 minutes.

Kubernetes

# Your code here

Need a hint?

Use the alert, expr, and for fields inside a rules list.

2

Configure Alertmanager Receiver

Create a file called alertmanager.yaml with a receiver named team-email that sends alerts to the email address [e-mail].

Kubernetes

groups:
  - name: example
    rules:
      - alert: HighCpuUsage
        expr: rate(container_cpu_usage_seconds_total[5m]) > 0.5
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High CPU usage detected"
          description: "CPU usage is above 0.5 for more than 5 minutes."

# Your code here

Need a hint?

Define receivers with a name and email_configs including the to email address.

3

Connect Prometheus to Alertmanager

Add the Alertmanager configuration to Prometheus by creating a prometheus.yaml file that includes the Alertmanager URL http://alertmanager:9093 under the alerting section.

Kubernetes

groups:
  - name: example
    rules:
      - alert: HighCpuUsage
        expr: rate(container_cpu_usage_seconds_total[5m]) > 0.5
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High CPU usage detected"
          description: "CPU usage is above 0.5 for more than 5 minutes."

receivers:
  - name: team-email
    email_configs:
      - to: [e-mail]

# Your code here

Need a hint?

Under alerting, add alertmanagers with static_configs and targets listing the Alertmanager URL.

4

Test Alert and Display Output

Print the message Alert fired: HighCpuUsage to simulate the alert firing.

Kubernetes

groups:
  - name: example
    rules:
      - alert: HighCpuUsage
        expr: rate(container_cpu_usage_seconds_total[5m]) > 0.5
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High CPU usage detected"
          description: "CPU usage is above 0.5 for more than 5 minutes."

receivers:
  - name: team-email
    email_configs:
      - to: [e-mail]

alerting:
  alertmanagers:
    - static_configs:
        - targets:
            - 'alertmanager:9093'

# Your code here

Need a hint?

Use a print statement to show the alert firing message exactly as given.