How to Set Up Alerting in Kubernetes: Simple Guide

To set up alerting in Kubernetes, deploy Prometheus to collect metrics and Alertmanager to handle alerts. Define alert rules in Prometheus and configure Alertmanager to send notifications via email, Slack, or other channels.

Syntax

Setting up alerting in Kubernetes involves these parts:

• Prometheus: Collects metrics from your cluster.
• Alert Rules: Define conditions to trigger alerts.
• Alertmanager: Manages alerts and sends notifications.
• Notification Config: Setup channels like email or Slack.

yaml

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: example-alert
  namespace: monitoring
spec:
  groups:
  - name: example.rules
    rules:
    - alert: HighCpuUsage
      expr: sum(rate(container_cpu_usage_seconds_total[5m])) by (pod) > 0.8
      for: 5m
      labels:
        severity: warning
      annotations:
        summary: "High CPU usage detected"
        description: "Pod {{ $labels.pod }} CPU usage is above 80% for 5 minutes."

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: alertmanager-config
  namespace: monitoring
data:
  alertmanager.yml: |
    global:
      resolve_timeout: 5m
    route:
      receiver: 'slack-notifications'
    receivers:
    - name: 'slack-notifications'
      slack_configs:
      - api_url: 'https://hooks.slack.com/services/XXX/YYY/ZZZ'
        channel: '#alerts'

Example

This example shows how to create a Prometheus alert rule for high CPU usage and configure Alertmanager to send Slack notifications.

yaml

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: cpu-alert
  namespace: monitoring
spec:
  groups:
  - name: cpu.rules
    rules:
    - alert: HighCpuUsage
      expr: sum(rate(container_cpu_usage_seconds_total[5m])) by (pod) > 0.8
      for: 5m
      labels:
        severity: critical
      annotations:
        summary: "CPU usage is high"
        description: "Pod {{ $labels.pod }} CPU usage > 80% for 5 minutes."

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: alertmanager-config
  namespace: monitoring
data:
  alertmanager.yml: |
    global:
      resolve_timeout: 5m
    route:
      receiver: 'email-alerts'
    receivers:
    - name: 'email-alerts'
      email_configs:
      - to: '[e-mail]'
        from: '[e-mail]'
        smarthost: 'smtp.example.com:587'
        auth_username: '[e-mail]'
        auth_password: 'password'
        require_tls: true

Output

PrometheusRule "cpu-alert" created
ConfigMap "alertmanager-config" created

Common Pitfalls

Common mistakes when setting up Kubernetes alerting include:

• Not deploying Alertmanager alongside Prometheus, so alerts have nowhere to go.
• Incorrect alert rule expressions causing no alerts or false alerts.
• Misconfiguring notification channels like wrong Slack webhook URLs or email SMTP settings.
• Forgetting to reload Prometheus or Alertmanager after config changes.

yaml

### Wrong alert expression (no alerts triggered)
expr: sum(rate(container_cpu_usage_seconds_total[5m])) by (pod) > 80

### Correct alert expression (CPU usage as fraction)
expr: sum(rate(container_cpu_usage_seconds_total[5m])) by (pod) > 0.8

Quick Reference

Summary tips for Kubernetes alerting setup:

• Use Prometheus to collect metrics and define alert rules.
• Deploy Alertmanager to route and send alerts.
• Test alert rules with promtool or by simulating conditions.
• Configure notification channels carefully (Slack, email, PagerDuty).
• Reload configs after changes using kubectl rollout restart or API.

Key Takeaways

Deploy Prometheus and Alertmanager in your Kubernetes cluster to enable alerting.

Write clear alert rules using PromQL expressions to detect issues like high CPU usage.

Configure Alertmanager with correct notification channels to receive alerts.

Always test alert rules and notification settings before relying on them.

Reload Prometheus and Alertmanager configurations after any changes.