Alerting with Prometheus Alertmanager in Kubernetes - Step-by-Step Execution

Process Flow - Alerting with Prometheus Alertmanager

Prometheus scrapes metrics

↓

Prometheus evaluates alert rules

↓

Alert fires if condition met

↓

Alert sent to Alertmanager

↓

Alertmanager groups and deduplicates alerts

↓

Alertmanager sends notifications

↓

Notification received by user or system

Prometheus collects metrics, checks alert rules, sends alerts to Alertmanager, which groups alerts and sends notifications.

Execution Sample

Kubernetes

groups:
- name: example
  rules:
  - alert: HighCpuUsage
    expr: cpu_usage > 80
    for: 5m
    labels:
      severity: critical
    annotations:
      summary: "CPU usage is above 80%"

This alert rule fires if CPU usage is above 80% for 5 minutes, labeling it as critical.

Process Table

Step	Prometheus Metric	Alert Rule Condition	Condition Result	Alertmanager Action	Notification Sent
1	cpu_usage=75	cpu_usage > 80 for 5m	False	No alert sent	No
2	cpu_usage=85 (for 3m)	cpu_usage > 80 for 5m	False (time not reached)	No alert sent	No
3	cpu_usage=85 (for 5m)	cpu_usage > 80 for 5m	True	Alert fired and sent	Yes
4	cpu_usage=85 (continued)	Alert active	True	Alertmanager groups alert	Notification sent
5	cpu_usage=70	cpu_usage > 80 for 5m	False	Alert resolved	Resolved notification sent

💡 Alert resolved when cpu_usage drops below threshold; alert lifecycle ends.

Status Tracker

Variable	Start	After Step 1	After Step 2	After Step 3	After Step 4	After Step 5
cpu_usage	70	75	85	85	85	70
Alert State	Inactive	Inactive	Inactive	Firing	Firing	Resolved
Notification Sent	No	No	No	Yes	Yes	Yes (resolved)

Key Moments - 3 Insights

Why doesn't the alert fire immediately when cpu_usage first goes above 80?

Because the alert rule requires cpu_usage > 80 for 5 continuous minutes (see Step 2 and 3 in execution_table). The condition must hold for the full duration before firing.

What does Alertmanager do when it receives multiple alerts for the same issue?

Alertmanager groups and deduplicates alerts to avoid spamming notifications, as shown in Step 4 where it groups the alert before sending notification.

How does the alert get resolved?

When the metric drops below the threshold (cpu_usage < 80), Prometheus marks the alert as resolved and Alertmanager sends a resolved notification (Step 5).

Visual Quiz - 3 Questions

Look at the execution_table, at which step does the alert first fire?

A Step 1

B Step 3

C Step 2

D Step 5

💡 Hint

Check the 'Condition Result' column to see when it becomes True for the first time.

According to variable_tracker, what is the Alert State after Step 4?

A Firing

B Inactive

C Resolved

D Unknown

💡 Hint

Look at the 'Alert State' row under 'After Step 4' column.

If the cpu_usage stayed above 80 for only 3 minutes, what would happen to the alert?

A Alert would fire immediately

B Alert would fire after 3 minutes

C Alert would never fire

D Alertmanager would send resolved notification

💡 Hint

Refer to Step 2 in execution_table where condition is false because time threshold is not met.

Concept Snapshot

Prometheus Alerting:
- Define alert rules with conditions and duration
- Prometheus evaluates rules regularly
- Alerts fire only if condition holds for specified time
- Alerts sent to Alertmanager
- Alertmanager groups alerts and sends notifications
- Alerts resolve when conditions clear

Full Transcript

Prometheus collects metrics and checks alert rules. If a metric exceeds a threshold for a set time, an alert fires. This alert is sent to Alertmanager, which groups similar alerts to avoid duplicates and sends notifications to users or systems. When the metric returns to normal, the alert resolves and Alertmanager sends a resolved notification. This process helps monitor system health and notify teams only when needed.