Alerting with Prometheus Alertmanager in Kubernetes - Step-by-Step Execution

Process Flow - Alerting with Prometheus Alertmanager

Prometheus scrapes metrics

↓

Prometheus evaluates alert rules

↓

Alert fires if condition met

↓

Alert sent to Alertmanager

↓

Alertmanager groups and deduplicates alerts

↓

Alertmanager sends notifications

↓

Notification received by user or system

Prometheus collects metrics, checks alert rules, sends alerts to Alertmanager, which groups alerts and sends notifications.

Execution Sample

Kubernetes

groups:
- name: example
  rules:
  - alert: HighCpuUsage
    expr: cpu_usage > 80
    for: 5m
    labels:
      severity: critical
    annotations:
      summary: "CPU usage is above 80%"

This alert rule fires if CPU usage is above 80% for 5 minutes, labeling it as critical.

Process Table

Step	Prometheus Metric	Alert Rule Condition	Condition Result	Alertmanager Action	Notification Sent
1	cpu_usage=75	cpu_usage > 80 for 5m	False	No alert sent	No
2	cpu_usage=85 (for 3m)	cpu_usage > 80 for 5m	False (time not reached)	No alert sent	No
3	cpu_usage=85 (for 5m)	cpu_usage > 80 for 5m	True	Alert fired and sent	Yes
4	cpu_usage=85 (continued)	Alert active	True	Alertmanager groups alert	Notification sent
5	cpu_usage=70	cpu_usage > 80 for 5m	False	Alert resolved	Resolved notification sent

💡 Alert resolved when cpu_usage drops below threshold; alert lifecycle ends.

Status Tracker

Variable	Start	After Step 1	After Step 2	After Step 3	After Step 4	After Step 5
cpu_usage	70	75	85	85	85	70
Alert State	Inactive	Inactive	Inactive	Firing	Firing	Resolved
Notification Sent	No	No	No	Yes	Yes	Yes (resolved)

Key Moments - 3 Insights

Why doesn't the alert fire immediately when cpu_usage first goes above 80?

Because the alert rule requires cpu_usage > 80 for 5 continuous minutes (see Step 2 and 3 in execution_table). The condition must hold for the full duration before firing.

What does Alertmanager do when it receives multiple alerts for the same issue?

Alertmanager groups and deduplicates alerts to avoid spamming notifications, as shown in Step 4 where it groups the alert before sending notification.

How does the alert get resolved?

When the metric drops below the threshold (cpu_usage < 80), Prometheus marks the alert as resolved and Alertmanager sends a resolved notification (Step 5).

Visual Quiz - 3 Questions

Look at the execution_table, at which step does the alert first fire?

A Step 1

B Step 3

C Step 2

D Step 5

💡 Hint

Check the 'Condition Result' column to see when it becomes True for the first time.

According to variable_tracker, what is the Alert State after Step 4?

A Firing

B Inactive

C Resolved

D Unknown

💡 Hint

Look at the 'Alert State' row under 'After Step 4' column.

If the cpu_usage stayed above 80 for only 3 minutes, what would happen to the alert?

A Alert would fire immediately

B Alert would fire after 3 minutes

C Alert would never fire

D Alertmanager would send resolved notification

💡 Hint

Refer to Step 2 in execution_table where condition is false because time threshold is not met.

Concept Snapshot

Prometheus Alerting:
- Define alert rules with conditions and duration
- Prometheus evaluates rules regularly
- Alerts fire only if condition holds for specified time
- Alerts sent to Alertmanager
- Alertmanager groups alerts and sends notifications
- Alerts resolve when conditions clear

Full Transcript

Prometheus collects metrics and checks alert rules. If a metric exceeds a threshold for a set time, an alert fires. This alert is sent to Alertmanager, which groups similar alerts to avoid duplicates and sends notifications to users or systems. When the metric returns to normal, the alert resolves and Alertmanager sends a resolved notification. This process helps monitor system health and notify teams only when needed.

Practice

1. What is the main role of Prometheus Alertmanager in Kubernetes monitoring?

easy

A. To collect metrics from Kubernetes nodes

B. To send notifications when Prometheus detects alerts

C. To store logs from containers

D. To deploy applications automatically

Solution

1. Step 1: Understand Prometheus and Alertmanager roles

   Prometheus collects metrics and detects alerts based on rules.

2. Step 2: Identify Alertmanager's function

   Alertmanager receives alerts from Prometheus and sends notifications to users or systems.

3. Final Answer:

   To send notifications when Prometheus detects alerts -> Option B

4. Quick Check:

   Alertmanager = Notification sender [OK]

Hint: Alertmanager handles alert notifications, not metric collection [OK]

Common Mistakes:

• Confusing Alertmanager with Prometheus server
• Thinking Alertmanager collects metrics
• Assuming Alertmanager deploys apps

2. Which of the following is the correct YAML snippet to define an email receiver named 'team-email' in Alertmanager?

easy

A. receivers:
- name: team-email
email_configs:
- to: '[e-mail]'

B. receivers:
- team-email:
email: '[e-mail]'

C. receiver:
name: team-email
email: '[e-mail]'

D. receivers:
- name: team-email
slack_configs:
- channel: '#alerts'

Solution

1. Step 1: Review Alertmanager receiver syntax

   Receivers are defined under 'receivers' list with 'name' and config type like 'email_configs'.

2. Step 2: Match correct YAML structure

   receivers: - name: team-email email_configs: - to: '[e-mail]' correctly uses 'receivers', 'name', and 'email_configs' with 'to' field.

3. Final Answer:

   Correct YAML with 'receivers', 'name', and 'email_configs' -> Option A

4. Quick Check:

   Receiver YAML uses 'name' and 'email_configs' [OK]

Hint: Receiver configs use 'name' and specific config like 'email_configs' [OK]

Common Mistakes:

• Using 'receiver' instead of 'receivers'
• Incorrect nesting of email fields
• Confusing slack_configs with email_configs

3. Given this Alertmanager config snippet, what will happen when multiple alerts fire simultaneously?

route:
  group_by: ['alertname']
  receiver: 'team-email'
receivers:
  - name: 'team-email'
    email_configs:
      - to: '[e-mail]'

medium

A. Alerts with the same 'alertname' will be grouped into one notification

B. Each alert will send a separate email regardless of grouping

C. No alerts will be sent because 'group_wait' is missing

D. Alerts will be sent only to Slack, not email

Solution

1. Step 1: Understand 'group_by' in Alertmanager route

   'group_by' groups alerts by specified labels; here, alerts with same 'alertname' are grouped.

2. Step 2: Check receiver and notification method

   Receiver 'team-email' uses email_configs, so grouped alerts send one email per alertname.

3. Final Answer:

   Alerts with the same 'alertname' will be grouped into one notification -> Option A

4. Quick Check:

   'group_by' controls alert grouping [OK]

Hint: 'group_by' label controls alert grouping in notifications [OK]

Common Mistakes:

• Assuming each alert sends separate email
• Thinking 'group_wait' is required to send alerts
• Confusing receiver type with Slack

4. You configured Alertmanager but no notifications are sent. Which of these is a likely cause based on this snippet?

receivers:
  - name: 'team-email'
    email_configs:
      - to: '[e-mail]'
route:
  receiver: 'team-email'
  group_by: ['alertname']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 1h

medium

A. Alertmanager does not support email notifications

B. Incorrect 'group_by' label causes no alerts

C. Receiver name does not match route receiver

D. Missing SMTP server configuration in Alertmanager

Solution

1. Step 1: Check email notification requirements

   Email notifications require SMTP server settings in Alertmanager config, not shown here.

2. Step 2: Verify receiver and route match

   Receiver name 'team-email' matches route receiver, so routing is correct.

3. Final Answer:

   Missing SMTP server configuration in Alertmanager -> Option D

4. Quick Check:

   Email needs SMTP setup to send alerts [OK]

Hint: Email alerts need SMTP server configured in Alertmanager [OK]

Common Mistakes:

• Assuming 'group_by' label stops alerts
• Thinking receiver name mismatch causes no alerts here
• Believing Alertmanager can't send emails

5. You want to avoid alert spam by grouping alerts by both 'alertname' and 'severity', and send notifications to Slack channel '#alerts'. Which Alertmanager route and receiver config is correct?

hard

A. route:
group_by: ['severity']
receiver: 'email-team'
receivers:
- name: 'email-team'
slack_configs:
- channel: '#alerts'

B. route:
group_by: ['alertname']
receiver: 'slack-notifications'
receivers:
- name: 'slack-notifications'
email_configs:
- to: '#alerts'

C. route:
group_by: ['alertname', 'severity']
receiver: 'slack-notifications'
receivers:
- name: 'slack-notifications'
slack_configs:
- channel: '#alerts'
send_resolved: true

D. route:
group_by: ['alertname', 'severity']
receiver: 'email-team'
receivers:
- name: 'email-team'
email_configs:
- to: '[e-mail]'

Solution

1. Step 1: Set grouping labels in route

   To group alerts by 'alertname' and 'severity', list both in 'group_by'.

2. Step 2: Configure Slack receiver correctly

   Receiver named 'slack-notifications' uses 'slack_configs' with channel '#alerts' and 'send_resolved' true.

3. Final Answer:

   Route groups by alertname and severity; receiver sends Slack messages to #alerts -> Option C

4. Quick Check:

   Group by multiple labels and use correct receiver config [OK]

Hint: Group by multiple labels and match receiver type to notification [OK]

Common Mistakes:

• Using email_configs for Slack notifications
• Grouping by only one label when two needed
• Mismatch between route receiver and receiver name