Alerting with Prometheus Alertmanager in Kubernetes - Commands & Configuration

Introduction

When your applications or servers have problems, you want to know quickly so you can fix them. Prometheus Alertmanager helps by sending alerts when something goes wrong based on rules you set.

When you want to get notified if your server CPU is too high for a long time.

When your application is not responding and you want to receive an alert.

When disk space is running low and you need to clean up before it causes issues.

When a Kubernetes pod crashes repeatedly and you want to investigate.

When you want to send alerts to email or chat apps automatically.

Config File - alertmanager.yml

alertmanager.yml

global:
  resolve_timeout: 5m

route:
  receiver: 'team-email'
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 3h

receivers:
- name: 'team-email'
  email_configs:
  - to: '[e-mail]'
    from: '[e-mail]'
    smarthost: 'smtp.example.com:587'
    auth_username: '[e-mail]'
    auth_password: 'securepassword'
    require_tls: true

inhibit_rules:
- source_match:
    severity: 'critical'
  target_match:
    severity: 'warning'
  equal: ['alertname', 'dev', 'instance']

global: Sets general settings like how long to wait before marking alerts as resolved.

route: Defines how alerts are grouped and sent to receivers.

receivers: Lists where alerts go, here an email address with SMTP settings.

inhibit_rules: Prevents sending less important alerts if a critical alert is already firing for the same issue.

Commands

This command applies the Alertmanager configuration to the monitoring namespace in Kubernetes so Alertmanager knows how to send alerts.

Terminal

kubectl apply -f alertmanager.yml -n monitoring

Expected Output

configmap/alertmanager configured

→

-f - Specifies the configuration file to apply

→

-n - Specifies the Kubernetes namespace

Check that the Alertmanager pod is running properly after applying the configuration.

Terminal

kubectl get pods -n monitoring

Expected Output

NAME                                   READY   STATUS    RESTARTS   AGE

alertmanager-main-0                     2/2     Running   0          3m

→

-n - Shows pods in the monitoring namespace

View the logs of the Alertmanager pod to verify it started correctly and is using the new config.

Terminal

kubectl logs alertmanager-main-0 -n monitoring

Expected Output

level=info ts=2024-06-01T12:00:00.000Z caller=main.go:123 msg="Starting Alertmanager" version="0.25.0"

level=info ts=2024-06-01T12:00:01.000Z caller=main.go:456 msg="Loading configuration file" file=alertmanager.yml

level=info ts=2024-06-01T12:00:01.500Z caller=main.go:789 msg="Listening on :9093"

Key Concept

If you remember nothing else from this pattern, remember: Alertmanager sends notifications based on rules you configure to help you fix problems fast.

Common Mistakes

Not applying the Alertmanager config in the correct Kubernetes namespace.

Alertmanager runs in a specific namespace, so applying config elsewhere means it won't pick up your settings.

Always use the correct namespace with -n monitoring when applying or checking Alertmanager resources.

Forgetting to configure receivers like email or chat in alertmanager.yml.

Without receivers, Alertmanager has nowhere to send alerts, so you won't get notified.

Add at least one receiver with proper settings like email_configs or webhook_configs.

Not checking Alertmanager pod logs after applying config.

Errors in the config file can prevent Alertmanager from starting or sending alerts.

Always check logs with kubectl logs to confirm Alertmanager loaded the config without errors.

Summary

Apply the Alertmanager configuration file to set alert rules and receivers.

Verify the Alertmanager pod is running in the monitoring namespace.

Check Alertmanager logs to ensure the configuration loaded correctly and it is ready to send alerts.

Practice

1. What is the main role of Prometheus Alertmanager in Kubernetes monitoring?

easy

A. To collect metrics from Kubernetes nodes

B. To send notifications when Prometheus detects alerts

C. To store logs from containers

D. To deploy applications automatically

Solution

1. Step 1: Understand Prometheus and Alertmanager roles

   Prometheus collects metrics and detects alerts based on rules.

2. Step 2: Identify Alertmanager's function

   Alertmanager receives alerts from Prometheus and sends notifications to users or systems.

3. Final Answer:

   To send notifications when Prometheus detects alerts -> Option B

4. Quick Check:

   Alertmanager = Notification sender [OK]

Hint: Alertmanager handles alert notifications, not metric collection [OK]

Common Mistakes:

• Confusing Alertmanager with Prometheus server
• Thinking Alertmanager collects metrics
• Assuming Alertmanager deploys apps

2. Which of the following is the correct YAML snippet to define an email receiver named 'team-email' in Alertmanager?

easy

A. receivers:
- name: team-email
email_configs:
- to: '[e-mail]'

B. receivers:
- team-email:
email: '[e-mail]'

C. receiver:
name: team-email
email: '[e-mail]'

D. receivers:
- name: team-email
slack_configs:
- channel: '#alerts'

Solution

1. Step 1: Review Alertmanager receiver syntax

   Receivers are defined under 'receivers' list with 'name' and config type like 'email_configs'.

2. Step 2: Match correct YAML structure

   receivers: - name: team-email email_configs: - to: '[e-mail]' correctly uses 'receivers', 'name', and 'email_configs' with 'to' field.

3. Final Answer:

   Correct YAML with 'receivers', 'name', and 'email_configs' -> Option A

4. Quick Check:

   Receiver YAML uses 'name' and 'email_configs' [OK]

Hint: Receiver configs use 'name' and specific config like 'email_configs' [OK]

Common Mistakes:

• Using 'receiver' instead of 'receivers'
• Incorrect nesting of email fields
• Confusing slack_configs with email_configs

3. Given this Alertmanager config snippet, what will happen when multiple alerts fire simultaneously?

route:
  group_by: ['alertname']
  receiver: 'team-email'
receivers:
  - name: 'team-email'
    email_configs:
      - to: '[e-mail]'

medium

A. Alerts with the same 'alertname' will be grouped into one notification

B. Each alert will send a separate email regardless of grouping

C. No alerts will be sent because 'group_wait' is missing

D. Alerts will be sent only to Slack, not email

Solution

1. Step 1: Understand 'group_by' in Alertmanager route

   'group_by' groups alerts by specified labels; here, alerts with same 'alertname' are grouped.

2. Step 2: Check receiver and notification method

   Receiver 'team-email' uses email_configs, so grouped alerts send one email per alertname.

3. Final Answer:

   Alerts with the same 'alertname' will be grouped into one notification -> Option A

4. Quick Check:

   'group_by' controls alert grouping [OK]

Hint: 'group_by' label controls alert grouping in notifications [OK]

Common Mistakes:

• Assuming each alert sends separate email
• Thinking 'group_wait' is required to send alerts
• Confusing receiver type with Slack

4. You configured Alertmanager but no notifications are sent. Which of these is a likely cause based on this snippet?

receivers:
  - name: 'team-email'
    email_configs:
      - to: '[e-mail]'
route:
  receiver: 'team-email'
  group_by: ['alertname']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 1h

medium

A. Alertmanager does not support email notifications

B. Incorrect 'group_by' label causes no alerts

C. Receiver name does not match route receiver

D. Missing SMTP server configuration in Alertmanager

Solution

1. Step 1: Check email notification requirements

   Email notifications require SMTP server settings in Alertmanager config, not shown here.

2. Step 2: Verify receiver and route match

   Receiver name 'team-email' matches route receiver, so routing is correct.

3. Final Answer:

   Missing SMTP server configuration in Alertmanager -> Option D

4. Quick Check:

   Email needs SMTP setup to send alerts [OK]

Hint: Email alerts need SMTP server configured in Alertmanager [OK]

Common Mistakes:

• Assuming 'group_by' label stops alerts
• Thinking receiver name mismatch causes no alerts here
• Believing Alertmanager can't send emails

5. You want to avoid alert spam by grouping alerts by both 'alertname' and 'severity', and send notifications to Slack channel '#alerts'. Which Alertmanager route and receiver config is correct?

hard

A. route:
group_by: ['severity']
receiver: 'email-team'
receivers:
- name: 'email-team'
slack_configs:
- channel: '#alerts'

B. route:
group_by: ['alertname']
receiver: 'slack-notifications'
receivers:
- name: 'slack-notifications'
email_configs:
- to: '#alerts'

C. route:
group_by: ['alertname', 'severity']
receiver: 'slack-notifications'
receivers:
- name: 'slack-notifications'
slack_configs:
- channel: '#alerts'
send_resolved: true

D. route:
group_by: ['alertname', 'severity']
receiver: 'email-team'
receivers:
- name: 'email-team'
email_configs:
- to: '[e-mail]'

Solution

1. Step 1: Set grouping labels in route

   To group alerts by 'alertname' and 'severity', list both in 'group_by'.

2. Step 2: Configure Slack receiver correctly

   Receiver named 'slack-notifications' uses 'slack_configs' with channel '#alerts' and 'send_resolved' true.

3. Final Answer:

   Route groups by alertname and severity; receiver sends Slack messages to #alerts -> Option C

4. Quick Check:

   Group by multiple labels and use correct receiver config [OK]

Hint: Group by multiple labels and match receiver type to notification [OK]

Common Mistakes:

• Using email_configs for Slack notifications
• Grouping by only one label when two needed
• Mismatch between route receiver and receiver name