Alerting with Prometheus Alertmanager in Kubernetes - Interactive Code Practice

Practice - 5 Tasks

1fill in blank

easy

Complete the code to specify the Alertmanager service in Prometheus configuration.

Kubernetes

alerting:
  alertmanagers:
  - static_configs:
    - targets: ['[1]']

Drag options to blanks, or click blank then click option'

Aalertmanager:9093

Bprometheus:9090

Cnode-exporter:9100

Dgrafana:3000

Attempts:

3 left

💡 Hint

Common Mistakes

Using Prometheus port 9090 instead of Alertmanager port 9093.

Using unrelated service names like Grafana or node-exporter.

The Alertmanager service usually runs on port 9093 and is referenced by its service name in Kubernetes, such as 'alertmanager:9093'.

2fill in blank

medium

Complete the Alertmanager configuration to set the receiver name.

Kubernetes

route:
  receiver: '[1]'

Drag options to blanks, or click blank then click option'

Adefault

Bprometheus

Cemail-notifications

Dslack-alerts

Attempts:

3 left

💡 Hint

Common Mistakes

Using a receiver name not defined in the configuration.

Confusing receiver names with service names.

The 'default' receiver is commonly used as the main receiver in Alertmanager routing.

3fill in blank

hard

Fix the error in the Alertmanager receiver configuration to send alerts to Slack.

Kubernetes

receivers:
- name: 'slack-alerts'
  slack_configs:
  - api_url: '[1]'
    channel: '#alerts'

Drag options to blanks, or click blank then click option'

Ahttp://localhost:9093

Bhttps://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXXXXXXXX

Csmtp://user:pass@mailserver

Dhttp://alertmanager:9093

Attempts:

3 left

💡 Hint

Common Mistakes

Using local or HTTP URLs instead of Slack webhook URLs.

Confusing Slack webhook URL with SMTP or Alertmanager URLs.

The Slack webhook URL must be a valid HTTPS URL provided by Slack for incoming webhooks.

4fill in blank

hard

Fill both blanks to create a Prometheus alert rule that triggers when CPU usage is high.

Kubernetes

groups:
- name: cpu_alerts
  rules:
  - alert: HighCPUUsage
    expr: sum(rate(container_cpu_usage_seconds_total[1][5m])) by (instance) [2] 0.8
    for: 2m
    labels:
      severity: warning

Drag options to blanks, or click blank then click option'

Acontainer="myapp"

B>

C<

Djob="kubelet"

Attempts:

3 left

💡 Hint

Common Mistakes

Using incorrect label selectors or operators.

Forgetting to wrap label selectors in curly braces.

The expression filters CPU usage for the container named 'myapp' and triggers if usage is greater than 0.8.

5fill in blank

hard

Fill all three blanks to define an Alertmanager route that sends critical alerts to email and slack receivers.

Kubernetes

route:
  group_by: ['alertname']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 3h
  receiver: '[1]'
  routes:
  - match:
      severity: '[2]'
    receiver: '[3]'

Drag options to blanks, or click blank then click option'

Adefault

Bcritical

Cemail-notifications

Dslack-alerts

Attempts:

3 left

💡 Hint

Common Mistakes

Mixing receiver names or severity labels incorrectly.

Using receivers not defined in the configuration.

The main route uses 'default' receiver, and critical alerts are matched to 'email-notifications' receiver.

Practice

1. What is the main role of Prometheus Alertmanager in Kubernetes monitoring?

easy

A. To collect metrics from Kubernetes nodes

B. To send notifications when Prometheus detects alerts

C. To store logs from containers

D. To deploy applications automatically

Solution

1. Step 1: Understand Prometheus and Alertmanager roles

   Prometheus collects metrics and detects alerts based on rules.

2. Step 2: Identify Alertmanager's function

   Alertmanager receives alerts from Prometheus and sends notifications to users or systems.

3. Final Answer:

   To send notifications when Prometheus detects alerts -> Option B

4. Quick Check:

   Alertmanager = Notification sender [OK]

Hint: Alertmanager handles alert notifications, not metric collection [OK]

Common Mistakes:

• Confusing Alertmanager with Prometheus server
• Thinking Alertmanager collects metrics
• Assuming Alertmanager deploys apps

2. Which of the following is the correct YAML snippet to define an email receiver named 'team-email' in Alertmanager?

easy

A. receivers:
- name: team-email
email_configs:
- to: '[e-mail]'

B. receivers:
- team-email:
email: '[e-mail]'

C. receiver:
name: team-email
email: '[e-mail]'

D. receivers:
- name: team-email
slack_configs:
- channel: '#alerts'

Solution

1. Step 1: Review Alertmanager receiver syntax

   Receivers are defined under 'receivers' list with 'name' and config type like 'email_configs'.

2. Step 2: Match correct YAML structure

   receivers: - name: team-email email_configs: - to: '[e-mail]' correctly uses 'receivers', 'name', and 'email_configs' with 'to' field.

3. Final Answer:

   Correct YAML with 'receivers', 'name', and 'email_configs' -> Option A

4. Quick Check:

   Receiver YAML uses 'name' and 'email_configs' [OK]

Hint: Receiver configs use 'name' and specific config like 'email_configs' [OK]

Common Mistakes:

• Using 'receiver' instead of 'receivers'
• Incorrect nesting of email fields
• Confusing slack_configs with email_configs

3. Given this Alertmanager config snippet, what will happen when multiple alerts fire simultaneously?

route:
  group_by: ['alertname']
  receiver: 'team-email'
receivers:
  - name: 'team-email'
    email_configs:
      - to: '[e-mail]'

medium

A. Alerts with the same 'alertname' will be grouped into one notification

B. Each alert will send a separate email regardless of grouping

C. No alerts will be sent because 'group_wait' is missing

D. Alerts will be sent only to Slack, not email

Solution

1. Step 1: Understand 'group_by' in Alertmanager route

   'group_by' groups alerts by specified labels; here, alerts with same 'alertname' are grouped.

2. Step 2: Check receiver and notification method

   Receiver 'team-email' uses email_configs, so grouped alerts send one email per alertname.

3. Final Answer:

   Alerts with the same 'alertname' will be grouped into one notification -> Option A

4. Quick Check:

   'group_by' controls alert grouping [OK]

Hint: 'group_by' label controls alert grouping in notifications [OK]

Common Mistakes:

• Assuming each alert sends separate email
• Thinking 'group_wait' is required to send alerts
• Confusing receiver type with Slack

4. You configured Alertmanager but no notifications are sent. Which of these is a likely cause based on this snippet?

receivers:
  - name: 'team-email'
    email_configs:
      - to: '[e-mail]'
route:
  receiver: 'team-email'
  group_by: ['alertname']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 1h

medium

A. Alertmanager does not support email notifications

B. Incorrect 'group_by' label causes no alerts

C. Receiver name does not match route receiver

D. Missing SMTP server configuration in Alertmanager

Solution

1. Step 1: Check email notification requirements

   Email notifications require SMTP server settings in Alertmanager config, not shown here.

2. Step 2: Verify receiver and route match

   Receiver name 'team-email' matches route receiver, so routing is correct.

3. Final Answer:

   Missing SMTP server configuration in Alertmanager -> Option D

4. Quick Check:

   Email needs SMTP setup to send alerts [OK]

Hint: Email alerts need SMTP server configured in Alertmanager [OK]

Common Mistakes:

• Assuming 'group_by' label stops alerts
• Thinking receiver name mismatch causes no alerts here
• Believing Alertmanager can't send emails

5. You want to avoid alert spam by grouping alerts by both 'alertname' and 'severity', and send notifications to Slack channel '#alerts'. Which Alertmanager route and receiver config is correct?

hard

A. route:
group_by: ['severity']
receiver: 'email-team'
receivers:
- name: 'email-team'
slack_configs:
- channel: '#alerts'

B. route:
group_by: ['alertname']
receiver: 'slack-notifications'
receivers:
- name: 'slack-notifications'
email_configs:
- to: '#alerts'

C. route:
group_by: ['alertname', 'severity']
receiver: 'slack-notifications'
receivers:
- name: 'slack-notifications'
slack_configs:
- channel: '#alerts'
send_resolved: true

D. route:
group_by: ['alertname', 'severity']
receiver: 'email-team'
receivers:
- name: 'email-team'
email_configs:
- to: '[e-mail]'

Solution

1. Step 1: Set grouping labels in route

   To group alerts by 'alertname' and 'severity', list both in 'group_by'.

2. Step 2: Configure Slack receiver correctly

   Receiver named 'slack-notifications' uses 'slack_configs' with channel '#alerts' and 'send_resolved' true.

3. Final Answer:

   Route groups by alertname and severity; receiver sends Slack messages to #alerts -> Option C

4. Quick Check:

   Group by multiple labels and use correct receiver config [OK]

Hint: Group by multiple labels and match receiver type to notification [OK]

Common Mistakes:

• Using email_configs for Slack notifications
• Grouping by only one label when two needed
• Mismatch between route receiver and receiver name