Alerting with Prometheus Alertmanager in Kubernetes - Practice Problems & Coding Challenges

Challenge - 5 Problems

💻 Command Output

intermediate

Prometheus Alertmanager Configuration Reload

You updated the Alertmanager configuration file alertmanager.yml in your Kubernetes cluster. Which command will correctly reload the Alertmanager configuration without restarting the pod?

A kubectl exec -n monitoring alertmanager-0 -- kill -SIGTERM 1

B kubectl rollout restart deployment alertmanager -n monitoring

C kubectl exec -n monitoring alertmanager-0 -- kill -HUP 1

D kubectl delete pod -n monitoring alertmanager-0

💡 Hint

Sending the HUP signal to the Alertmanager process triggers a config reload.

Alertmanager listens for the SIGHUP signal (kill -HUP) to reload its configuration without restarting. Restarting the pod or deployment is unnecessary and causes downtime.

🧠 Conceptual

intermediate

Understanding Alertmanager Routing Tree

In Alertmanager, what is the purpose of the route section in the configuration file?

A To define how alerts are grouped and sent to different receivers based on labels

B To specify the storage backend for alert data

C To configure Prometheus scrape intervals

D To set the CPU and memory limits for Alertmanager pods

💡 Hint

Think about how alerts get matched and delivered.

The route section defines rules for grouping alerts and routing them to specific receivers like email or Slack based on alert labels.

❓ Troubleshoot

advanced

Alertmanager Not Sending Notifications

You notice that Alertmanager is not sending notifications to your Slack channel despite alerts firing. Which of the following is the most likely cause?

A The Slack webhook URL is missing or incorrect in the receiver configuration

B Prometheus scrape interval is too low

C Alertmanager pod is running with insufficient CPU resources

D The Kubernetes cluster has no internet access

💡 Hint

Check the receiver configuration for external service URLs.

If the Slack webhook URL is missing or wrong, Alertmanager cannot send notifications to Slack. Other options may cause issues but are less direct causes.

🔀 Workflow

advanced

Setting Up Alertmanager with Multiple Receivers

You want to send critical alerts to both email and PagerDuty, but non-critical alerts only to email. Which Alertmanager configuration snippet correctly implements this routing?

Kubernetes

route:
  receiver: 'email'
  routes:
  - match:
      severity: 'critical'
    receiver: 'pagerduty'
    continue: true
receivers:
  - name: 'email'
    email_configs:
    - to: '[e-mail]'
  - name: 'pagerduty'
    pagerduty_configs:
    - service_key: 'your-service-key'

A The snippet sends non-critical alerts only to PagerDuty

B The snippet sends only critical alerts to PagerDuty and ignores email

C The snippet sends all alerts only to PagerDuty

D The snippet sends all alerts to email and critical alerts also to PagerDuty

💡 Hint

The top-level receiver is the default for unmatched alerts.

The top-level receiver: 'email' handles all alerts by default. The nested route matches critical alerts and sends them to PagerDuty as well.

✅ Best Practice

expert

Best Practice for Alertmanager High Availability

Which approach is best to ensure Alertmanager remains available and does not lose alerts during pod restarts or failures in a Kubernetes environment?

A Use a ConfigMap for Alertmanager config and restart pods on every config change without replicas

B Run multiple Alertmanager replicas with a shared persistent volume for configuration and use clustering

C Run a single Alertmanager pod with frequent restarts to refresh configuration

D Disable Alertmanager clustering and rely on Prometheus to resend alerts

💡 Hint

Think about avoiding single points of failure and data loss.

Running multiple Alertmanager replicas in a cluster with shared persistent storage ensures high availability and state synchronization, preventing alert loss.