Airflow metrics with Prometheus - Commands & Configuration

Introduction

When running many workflows, it is hard to know how well Airflow is performing or if tasks are failing often. Prometheus helps by collecting detailed metrics from Airflow so you can watch its health and performance in real time.

When you want to monitor how many tasks are running or failing in Airflow over time

When you need to alert your team if Airflow scheduler or workers stop working

When you want to see how long tasks take to run and identify slow workflows

When you want to collect metrics from Airflow and visualize them in Grafana dashboards

When you want to track Airflow resource usage and task success rates automatically

Config File - airflow.cfg

airflow.cfg

[metrics]
# Enable StatsD metrics exporter
statsd_on = True
statsd_host = localhost
statsd_port = 8125
statsd_prefix = airflow

[metrics.prometheus]
enabled = True
port = 9090

[webserver]
# Expose Prometheus metrics endpoint
expose_prometheus = True

This configuration enables Airflow to send metrics to Prometheus.

statsd_on turns on metrics collection.

statsd_host and statsd_port tell Airflow where the StatsD server listens.

expose_prometheus in the webserver section allows Prometheus to scrape metrics directly from Airflow's webserver.

Commands

Starts the Airflow webserver which now exposes Prometheus metrics on port 8080.

Terminal

airflow webserver --port 8080

Expected Output

[2024-06-01 12:00:00,000] {webserver.py:123} INFO - Starting Airflow webserver on port 8080

[2024-06-01 12:00:01,000] {prometheus.py:45} INFO - Prometheus metrics endpoint enabled at /metrics

→

--port - Specifies the port where the Airflow webserver listens

Starts the Airflow scheduler which runs tasks and updates metrics that Prometheus will collect.

Terminal

airflow scheduler

Expected Output

[2024-06-01 12:00:05,000] {scheduler.py:78} INFO - Starting the scheduler

[2024-06-01 12:00:06,000] {scheduler.py:90} INFO - Scheduler heartbeat

Fetches the raw Prometheus metrics from Airflow's metrics endpoint to verify metrics are exposed.

Terminal

curl http://localhost:8080/metrics

Expected Output

# HELP airflow_task_duration_seconds Duration of Airflow tasks in seconds

# TYPE airflow_task_duration_seconds histogram

airflow_task_duration_seconds_bucket{task_id="example_task",le="5"} 3

airflow_task_duration_seconds_bucket{task_id="example_task",le="10"} 5

# HELP airflow_scheduler_heartbeat_seconds Time between scheduler heartbeats

# TYPE airflow_scheduler_heartbeat_seconds gauge

airflow_scheduler_heartbeat_seconds 1.0

Starts Prometheus server with configuration to scrape Airflow metrics from the webserver endpoint.

Terminal

prometheus --config.file=prometheus.yml

Expected Output

level=info ts=2024-06-01T12:00:10.000Z caller=main.go:123 msg="Starting Prometheus" version="2.45.0"

level=info ts=2024-06-01T12:00:10.001Z caller=main.go:456 msg="Loading configuration file" filename=prometheus.yml

level=info ts=2024-06-01T12:00:10.002Z caller=main.go:789 msg="Server is ready to receive web requests."

→

--config.file - Specifies the Prometheus configuration file to use

Key Concept

If you remember nothing else from this pattern, remember: enabling Prometheus metrics in Airflow lets you monitor task and scheduler health in real time.

Common Mistakes

Not enabling the Prometheus metrics endpoint in airflow.cfg

Prometheus cannot scrape metrics if Airflow does not expose them, so no data is collected.

Set 'expose_prometheus = True' under the [webserver] section in airflow.cfg.

Starting Prometheus without configuring it to scrape Airflow metrics

Prometheus will run but will not collect any Airflow metrics, making monitoring useless.

Add Airflow's metrics endpoint to prometheus.yml scrape_configs.

Running Airflow webserver on a different port than Prometheus expects

Prometheus will fail to connect and collect metrics, causing missing data.

Ensure the webserver port matches the one configured in Prometheus scrape target.

Summary

Configure airflow.cfg to enable Prometheus metrics and expose the metrics endpoint.

Start Airflow webserver and scheduler to generate and expose metrics.

Verify metrics are available by fetching the /metrics endpoint.

Run Prometheus with a config that scrapes Airflow metrics for monitoring.

Practice

1. What is the main purpose of integrating Prometheus with Airflow metrics?

easy

A. To schedule Airflow DAGs automatically

B. To store Airflow logs permanently

C. To monitor Airflow task and system health in real-time

D. To send emails from Airflow tasks

Solution

1. Step 1: Understand Airflow metrics role

   Airflow metrics provide data about task execution and system performance.

2. Step 2: Identify Prometheus integration purpose

   Prometheus collects and visualizes these metrics for monitoring and alerting.

3. Final Answer:

   To monitor Airflow task and system health in real-time -> Option C

4. Quick Check:

   Monitoring = B [OK]

Hint: Prometheus is for monitoring, not scheduling or logging [OK]

Common Mistakes:

• Confusing monitoring with scheduling
• Thinking Prometheus stores logs
• Assuming Prometheus sends emails

2. Which configuration option must be enabled in Airflow to expose metrics for Prometheus scraping?

easy

A. metrics_enabled = False

B. statsd_on = True

C. prometheus_exporter = False

D. logging_level = DEBUG

Solution

1. Step 1: Identify Airflow metric exposure setting

   Airflow uses StatsD to collect metrics, so enabling StatsD is required.

2. Step 2: Match correct config option

   The correct option to enable StatsD is statsd_on = True.

3. Final Answer:

   statsd_on = True -> Option B

4. Quick Check:

   Enable StatsD = A [OK]

Hint: Enable StatsD to expose metrics for Prometheus [OK]

Common Mistakes:

• Disabling metrics with false flags
• Confusing logging level with metrics
• Assuming Prometheus exporter is off by default

3. Given this Airflow config snippet:

[metrics]
statsd_on = True
statsd_host = "localhost"
statsd_port = 8125

What will Prometheus scrape from Airflow?

medium

A. Airflow task execution metrics via StatsD exporter

B. Airflow DAG definitions

C. Airflow database contents

D. Airflow log files

Solution

1. Step 1: Analyze config snippet

   The snippet enables StatsD metrics collection on localhost port 8125.

2. Step 2: Understand Prometheus scraping

   Prometheus scrapes metrics exposed by StatsD exporter, which includes task execution data.

3. Final Answer:

   Airflow task execution metrics via StatsD exporter -> Option A

4. Quick Check:

   StatsD metrics = D [OK]

Hint: StatsD config means Prometheus scrapes metrics, not logs or DB [OK]

Common Mistakes:

• Thinking Prometheus scrapes DAG code
• Assuming Prometheus reads database directly
• Confusing logs with metrics

4. You enabled StatsD in Airflow but Prometheus shows no metrics. What is a likely cause?

medium

A. Prometheus scrape config missing or incorrect

B. Airflow DAGs are paused

C. Airflow scheduler is disabled

D. Airflow logs are too large

Solution

1. Step 1: Check Prometheus scraping setup

   If Prometheus shows no metrics, it may not be scraping the StatsD exporter endpoint correctly.

2. Step 2: Rule out unrelated causes

   Paused DAGs or scheduler off do not stop metrics exposure; large logs do not affect metrics scraping.

3. Final Answer:

   Prometheus scrape config missing or incorrect -> Option A

4. Quick Check:

   Scrape config issue = C [OK]

Hint: Check Prometheus scrape config if no metrics appear [OK]

Common Mistakes:

• Blaming paused DAGs for missing metrics
• Assuming scheduler off stops metrics
• Thinking logs size affects metrics

5. You want to create an alert in Prometheus for Airflow tasks that fail more than 5 times in 10 minutes. Which Prometheus query is correct?

hard

A. count(airflow_task_failures_total) == 5

B. sum(airflow_task_success_total) > 5

C. rate(airflow_task_failures_total[5m]) < 5

D. increase(airflow_task_failures_total[10m]) > 5

Solution

1. Step 1: Understand alert condition

   The alert triggers if failures increase by more than 5 in 10 minutes.

2. Step 2: Match Prometheus query

   increase(metric[10m]) > 5 checks if failures increased over 5 in last 10 minutes.

3. Step 3: Eliminate wrong options

   sum(airflow_task_success_total) > 5 checks successes, not failures; C uses wrong time and comparison; D counts current samples, not increase.

4. Final Answer:

   increase(airflow_task_failures_total[10m]) > 5 -> Option D

5. Quick Check:

   Use increase() over 10m > 5 for failure alert [OK]

Hint: Use increase() over time window to detect failure spikes [OK]

Common Mistakes:

• Using success metric instead of failure
• Using rate() incorrectly for count threshold
• Counting samples instead of increase