Airflow metrics with Prometheus - Step-by-Step Execution

Process Flow - Airflow metrics with Prometheus

Airflow Scheduler runs tasks

↓

Airflow exposes metrics endpoint

↓

Prometheus scrapes metrics endpoint

↓

Prometheus stores metrics data

↓

User queries metrics or visualizes in Grafana

Airflow runs tasks and exposes metrics on an HTTP endpoint. Prometheus regularly collects these metrics to store and visualize them.

Execution Sample

Apache Airflow

from prometheus_client import start_http_server, Gauge

metrics_exporter = Gauge('airflow_task_status', 'Status of Airflow tasks')
start_http_server(8080)
# Airflow runs and updates metrics
# Prometheus scrapes http://localhost:8080/metrics

Starts a Prometheus metrics server in Airflow and shows how Prometheus scrapes metrics.

Process Table

Step	Action	Airflow State	Prometheus Action	Result
1	Airflow Scheduler starts task	Task running	No scrape yet	Metrics updated internally
2	Prometheus scrapes /metrics	Task running	Scrape metrics endpoint	Metrics data collected
3	Task completes	Task succeeded	No scrape	Metrics reflect task success
4	Prometheus scrapes /metrics again	Task succeeded	Scrape metrics endpoint	Updated metrics collected
5	User queries Prometheus	N/A	Query metrics data	Task success metrics shown
6	User visualizes in Grafana	N/A	Use Prometheus data	Dashboard shows Airflow metrics

💡 Process continues as Airflow runs tasks and Prometheus scrapes metrics periodically

Status Tracker

Variable	Start	After Step 1	After Step 3	After Step 4	Final
Airflow Task State	None	Running	Succeeded	Succeeded	Succeeded
Prometheus Metrics Data	Empty	Empty	Old metrics	Updated metrics	Updated metrics

Key Moments - 3 Insights

Why doesn't Prometheus collect metrics before scraping?

Prometheus only collects metrics when it scrapes the Airflow metrics endpoint, so before scraping, no data is collected (see execution_table step 1).

How does Airflow update metrics when tasks change state?

Airflow updates internal metrics as task states change, but these updates are only visible to Prometheus after the next scrape (see steps 3 and 4).

Why do we need a metrics endpoint in Airflow?

The metrics endpoint exposes Airflow's internal metrics in a format Prometheus understands, enabling Prometheus to scrape and store them (see concept_flow).

Visual Quiz - 3 Questions

Look at the execution_table, what is the Airflow Task State after step 3?

A Failed

B Running

C Succeeded

D None

💡 Hint

Check the 'Airflow Task State' column after step 3 in variable_tracker

At which step does Prometheus first collect updated metrics reflecting task completion?

A Step 2

B Step 4

C Step 3

D Step 5

💡 Hint

Look at execution_table rows where Prometheus scrapes metrics after task completion

If Airflow did not expose a metrics endpoint, what would happen?

A Prometheus would fail to collect metrics

B Airflow tasks would not run

C Prometheus would scrape metrics normally

D Metrics would be collected automatically

💡 Hint

Refer to key_moments about the importance of the metrics endpoint

Concept Snapshot

Airflow exposes task metrics via an HTTP endpoint.
Prometheus scrapes this endpoint regularly to collect metrics.
Metrics reflect task states like running or succeeded.
Prometheus stores and allows querying these metrics.
Visualize metrics with tools like Grafana.
Without the endpoint, Prometheus cannot collect metrics.

Full Transcript

Airflow runs tasks and updates internal metrics about their states. It exposes these metrics on a special HTTP endpoint. Prometheus regularly scrapes this endpoint to collect the latest metrics data. When a task starts, Airflow updates metrics internally, but Prometheus only sees these updates after scraping. When the task completes, Airflow updates metrics again, and Prometheus collects the updated data on the next scrape. Users can query this data in Prometheus or visualize it in Grafana dashboards. The key is that Airflow must expose a metrics endpoint for Prometheus to collect metrics. Without it, Prometheus cannot gather any data. This flow repeats continuously as Airflow runs tasks and Prometheus scrapes metrics.

Practice

1. What is the main purpose of integrating Prometheus with Airflow metrics?

easy

A. To schedule Airflow DAGs automatically

B. To store Airflow logs permanently

C. To monitor Airflow task and system health in real-time

D. To send emails from Airflow tasks

Solution

1. Step 1: Understand Airflow metrics role

   Airflow metrics provide data about task execution and system performance.

2. Step 2: Identify Prometheus integration purpose

   Prometheus collects and visualizes these metrics for monitoring and alerting.

3. Final Answer:

   To monitor Airflow task and system health in real-time -> Option C

4. Quick Check:

   Monitoring = B [OK]

Hint: Prometheus is for monitoring, not scheduling or logging [OK]

Common Mistakes:

• Confusing monitoring with scheduling
• Thinking Prometheus stores logs
• Assuming Prometheus sends emails

2. Which configuration option must be enabled in Airflow to expose metrics for Prometheus scraping?

easy

A. metrics_enabled = False

B. statsd_on = True

C. prometheus_exporter = False

D. logging_level = DEBUG

Solution

1. Step 1: Identify Airflow metric exposure setting

   Airflow uses StatsD to collect metrics, so enabling StatsD is required.

2. Step 2: Match correct config option

   The correct option to enable StatsD is statsd_on = True.

3. Final Answer:

   statsd_on = True -> Option B

4. Quick Check:

   Enable StatsD = A [OK]

Hint: Enable StatsD to expose metrics for Prometheus [OK]

Common Mistakes:

• Disabling metrics with false flags
• Confusing logging level with metrics
• Assuming Prometheus exporter is off by default

3. Given this Airflow config snippet:

[metrics]
statsd_on = True
statsd_host = "localhost"
statsd_port = 8125

What will Prometheus scrape from Airflow?

medium

A. Airflow task execution metrics via StatsD exporter

B. Airflow DAG definitions

C. Airflow database contents

D. Airflow log files

Solution

1. Step 1: Analyze config snippet

   The snippet enables StatsD metrics collection on localhost port 8125.

2. Step 2: Understand Prometheus scraping

   Prometheus scrapes metrics exposed by StatsD exporter, which includes task execution data.

3. Final Answer:

   Airflow task execution metrics via StatsD exporter -> Option A

4. Quick Check:

   StatsD metrics = D [OK]

Hint: StatsD config means Prometheus scrapes metrics, not logs or DB [OK]

Common Mistakes:

• Thinking Prometheus scrapes DAG code
• Assuming Prometheus reads database directly
• Confusing logs with metrics

4. You enabled StatsD in Airflow but Prometheus shows no metrics. What is a likely cause?

medium

A. Prometheus scrape config missing or incorrect

B. Airflow DAGs are paused

C. Airflow scheduler is disabled

D. Airflow logs are too large

Solution

1. Step 1: Check Prometheus scraping setup

   If Prometheus shows no metrics, it may not be scraping the StatsD exporter endpoint correctly.

2. Step 2: Rule out unrelated causes

   Paused DAGs or scheduler off do not stop metrics exposure; large logs do not affect metrics scraping.

3. Final Answer:

   Prometheus scrape config missing or incorrect -> Option A

4. Quick Check:

   Scrape config issue = C [OK]

Hint: Check Prometheus scrape config if no metrics appear [OK]

Common Mistakes:

• Blaming paused DAGs for missing metrics
• Assuming scheduler off stops metrics
• Thinking logs size affects metrics

5. You want to create an alert in Prometheus for Airflow tasks that fail more than 5 times in 10 minutes. Which Prometheus query is correct?

hard

A. count(airflow_task_failures_total) == 5

B. sum(airflow_task_success_total) > 5

C. rate(airflow_task_failures_total[5m]) < 5

D. increase(airflow_task_failures_total[10m]) > 5

Solution

1. Step 1: Understand alert condition

   The alert triggers if failures increase by more than 5 in 10 minutes.

2. Step 2: Match Prometheus query

   increase(metric[10m]) > 5 checks if failures increased over 5 in last 10 minutes.

3. Step 3: Eliminate wrong options

   sum(airflow_task_success_total) > 5 checks successes, not failures; C uses wrong time and comparison; D counts current samples, not increase.

4. Final Answer:

   increase(airflow_task_failures_total[10m]) > 5 -> Option D

5. Quick Check:

   Use increase() over 10m > 5 for failure alert [OK]

Hint: Use increase() over time window to detect failure spikes [OK]

Common Mistakes:

• Using success metric instead of failure
• Using rate() incorrectly for count threshold
• Counting samples instead of increase