Airflow metrics with Prometheus - Step-by-Step Execution

Process Flow - Airflow metrics with Prometheus

Airflow Scheduler runs tasks

↓

Airflow exposes metrics endpoint

↓

Prometheus scrapes metrics endpoint

↓

Prometheus stores metrics data

↓

User queries metrics or visualizes in Grafana

Airflow runs tasks and exposes metrics on an HTTP endpoint. Prometheus regularly collects these metrics to store and visualize them.

Execution Sample

Apache Airflow

from prometheus_client import start_http_server, Gauge

metrics_exporter = Gauge('airflow_task_status', 'Status of Airflow tasks')
start_http_server(8080)
# Airflow runs and updates metrics
# Prometheus scrapes http://localhost:8080/metrics

Starts a Prometheus metrics server in Airflow and shows how Prometheus scrapes metrics.

Process Table

Step	Action	Airflow State	Prometheus Action	Result
1	Airflow Scheduler starts task	Task running	No scrape yet	Metrics updated internally
2	Prometheus scrapes /metrics	Task running	Scrape metrics endpoint	Metrics data collected
3	Task completes	Task succeeded	No scrape	Metrics reflect task success
4	Prometheus scrapes /metrics again	Task succeeded	Scrape metrics endpoint	Updated metrics collected
5	User queries Prometheus	N/A	Query metrics data	Task success metrics shown
6	User visualizes in Grafana	N/A	Use Prometheus data	Dashboard shows Airflow metrics

💡 Process continues as Airflow runs tasks and Prometheus scrapes metrics periodically

Status Tracker

Variable	Start	After Step 1	After Step 3	After Step 4	Final
Airflow Task State	None	Running	Succeeded	Succeeded	Succeeded
Prometheus Metrics Data	Empty	Empty	Old metrics	Updated metrics	Updated metrics

Key Moments - 3 Insights

Why doesn't Prometheus collect metrics before scraping?

Prometheus only collects metrics when it scrapes the Airflow metrics endpoint, so before scraping, no data is collected (see execution_table step 1).

How does Airflow update metrics when tasks change state?

Airflow updates internal metrics as task states change, but these updates are only visible to Prometheus after the next scrape (see steps 3 and 4).

Why do we need a metrics endpoint in Airflow?

The metrics endpoint exposes Airflow's internal metrics in a format Prometheus understands, enabling Prometheus to scrape and store them (see concept_flow).

Visual Quiz - 3 Questions

Look at the execution_table, what is the Airflow Task State after step 3?

A Failed

B Running

C Succeeded

D None

💡 Hint

Check the 'Airflow Task State' column after step 3 in variable_tracker

At which step does Prometheus first collect updated metrics reflecting task completion?

A Step 2

B Step 4

C Step 3

D Step 5

💡 Hint

Look at execution_table rows where Prometheus scrapes metrics after task completion

If Airflow did not expose a metrics endpoint, what would happen?

A Prometheus would fail to collect metrics

B Airflow tasks would not run

C Prometheus would scrape metrics normally

D Metrics would be collected automatically

💡 Hint

Refer to key_moments about the importance of the metrics endpoint

Concept Snapshot

Airflow exposes task metrics via an HTTP endpoint.
Prometheus scrapes this endpoint regularly to collect metrics.
Metrics reflect task states like running or succeeded.
Prometheus stores and allows querying these metrics.
Visualize metrics with tools like Grafana.
Without the endpoint, Prometheus cannot collect metrics.

Full Transcript

Airflow runs tasks and updates internal metrics about their states. It exposes these metrics on a special HTTP endpoint. Prometheus regularly scrapes this endpoint to collect the latest metrics data. When a task starts, Airflow updates metrics internally, but Prometheus only sees these updates after scraping. When the task completes, Airflow updates metrics again, and Prometheus collects the updated data on the next scrape. Users can query this data in Prometheus or visualize it in Grafana dashboards. The key is that Airflow must expose a metrics endpoint for Prometheus to collect metrics. Without it, Prometheus cannot gather any data. This flow repeats continuously as Airflow runs tasks and Prometheus scrapes metrics.