Airflow metrics with Prometheus - Deep Dive

Overview - Airflow metrics with Prometheus

What is it?

Airflow metrics with Prometheus means collecting and monitoring data about Airflow's performance and behavior using Prometheus, a popular monitoring tool. Airflow is a system that helps schedule and run tasks automatically. Prometheus gathers numbers like how many tasks are running, how long they take, and if any fail. This helps teams keep Airflow healthy and fix problems quickly.

Why it matters

Without monitoring Airflow, teams might not notice when tasks fail or slow down, causing delays or errors in important workflows. Prometheus metrics give clear, real-time insight into Airflow’s state, helping prevent downtime and improve reliability. This saves time, reduces stress, and keeps business processes running smoothly.

Where it fits

Before learning this, you should understand basic Airflow concepts like DAGs and tasks, and know what monitoring means. After this, you can learn how to create alerts from metrics or visualize them with tools like Grafana for better decision-making.

Mental Model

Core Idea

Airflow metrics with Prometheus is about turning Airflow’s internal events into numbers that Prometheus can collect and track over time to show how well Airflow is working.

Think of it like...

It’s like having a fitness tracker on your body that counts your steps, heart rate, and sleep quality so you can see how healthy you are and spot problems early.

┌─────────────┐      ┌───────────────┐      ┌───────────────┐
│  Airflow    │─────▶│  Metrics      │─────▶│  Prometheus   │
│  Scheduler  │      │  Exporter     │      │  Server       │
└─────────────┘      └───────────────┘      └───────────────┘
       │                    │                      │
       ▼                    ▼                      ▼
  Tasks run,          Metrics exposed        Metrics scraped
  statuses change     as numbers             and stored

Build-Up - 7 Steps

1

Foundation

Understanding Airflow Basics

Concept: Learn what Airflow is and how it schedules and runs tasks.

Airflow is a tool that lets you define workflows as code. These workflows are called DAGs (Directed Acyclic Graphs). Each DAG has tasks that run in order. Airflow manages when and how these tasks run, retrying if they fail.

Result

You know how Airflow organizes and runs tasks automatically.

Understanding Airflow’s core helps you see why monitoring its tasks and scheduler is important.

2

Foundation

What is Prometheus Monitoring?

Concept: Learn what Prometheus does and how it collects metrics.

Prometheus is a system that collects numbers called metrics from software. It asks software for these numbers regularly (called scraping). It stores these numbers over time so you can see trends or spot problems.

Result

You understand how Prometheus gathers and stores monitoring data.

Knowing Prometheus basics prepares you to connect it with Airflow metrics.

3

Intermediate

Airflow Metrics Exporter Role

🤔Before reading on: do you think Airflow sends metrics directly to Prometheus, or does it need a middle step? Commit to your answer.

Concept: Airflow does not send metrics directly; it exposes them via an exporter that Prometheus scrapes.

Airflow has a built-in web server that can expose metrics in a format Prometheus understands. This is called the metrics exporter. Prometheus regularly requests these metrics from Airflow’s exporter endpoint.

Result

You know that Airflow exposes metrics via an HTTP endpoint for Prometheus to collect.

Understanding the exporter role clarifies how Airflow and Prometheus communicate without direct pushing.

4

Intermediate

Key Airflow Metrics to Monitor

🤔Before reading on: which do you think is more important to monitor—task success rates or scheduler uptime? Commit to your answer.

Concept: Learn which Airflow metrics give the best insight into system health and task performance.

Important metrics include: task instance states (success, failure, running), DAG run durations, scheduler heartbeat (to check if scheduler is alive), and queue sizes. These metrics help detect failures, delays, or bottlenecks.

Result

You can identify which metrics to watch to keep Airflow healthy.

Knowing key metrics focuses your monitoring efforts on what really matters.

5

Intermediate

Configuring Prometheus to Scrape Airflow

Concept: Learn how to tell Prometheus where to find Airflow metrics.

In Prometheus’s configuration file, you add a job that points to Airflow’s metrics endpoint URL. Prometheus then scrapes this URL at regular intervals to collect metrics data.

Result

Prometheus starts collecting Airflow metrics automatically.

Configuring scraping correctly ensures you get fresh and accurate data.

6

Advanced

Visualizing Airflow Metrics with Grafana

🤔Before reading on: do you think raw metrics are easy to understand, or do they need visualization? Commit to your answer.

Concept: Learn how to use Grafana to create dashboards that show Airflow metrics clearly.

Grafana connects to Prometheus and lets you build charts and graphs from metrics. You can create dashboards showing task success rates over time, scheduler health, and task durations to quickly spot issues.

Result

You can see Airflow’s health visually and detect problems faster.

Visualization turns raw numbers into actionable insights for teams.

7

Expert

Custom Metrics and Alerting Strategies

🤔Before reading on: do you think default Airflow metrics cover all monitoring needs, or might custom metrics be needed? Commit to your answer.

Concept: Learn how to add custom metrics and set alerts to catch specific Airflow issues early.

You can extend Airflow by adding custom Prometheus metrics in your DAGs or plugins. For example, count specific task failures or data quality checks. Then, configure Prometheus alert rules to notify you when metrics cross thresholds, like too many failures or scheduler downtime.

Result

You have a tailored monitoring setup that fits your Airflow workflows and alerts you proactively.

Custom metrics and alerts let you catch unique problems before they impact users.

Under the Hood

Airflow’s webserver exposes an HTTP endpoint that outputs metrics in Prometheus’s text-based format. Prometheus periodically sends HTTP requests to this endpoint, scraping the current metrics. These metrics include counters, gauges, and histograms representing Airflow’s internal states and events. Prometheus stores these time-series data efficiently for querying and alerting.

Why designed this way?

This pull-based model (Prometheus scraping metrics) avoids the complexity of Airflow pushing data, which can be unreliable or require extra infrastructure. It also standardizes metrics exposure, making it easy to integrate with many tools. The text format is simple and human-readable, easing debugging and extension.

┌───────────────┐       scrape        ┌───────────────┐
│  Prometheus   │◀───────────────────│ Airflow       │
│  Server       │                    │ Metrics       │
│               │                    │ Exporter      │
└───────────────┘                    └───────────────┘
       │                                    ▲
       │                                    │
       │                                    │
       ▼                                    │
┌───────────────┐                    ┌───────────────┐
│  Alerting &   │                    │ Airflow Tasks │
│  Visualization│                    │ Scheduler     │
│  (Grafana)    │                    └───────────────┘
└───────────────┘

Myth Busters - 4 Common Misconceptions

Quick: Does Airflow push metrics to Prometheus automatically? Commit yes or no.

Common Belief: Airflow automatically sends metrics to Prometheus without extra setup.

Reality: Airflow exposes metrics via an HTTP endpoint that Prometheus must scrape; Airflow does not push metrics.

Why it matters: Assuming push leads to missing metrics because Prometheus is not configured to scrape, causing blind spots in monitoring.

Quick: Are all Airflow metrics equally important to monitor? Commit yes or no.

Common Belief: All Airflow metrics are equally important and should be monitored the same way.

Reality: Some metrics like task failures and scheduler heartbeat are critical, while others are less urgent and can be monitored less frequently.

Why it matters: Monitoring everything equally wastes resources and can overwhelm teams with noise, hiding real issues.

Quick: Can you rely solely on Prometheus metrics for Airflow debugging? Commit yes or no.

Common Belief: Prometheus metrics alone are enough to debug all Airflow problems.

Reality: Metrics provide clues but logs and Airflow UI are also needed for full debugging and root cause analysis.

Why it matters: Relying only on metrics can delay problem resolution because detailed error information is missing.

Quick: Does adding many custom metrics always improve monitoring? Commit yes or no.

Common Belief: Adding many custom metrics always improves monitoring quality.

Reality: Too many custom metrics can clutter dashboards and increase system load, making monitoring harder.

Why it matters: Overloading metrics leads to confusion and performance issues, reducing monitoring effectiveness.

Expert Zone

1

Airflow’s metrics exporter uses Prometheus client libraries that support metric types like counters, gauges, and histograms, each suited for different data patterns.

2

Scheduler heartbeat metric is a subtle but critical indicator of Airflow’s health, often overlooked until failures occur.

3

Custom metrics must be carefully designed to avoid high cardinality (too many unique labels), which can degrade Prometheus performance.

When NOT to use

If your Airflow deployment is very small or short-lived, full Prometheus monitoring might be overkill. In such cases, simple logging or Airflow’s built-in UI may suffice. For extremely high-scale environments, consider specialized monitoring solutions or managed services that handle metrics at scale.

Production Patterns

In production, teams often combine Prometheus metrics with Grafana dashboards and alerting rules to monitor Airflow’s task success rates, scheduler health, and queue sizes. They also add custom metrics for business-specific checks and integrate alerts with communication tools like Slack or PagerDuty for fast incident response.

Connections

Observability

Airflow metrics with Prometheus is a key part of observability, which includes metrics, logs, and traces.

Understanding metrics collection helps grasp how observability provides a full picture of system health.

Time-Series Databases

Prometheus stores Airflow metrics as time-series data, a specialized database type.

Knowing how time-series databases work explains why Prometheus is efficient for monitoring changing data over time.

Human Physiology Monitoring

Like monitoring Airflow with Prometheus, fitness trackers monitor human health metrics continuously.

This cross-domain link shows how continuous metric collection helps detect problems early in both machines and humans.

Common Pitfalls

#1 Not enabling the metrics endpoint in Airflow configuration.

Wrong approach:

[metrics]
enabled = false

# or missing metrics config entirely

Correct approach:

[metrics]
enabled = true

# This enables the Prometheus metrics endpoint

Root cause:

Learners assume metrics are available by default without enabling the feature.

#2 Configuring Prometheus to scrape the wrong Airflow URL or port.

Wrong approach:

scrape_configs:
- job_name: 'airflow'
static_configs:
- targets: ['localhost:8080'] # Wrong port or missing metrics path

Correct approach:

scrape_configs:
- job_name: 'airflow'
static_configs:
- targets: ['localhost:8793'] # Correct metrics port
metrics_path: '/metrics'

Root cause:

Misunderstanding Airflow’s metrics endpoint location and port.

#3 Adding too many labels to custom metrics causing high cardinality.

Wrong approach:

from prometheus_client import Counter

custom_metric = Counter('task_events', 'Count of task events', ['task_id', 'user_id', 'run_id', 'extra_label'])

Correct approach:

from prometheus_client import Counter

custom_metric = Counter('task_events', 'Count of task events', ['task_id'])

Root cause:

Not realizing that many unique label values increase storage and query complexity.

Key Takeaways

Airflow exposes important performance and health data as metrics that Prometheus can collect by scraping an HTTP endpoint.

Prometheus uses a pull model to gather metrics regularly, which is reliable and simple to integrate with Airflow.

Monitoring key metrics like task states and scheduler heartbeat helps detect problems early and keep workflows running smoothly.

Visualizing metrics with tools like Grafana turns raw data into clear insights for faster troubleshooting.

Custom metrics and alerting enhance monitoring but must be designed carefully to avoid overload and performance issues.