Airflow metrics with Prometheus - Mini Project: Build & Apply

Airflow Metrics with Prometheus

📖 Scenario: You are a DevOps engineer setting up monitoring for Apache Airflow using Prometheus. Monitoring helps you see how your Airflow workflows perform and catch problems early.In this project, you will create a simple Airflow DAG, configure Prometheus metrics collection, and verify the metrics output.

🎯 Goal: Build a basic Airflow DAG that exposes Prometheus metrics. Configure Airflow to enable Prometheus metrics export. Finally, check the metrics output to confirm it works.

📋 What You'll Learn

Create a simple Airflow DAG with one task

Enable Prometheus metrics in Airflow configuration

Add a Prometheus metrics exporter to Airflow

Verify metrics output by printing metrics endpoint response

💡 Why This Matters

🌍 Real World

Monitoring Airflow workflows helps detect failures and performance issues early. Prometheus is a popular tool to collect and visualize such metrics.

💼 Career

DevOps engineers and data engineers often set up monitoring for workflow tools like Airflow to ensure reliability and optimize resource use.

1

Create a simple Airflow DAG

Create a Python file called example_dag.py with a DAG named example_prometheus_dag. Add one task called print_hello that uses PythonOperator to print "Hello Airflow".

Apache Airflow

# Your code here to create the DAG and task

Hint

Use DAG and PythonOperator from Airflow. Define a function that prints "Hello Airflow" and assign it to the operator.

2

Enable Prometheus metrics in Airflow config

Add the following line to your Airflow configuration file airflow.cfg under the [metrics] section: statsd_on = True. Also set statsd_host = localhost and statsd_port = 8125.

Apache Airflow

from airflow import DAG
from airflow.operators.python import PythonOperator
from datetime import datetime

# Add metrics config to airflow.cfg
# Your code here

Hint

Edit airflow.cfg file manually or via your editor. Add the lines exactly under the [metrics] section.

3

Add Prometheus metrics exporter to Airflow

In your Airflow DAG file, import PrometheusMetrics from airflow_prometheus_exporter. Then add PrometheusMetrics().start_http_server(port=9090) at the top level to start the metrics server.

Apache Airflow

from airflow_prometheus_exporter import PrometheusMetrics

# Start Prometheus metrics server on port 9090
# Your code here

Hint

Import PrometheusMetrics and call start_http_server with port 9090 before defining the DAG.

4

Verify Prometheus metrics output

Run a command to fetch metrics from http://localhost:9090/metrics and print the response. Use Python's requests library to get the metrics and print the text.

Apache Airflow

import requests

# Fetch and print Prometheus metrics from localhost:9090
# Your code here

Hint

Use requests.get to fetch metrics and print the first 200 characters to confirm output.