DAG performance tracking in Apache Airflow - Mini Project: Build & Apply

DAG Performance Tracking in Airflow

📖 Scenario: You are managing workflows using Apache Airflow. You want to track the performance of your DAG runs by recording the duration of each run.This helps you understand how long your workflows take and identify any delays.

🎯 Goal: Build a simple Airflow DAG that tracks the duration of each DAG run and stores this information in a Python dictionary.You will create the data structure, add a configuration variable for minimum duration threshold, calculate durations, and finally print the durations.

📋 What You'll Learn

Create a dictionary to store DAG run durations

Add a minimum duration threshold variable

Calculate the duration of each DAG run using start and end times

Print the dictionary with DAG run durations

💡 Why This Matters

🌍 Real World

Tracking DAG run durations helps teams monitor workflow performance and detect delays or failures early.

💼 Career

Understanding how to track and analyze DAG performance is important for DevOps engineers and data engineers working with workflow automation.

1

Create a dictionary to store DAG run durations

Create a dictionary called dag_run_durations with these exact entries: 'run_1': 0, 'run_2': 0, 'run_3': 0.

Apache Airflow

# Create a dictionary called dag_run_durations with keys 'run_1', 'run_2', 'run_3' and values 0
# Your code here

Hint

Use curly braces to create a dictionary with keys and values.

2

Add a minimum duration threshold variable

Add a variable called min_duration_threshold and set it to 5 (minutes).

Apache Airflow

dag_run_durations = {'run_1': 0, 'run_2': 0, 'run_3': 0}
# Add a variable min_duration_threshold set to 5
# Your code here

Hint

Just assign the number 5 to the variable min_duration_threshold.

3

Calculate the duration of each DAG run

Given these start and end times in minutes for each run: start_times = {'run_1': 0, 'run_2': 10, 'run_3': 20} and end_times = {'run_1': 7, 'run_2': 15, 'run_3': 30}, use a for loop with variables run and start to iterate over start_times.items(). Calculate the duration as end_times[run] - start and update dag_run_durations[run] with this value.

Apache Airflow

dag_run_durations = {'run_1': 0, 'run_2': 0, 'run_3': 0}
min_duration_threshold = 5
start_times = {'run_1': 0, 'run_2': 10, 'run_3': 20}
end_times = {'run_1': 7, 'run_2': 15, 'run_3': 30}
# Use a for loop with run and start to calculate durations and update dag_run_durations
# Your code here

Hint

Use a for loop to go through start_times and calculate duration using end_times.

4

Print the dictionary with DAG run durations

Write a print statement to display the dag_run_durations dictionary.

Apache Airflow

dag_run_durations = {'run_1': 0, 'run_2': 0, 'run_3': 0}
min_duration_threshold = 5
start_times = {'run_1': 0, 'run_2': 10, 'run_3': 20}
end_times = {'run_1': 7, 'run_2': 15, 'run_3': 30}
for run, start in start_times.items():
    duration = end_times[run] - start
    dag_run_durations[run] = duration
# Print the dag_run_durations dictionary
# Your code here

Hint

Use print(dag_run_durations) to show the durations.