Unit testing DAGs in Apache Airflow - Mini Project: Build & Apply

Unit Testing Airflow DAGs

📖 Scenario: You work as a data engineer. You create workflows called DAGs in Apache Airflow to automate tasks. To keep your workflows reliable, you want to write simple tests that check if your DAGs are set up correctly.

🎯 Goal: Build a small unit test for an Airflow DAG. You will create a DAG, add a configuration variable, write a test function to check the DAG's tasks, and finally run the test to see the result.

📋 What You'll Learn

Create a DAG object with a specific DAG ID

Add a default argument dictionary with a start date

Write a test function that checks the DAG ID and task count

Print the test result output

💡 Why This Matters

🌍 Real World

Data engineers use Airflow DAGs to automate data workflows. Unit testing DAGs helps catch errors early and keeps pipelines reliable.

💼 Career

Knowing how to write unit tests for Airflow DAGs is a valuable skill for data engineers and DevOps professionals working with data pipelines.

1

Create a simple Airflow DAG

Import DAG from airflow.models and create a DAG object called test_dag with the DAG ID 'example_dag' and an empty default_args dictionary.

Apache Airflow

# Import DAG and create test_dag with dag_id 'example_dag' and empty default_args
# Your code here

Need a hint?

Use DAG(dag_id='example_dag', default_args=default_args) to create the DAG.

2

Add a start date to default_args

Add a start_date key to the default_args dictionary with the value datetime(2024, 1, 1). Import datetime from the datetime module.

Apache Airflow

from airflow.models import DAG
from datetime import datetime

default_args = {}
# Add start_date=datetime(2024, 1, 1) to default_args below
# Your code here

test_dag = DAG(dag_id='example_dag', default_args=default_args)

Need a hint?

Use default_args = {'start_date': datetime(2024, 1, 1)}.

3

Write a test function to check the DAG

Define a function called test_dag_structure that checks if test_dag.dag_id equals 'example_dag' and if the number of tasks in test_dag.tasks is zero. Return True if both conditions are met, otherwise False.

Apache Airflow

from airflow.models import DAG
from datetime import datetime

default_args = {'start_date': datetime(2024, 1, 1)}
test_dag = DAG(dag_id='example_dag', default_args=default_args)

# Define test_dag_structure function below
# Your code here

Need a hint?

Use def test_dag_structure(): and check conditions with if.

4

Run the test and print the result

Call the function test_dag_structure() and print its result using print().

Apache Airflow

from airflow.models import DAG
from datetime import datetime

default_args = {'start_date': datetime(2024, 1, 1)}
test_dag = DAG(dag_id='example_dag', default_args=default_args)

def test_dag_structure():
    if test_dag.dag_id == 'example_dag' and len(test_dag.tasks) == 0:
        return True
    else:
        return False

# Call test_dag_structure and print the result below
# Your code here

Need a hint?

Use print(test_dag_structure()) to show the test result.