How to Test DAG in Airflow: Simple Steps and Examples

To test a DAG in Airflow, you can use the airflow dags test command to run a specific task instance without scheduling. Additionally, writing unit tests with Python's unittest or pytest frameworks allows you to validate DAG structure and task logic programmatically.

Syntax

The main command to test a DAG task manually is airflow dags test <dag_id> <task_id> <execution_date>.

• dag_id: The identifier of your DAG.
• task_id: The specific task within the DAG to run.
• execution_date: The logical date for the DAG run, formatted as YYYY-MM-DD or YYYY-MM-DDTHH:MM:SS.

This command runs the task immediately without affecting the scheduler or database state.

bash

airflow dags test example_dag task1 2024-06-01

Example

This example shows how to write a simple unit test for an Airflow DAG using pytest. It checks that the DAG loads correctly and contains the expected tasks.

python

import pytest
from airflow.models import DagBag

def test_dag_loads():
    dagbag = DagBag(dag_folder="/path/to/dags", include_examples=False)
    dag = dagbag.get_dag("example_dag")
    assert dag is not None, "DAG did not load"
    assert "task1" in dag.task_ids, "task1 not found in DAG"
    assert "task2" in dag.task_ids, "task2 not found in DAG"

Output

============================= test session starts =============================
collected 1 item

test_dag.py .                                                             [100%]

============================== 1 passed in 0.05s ==============================

Common Pitfalls

Common mistakes when testing Airflow DAGs include:

• Using airflow dags test without specifying a valid execution_date, causing errors.
• Not isolating DAG imports in unit tests, which can cause side effects or slow tests.
• Assuming airflow dags test updates the scheduler or database state; it only runs the task once.
• Not mocking external dependencies in task code, leading to flaky tests.

bash

## Wrong way: Running dags test without execution date
# airflow dags test example_dag task1

## Right way: Include execution date
# airflow dags test example_dag task1 2024-06-01

Quick Reference

Tips for testing Airflow DAGs:

• Use airflow dags test for quick manual task runs.
• Write unit tests to check DAG structure and task dependencies.
• Mock external systems in tests to avoid side effects.
• Run tests in isolated environments to prevent interference.

Key Takeaways

Use airflow dags test with dag_id, task_id, and execution_date to run tasks manually.

Write unit tests with Python frameworks to validate DAG loading and task presence.

Always specify a valid execution date when testing DAGs from the command line.

Mock external dependencies in task code to ensure reliable tests.

Manual tests do not affect scheduler state; use unit tests for full validation.