DAG versioning strategies in Apache Airflow - Step-by-Step Execution

Process Flow - DAG versioning strategies

Create DAG code

↓

Save DAG file in repo

↓

Commit changes with version tag

↓

Deploy DAG to Airflow environment

↓

Airflow scheduler loads DAG

↓

Run DAG with specific version

↓

Monitor and maintain versions

↓

Update DAG code for new version

↩Back to Save DAG file

This flow shows how DAG code is created, versioned, deployed, and run in Airflow, with updates leading to new versions.

Execution Sample

Apache Airflow

from airflow import DAG
from airflow.operators.dummy import DummyOperator
from datetime import datetime

def create_dag(version):
    dag = DAG(
        dag_id=f"example_dag_v{version}",
        start_date=datetime(2023, 1, 1),
        schedule_interval='@daily'
    )
    start = DummyOperator(task_id='start', dag=dag)
    return dag

dag_v1 = create_dag(1)
dag_v2 = create_dag(2)

Defines two DAG versions with different version numbers in their IDs.

Process Table

Step	Action	DAG ID Created	Version Used	Result
1	Call create_dag(1)	example_dag_v1	1	DAG object for version 1 created
2	Call create_dag(2)	example_dag_v2	2	DAG object for version 2 created
3	Deploy DAGs to Airflow	example_dag_v1, example_dag_v2	1 and 2	Both DAG versions available in Airflow
4	Run DAG example_dag_v1	example_dag_v1	1	Runs tasks of version 1 DAG
5	Run DAG example_dag_v2	example_dag_v2	2	Runs tasks of version 2 DAG
6	Update DAG code to version 3	example_dag_v3	3	New DAG version created and deployed
7	Run DAG example_dag_v3	example_dag_v3	3	Runs tasks of version 3 DAG

💡 Execution stops after running the latest DAG version (3) and all versions are deployed.

Status Tracker

Variable	Start	After Step 1	After Step 2	After Step 6	Final
dag	None	example_dag_v1	example_dag_v2	example_dag_v3	example_dag_v3
version	None	1	2	3	3

Key Moments - 3 Insights

Why do we create separate DAG IDs for each version instead of overwriting the same DAG?

Each DAG version has a unique ID (see execution_table steps 1 and 2) so Airflow can track and run multiple versions independently without conflicts.

How does Airflow know which DAG version to run?

Airflow runs the DAG by its unique ID (execution_table steps 4 and 5), so specifying the version in the DAG ID directs Airflow to the correct version.

What happens when we update the DAG code to a new version?

A new DAG object with a new version ID is created and deployed (step 6), allowing Airflow to run the updated workflow alongside older versions.

Visual Quiz - 3 Questions

Look at the execution_table, what is the DAG ID created at step 2?

A example_dag_v2

B example_dag_v3

C example_dag_v1

D example_dag

💡 Hint

Check the 'DAG ID Created' column at step 2 in the execution_table.

At which step does the new DAG version 3 get deployed?

A Step 4

B Step 5

C Step 6

D Step 7

💡 Hint

Look for the step where 'Update DAG code to version 3' happens in the execution_table.

If we did not use version numbers in DAG IDs, what would happen?

A Airflow would run all versions separately.

B Older DAG versions would be overwritten and lost.

C Airflow would create duplicate DAGs automatically.

D No effect; Airflow handles versions internally.

💡 Hint

Refer to key_moments about unique DAG IDs and versioning.

Concept Snapshot

DAG Versioning Strategies in Airflow:
- Use unique DAG IDs with version numbers (e.g., example_dag_v1)
- Create and deploy each version as a separate DAG file
- Airflow runs DAGs by their IDs, enabling multiple versions
- Update DAG code by creating new versioned DAGs
- Maintain old versions for history or rollback

Full Transcript

This visual execution shows how to manage DAG versions in Airflow by creating separate DAG IDs with version numbers. We start by defining DAGs with different version numbers, deploy them, and run each version independently. Updating the DAG means creating a new version with a new ID. This approach prevents overwriting and allows running multiple versions side by side. Key points include unique DAG IDs, deployment steps, and running specific versions. The quiz tests understanding of DAG IDs, deployment steps, and consequences of not versioning.