DAG versioning strategies in Apache Airflow - Mini Project: Build & Apply

DAG Versioning Strategies in Apache Airflow

📖 Scenario: You work as a data engineer managing workflows in Apache Airflow. Your team wants to keep track of different versions of DAGs (Directed Acyclic Graphs) to safely update workflows without breaking existing runs.This project will guide you through creating a simple DAG with versioning information embedded, so you can see how to manage DAG versions clearly.

🎯 Goal: Build a basic Airflow DAG that includes a version number in its ID and a configuration variable for the version. Then, write code to print the DAG version when the DAG runs.

📋 What You'll Learn

Create a DAG dictionary with a fixed DAG ID and version number

Add a configuration variable to hold the DAG version

Write a function to print the DAG version during execution

Print the DAG version as the final output

💡 Why This Matters

🌍 Real World

In real Airflow projects, managing DAG versions helps teams deploy updates safely and track changes over time.

💼 Career

Understanding DAG versioning is important for data engineers and DevOps professionals who maintain reliable workflow pipelines.

1

Create the initial DAG dictionary

Create a dictionary called dag_info with these exact entries: 'dag_id': 'example_dag_v1' and 'version': '1.0'.

Apache Airflow

# Create the dag_info dictionary with dag_id and version
# Your code here

Hint

Use curly braces to create a dictionary with keys 'dag_id' and 'version'.

2

Add a configuration variable for DAG version

Create a variable called dag_version and set it equal to the 'version' value from the dag_info dictionary.

Apache Airflow

dag_info = {'dag_id': 'example_dag_v1', 'version': '1.0'}
# Create dag_version variable from dag_info['version']
# Your code here

Hint

Access the dictionary value using square brackets and the key 'version'.

3

Write a function to print the DAG version

Define a function called print_dag_version that takes no arguments and prints the text "DAG version: {dag_version}" using an f-string.

Apache Airflow

dag_info = {'dag_id': 'example_dag_v1', 'version': '1.0'}
dag_version = dag_info['version']
# Define print_dag_version function to print DAG version
# Your code here

Hint

Use def to define the function and an f-string inside print().

4

Print the DAG version output

Call the function print_dag_version() to display the DAG version.

Apache Airflow

dag_info = {'dag_id': 'example_dag_v1', 'version': '1.0'}
dag_version = dag_info['version']

def print_dag_version():
    print(f"DAG version: {dag_version}")
# Call print_dag_version to show the version
# Your code here

Hint

Simply call the function by its name followed by parentheses.

Practice

1. What is the main purpose of DAG versioning in Airflow?

easy

A. To speed up the execution of tasks in a DAG

B. To reduce the number of DAGs in the Airflow UI

C. To give each workflow version a unique name for safe management

D. To automatically fix errors in DAG code

Solution

1. Step 1: Understand DAG versioning concept

   DAG versioning means assigning unique names to different versions of workflows to avoid confusion and conflicts.

2. Step 2: Identify the main benefit

   This unique naming helps run and manage multiple workflow versions safely without overwriting or mixing them.

3. Final Answer:

   To give each workflow version a unique name for safe management -> Option C

4. Quick Check:

   DAG versioning = unique naming for safety [OK]

Hint: Remember: versioning means unique names for workflows [OK]

Common Mistakes:

• Thinking versioning speeds up task execution
• Confusing versioning with error fixing
• Believing versioning reduces DAG count

2. Which of the following is a correct way to name a DAG version in Airflow?

easy

A. my_dag_v1

B. my dag version 1

C. my-dag-version-1!

D. my.dag.version.1

Solution

1. Step 1: Check valid DAG naming rules

   DAG names should be simple strings without spaces or special characters like ! or dots.

2. Step 2: Identify the valid option

   my_dag_v1 uses underscores and no spaces or special characters, making it valid.

3. Final Answer:

   my_dag_v1 -> Option A

4. Quick Check:

   Valid DAG names use underscores, no spaces [OK]

Hint: Use underscores, avoid spaces and special chars [OK]

Common Mistakes:

• Using spaces in DAG names
• Including special characters like !
• Using dots which can cause issues

3. Given this DAG code snippet, what will be the DAG id when versioning is applied as shown?

version = 3
dag_id = f"example_dag_v{version}"
print(dag_id)

medium

A. example_dagv3

B. example_dag_version3

C. example_dag_3

D. example_dag_v3

Solution

1. Step 1: Understand f-string usage

   The f-string inserts the value of 'version' variable into the string where {version} is placed.

2. Step 2: Substitute version value

   Since version = 3, the string becomes 'example_dag_v3'.

3. Final Answer:

   example_dag_v3 -> Option D

4. Quick Check:

   f-string with version=3 = example_dag_v3 [OK]

Hint: f-string inserts variable value directly [OK]

Common Mistakes:

• Confusing variable placement in f-string
• Adding extra words like 'version' manually
• Missing underscores in the name

4. You have two DAG versions named data_pipeline_v1 and data_pipeline_v2. You accidentally overwrite data_pipeline_v1 code with v2 code but keep the same DAG id. What is the main problem here?

medium

A. Airflow will only run the latest code under one DAG id, losing version 1

B. The scheduler will crash due to duplicate DAG ids

C. Airflow will create duplicate DAGs with same id

D. Both versions run simultaneously causing conflicts

Solution

1. Step 1: Understand DAG id uniqueness

   Airflow identifies DAGs by their DAG id. If two versions share the same id, only one is recognized.

2. Step 2: Identify effect of overwriting code

   Overwriting v1 code with v2 under same id means v1 is lost and only v2 runs.

3. Final Answer:

   Airflow will only run the latest code under one DAG id, losing version 1 -> Option A

4. Quick Check:

   Same DAG id means only one version runs [OK]

Hint: Unique DAG ids prevent overwriting versions [OK]

Common Mistakes:

• Thinking both versions run at once
• Expecting duplicate DAGs with same id
• Assuming scheduler crashes on duplicates

5. You want to manage multiple DAG versions without changing DAG ids manually. Which strategy below best helps you achieve this in Airflow?

hard

A. Manually rename DAG files for each version

B. Use Airflow Variables to store the version and dynamically set DAG id

C. Use the same DAG id and overwrite code for each version

D. Disable all DAGs except the latest version

Solution

1. Step 1: Understand dynamic DAG id setting

   Using Airflow Variables allows you to store version info externally and inject it into DAG id dynamically.

2. Step 2: Compare with other options

   Manual renaming is error-prone, overwriting loses old versions, disabling DAGs is manual and not scalable.

3. Final Answer:

   Use Airflow Variables to store the version and dynamically set DAG id -> Option B

4. Quick Check:

   Dynamic DAG id via Variables = best versioning [OK]

Hint: Use Variables to automate version naming [OK]

Common Mistakes:

• Relying on manual file renaming
• Overwriting code without versioning
• Disabling DAGs instead of versioning