Task dependencies (>> and << operators) in Apache Airflow - Mini Project: Build & Apply

Task dependencies (>> and << operators) in Airflow

📖 Scenario: You are managing a simple data pipeline using Apache Airflow. You want to control the order in which tasks run by setting dependencies between them.Think of it like a kitchen where you must prepare ingredients before cooking. You want to make sure the tasks happen in the right order.

🎯 Goal: Build an Airflow DAG with three tasks: extract, transform, and load. Use the >> and operators to set the dependencies so that extract runs before transform, and transform runs before load.

📋 What You'll Learn

Create three Airflow tasks named extract, transform, and load

Use the >> operator to set extract to run before transform

Use the >> operator to set transform to run before load

Print the task dependencies to verify the order

💡 Why This Matters

🌍 Real World

In real data pipelines, tasks must run in a specific order to process data correctly. Using task dependencies ensures the pipeline runs smoothly without errors.

💼 Career

Understanding task dependencies is essential for Airflow users and DevOps engineers to build reliable automated workflows.

1

Create Airflow tasks

Create three Airflow tasks named extract, transform, and load using DummyOperator inside a DAG called example_dag. Use task_id values exactly as the task names.

Apache Airflow

from airflow import DAG
from airflow.operators.dummy import DummyOperator
from datetime import datetime

with DAG('example_dag', start_date=datetime(2024, 1, 1), schedule_interval='@daily') as dag:
    # Create DummyOperator tasks named extract, transform, and load
    # Your code here

Hint

Use DummyOperator to create simple placeholder tasks. Assign each to variables named exactly extract, transform, and load.

2

Add task dependency variable

Create a variable called dependency_order and set it to the expression extract >> transform to define that extract runs before transform.

Apache Airflow

from airflow import DAG
from airflow.operators.dummy import DummyOperator
from datetime import datetime

with DAG('example_dag', start_date=datetime(2024, 1, 1), schedule_interval='@daily') as dag:
    extract = DummyOperator(task_id='extract')
    transform = DummyOperator(task_id='transform')
    load = DummyOperator(task_id='load')
    # Create dependency_order variable with extract >> transform
    # Your code here

Hint

Use the >> operator between extract and transform and assign it to dependency_order.

3

Set full task dependencies

Use the >> operator to set transform to run before load by writing transform >> load. Add this to the existing dependencies so the full order is extract >> transform >> load.

Apache Airflow

from airflow import DAG
from airflow.operators.dummy import DummyOperator
from datetime import datetime

with DAG('example_dag', start_date=datetime(2024, 1, 1), schedule_interval='@daily') as dag:
    extract = DummyOperator(task_id='extract')
    transform = DummyOperator(task_id='transform')
    load = DummyOperator(task_id='load')
    dependency_order = extract >> transform
    # Add transform >> load to set full dependencies
    # Your code here

Hint

Use transform >> load to set the next dependency in the chain.

4

Print task dependencies

Print the downstream task IDs of extract using print(extract.downstream_task_ids) to verify the dependencies.

Apache Airflow

from airflow import DAG
from airflow.operators.dummy import DummyOperator
from datetime import datetime

with DAG('example_dag', start_date=datetime(2024, 1, 1), schedule_interval='@daily') as dag:
    extract = DummyOperator(task_id='extract')
    transform = DummyOperator(task_id='transform')
    load = DummyOperator(task_id='load')
    dependency_order = extract >> transform
    transform >> load
    # Print downstream task IDs of extract
    # Your code here

Hint

Use print(extract.downstream_task_ids) to see which tasks run after extract.

Practice

1. In Airflow, what does the operator task1 >> task2 mean?

easy

A. task1 runs before task2

B. task1 runs after task2

C. task1 and task2 run at the same time

D. task1 is unrelated to task2

Solution

1. Step 1: Understand the meaning of >> operator

   The >> operator in Airflow means "runs before" or "upstream to downstream" dependency.

2. Step 2: Apply to given tasks

   Given task1 >> task2, task1 must finish before task2 starts.

3. Final Answer:

   task1 runs before task2 -> Option A

4. Quick Check:

   >> means runs before [OK]

Hint: Remember: >> means runs before, << means runs after [OK]

Common Mistakes:

• Confusing >> with <<
• Thinking >> means runs after
• Assuming tasks run in parallel

2. Which of the following is the correct syntax to set task2 to run after task1 using Airflow operators?

easy

A. task1 >> task2

B. task2 << task1

C. task2 >> task1

D. task1 << task2

Solution

1. Step 1: Identify the operator meaning

   >> means "runs before"; so task1 >> task2 means task1 runs before task2.

2. Step 2: Match the requirement

   To make task2 run after task1, task1 must run before task2, so task1 >> task2 is correct.

3. Final Answer:

   task1 >> task2 -> Option A

4. Quick Check:

   task2 after task1 = task1 >> task2 [OK]

Hint: Use >> to say runs before, so task1 >> task2 means task2 after task1 [OK]

Common Mistakes:

• Using << backwards
• Swapping task order
• Confusing which task is upstream

3. Given the code snippet:

task1 >> task2 << task3

Which statement is true about the task execution order?

medium

A. task2 runs before task1 and task3

B. task1 runs after task2, and task3 runs after task2

C. task1 runs before task2, and task3 runs before task2

D. task1, task2, and task3 run in parallel

Solution

1. Step 1: Break down the dependencies

   task1 >> task2 means task1 runs before task2; task3 << task2 means task3 runs before task2 (because << means runs after, so task2 runs after task3).

2. Step 2: Combine the relations

   Both task1 and task3 must finish before task2 starts.

3. Final Answer:

   task1 runs before task2, and task3 runs before task2 -> Option C

4. Quick Check:

   task1 and task3 before task2 = task1 runs before task2, and task3 runs before task2 [OK]

Hint: >> means runs before; << means runs after (reverse order) [OK]

Common Mistakes:

• Misreading << as runs before
• Assuming task2 runs first
• Thinking tasks run in parallel

4. Identify the error in this Airflow task dependency code:

task1 >> >> task2

medium

A. task2 runs before task1

B. Syntax error due to double >> operators

C. No error, tasks run sequentially

D. task1 and task2 run in parallel

Solution

1. Step 1: Analyze the operator usage

   The code has two consecutive >> operators without a task in between, which is invalid syntax in Python and Airflow.

2. Step 2: Understand Airflow dependency syntax

   Only one >> operator is allowed between two tasks to set dependency; double >> causes syntax error.

3. Final Answer:

   Syntax error due to double >> operators -> Option B

4. Quick Check:

   Double >> is invalid syntax [OK]

Hint: Only one >> or << between tasks, no doubles [OK]

Common Mistakes:

• Writing multiple >> operators in a row
• Ignoring syntax errors
• Assuming double >> means stronger dependency

5. You want to create a DAG where task1 and task3 run first in parallel, then task2 runs after both finish. Which code correctly sets this dependency?

hard

A. task2 >> [task1, task3]

B. task1 << task2 >> task3

C. task1 >> task2 >> task3

D. [task1, task3] >> task2

Solution

1. Step 1: Understand the requirement

   task1 and task3 run in parallel first, then task2 runs after both finish.

2. Step 2: Translate to Airflow syntax

   Using list syntax [task1, task3] >> task2 means both task1 and task3 run before task2.

3. Step 3: Check other options

   task1 >> task2 >> task3 runs tasks sequentially; B and D have wrong directions.

4. Final Answer:

   [task1, task3] >> task2 -> Option D

5. Quick Check:

   List >> task means all list tasks before next [OK]

Hint: Use list syntax [task1, task3] >> task2 for parallel upstream tasks [OK]

Common Mistakes:

• Using >> in wrong order
• Not using list for parallel tasks
• Confusing << with >>