Task dependencies (>> and << operators) in Apache Airflow - Mini Project: Build & Apply

Task dependencies (>> and << operators) in Airflow

📖 Scenario: You are managing a simple data pipeline using Apache Airflow. You want to control the order in which tasks run by setting dependencies between them.Think of it like a kitchen where you must prepare ingredients before cooking. You want to make sure the tasks happen in the right order.

🎯 Goal: Build an Airflow DAG with three tasks: extract, transform, and load. Use the >> and operators to set the dependencies so that extract runs before transform, and transform runs before load.

📋 What You'll Learn

Create three Airflow tasks named extract, transform, and load

Use the >> operator to set extract to run before transform

Use the >> operator to set transform to run before load

Print the task dependencies to verify the order

💡 Why This Matters

🌍 Real World

In real data pipelines, tasks must run in a specific order to process data correctly. Using task dependencies ensures the pipeline runs smoothly without errors.

💼 Career

Understanding task dependencies is essential for Airflow users and DevOps engineers to build reliable automated workflows.

1

Create Airflow tasks

Create three Airflow tasks named extract, transform, and load using DummyOperator inside a DAG called example_dag. Use task_id values exactly as the task names.

Apache Airflow

from airflow import DAG
from airflow.operators.dummy import DummyOperator
from datetime import datetime

with DAG('example_dag', start_date=datetime(2024, 1, 1), schedule_interval='@daily') as dag:
    # Create DummyOperator tasks named extract, transform, and load
    # Your code here

Need a hint?

Use DummyOperator to create simple placeholder tasks. Assign each to variables named exactly extract, transform, and load.

2

Add task dependency variable

Create a variable called dependency_order and set it to the expression extract >> transform to define that extract runs before transform.

Apache Airflow

from airflow import DAG
from airflow.operators.dummy import DummyOperator
from datetime import datetime

with DAG('example_dag', start_date=datetime(2024, 1, 1), schedule_interval='@daily') as dag:
    extract = DummyOperator(task_id='extract')
    transform = DummyOperator(task_id='transform')
    load = DummyOperator(task_id='load')
    # Create dependency_order variable with extract >> transform
    # Your code here

Need a hint?

Use the >> operator between extract and transform and assign it to dependency_order.

3

Set full task dependencies

Use the >> operator to set transform to run before load by writing transform >> load. Add this to the existing dependencies so the full order is extract >> transform >> load.

Apache Airflow

from airflow import DAG
from airflow.operators.dummy import DummyOperator
from datetime import datetime

with DAG('example_dag', start_date=datetime(2024, 1, 1), schedule_interval='@daily') as dag:
    extract = DummyOperator(task_id='extract')
    transform = DummyOperator(task_id='transform')
    load = DummyOperator(task_id='load')
    dependency_order = extract >> transform
    # Add transform >> load to set full dependencies
    # Your code here

Need a hint?

Use transform >> load to set the next dependency in the chain.

4

Print task dependencies

Print the downstream task IDs of extract using print(extract.downstream_task_ids) to verify the dependencies.

Apache Airflow

from airflow import DAG
from airflow.operators.dummy import DummyOperator
from datetime import datetime

with DAG('example_dag', start_date=datetime(2024, 1, 1), schedule_interval='@daily') as dag:
    extract = DummyOperator(task_id='extract')
    transform = DummyOperator(task_id='transform')
    load = DummyOperator(task_id='load')
    dependency_order = extract >> transform
    transform >> load
    # Print downstream task IDs of extract
    # Your code here

Need a hint?

Use print(extract.downstream_task_ids) to see which tasks run after extract.