Why orchestration is needed for data pipelines in Apache Airflow - See It in Action

Why orchestration is needed for data pipelines

📖 Scenario: You work in a company that collects data from many sources every day. You want to process this data step-by-step to get useful reports. But the steps must happen in order, and some steps depend on others finishing first.This is like baking a cake: you must mix ingredients before baking, and bake before decorating. If you do steps in the wrong order, the cake will not be good.

🎯 Goal: You will create a simple example of a data pipeline using Airflow concepts. You will see why orchestration is needed to run tasks in the right order automatically.

📋 What You'll Learn

Create a list of data sources

Add a variable to count how many sources to process

Write a loop to print processing steps for each source

Print a final message showing all sources processed

💡 Why This Matters

🌍 Real World

Data pipelines collect and process data from many places. Orchestration tools like Airflow help run these steps in the right order automatically.

💼 Career

Understanding orchestration is key for roles like Data Engineer or DevOps Engineer who build reliable data workflows.

1

Create a list of data sources

Create a list called data_sources with these exact strings: 'source1', 'source2', 'source3'.

Apache Airflow

# Create a list called data_sources with 'source1', 'source2', 'source3'
# Your code here

Need a hint?

Use square brackets [] to create a list and separate items with commas.

2

Add a variable to count data sources

Create a variable called total_sources and set it to the length of the data_sources list using the len() function.

Apache Airflow

data_sources = ['source1', 'source2', 'source3']
# Create total_sources as length of data_sources
# Your code here

Need a hint?

Use len(data_sources) to get how many items are in the list.

3

Print processing steps for each data source

Use a for loop with variable source to iterate over data_sources. Inside the loop, print the exact text: Processing {source} using an f-string.

Apache Airflow

data_sources = ['source1', 'source2', 'source3']
total_sources = len(data_sources)
# Use for loop to print 'Processing {source}' for each source
# Your code here

Need a hint?

Use for source in data_sources: and inside print with f-string f"Processing {source}".

4

Print final message after processing all sources

After the loop, print the exact text: All 3 data sources processed. using the total_sources variable inside an f-string.

Apache Airflow

data_sources = ['source1', 'source2', 'source3']
total_sources = len(data_sources)
for source in data_sources:
    print(f"Processing {source}")
# Print final message using total_sources
# Your code here

Need a hint?

Use print(f"All {total_sources} data sources processed.") after the loop.