Why production Airflow needs careful setup - See It in Action

Why Production Airflow Needs Careful Setup

📖 Scenario: You are a DevOps engineer setting up Apache Airflow for a company's data workflows. Airflow will run many tasks daily, so it must be reliable and efficient.

🎯 Goal: Understand and demonstrate the basic setup steps that make Airflow production-ready, focusing on configuration and task management.

📋 What You'll Learn

Create a dictionary with Airflow configuration settings

Add a variable to control the maximum number of active tasks

Write a loop to check if tasks exceed the limit

Print a message about task status

💡 Why This Matters

🌍 Real World

In real companies, Airflow runs many automated jobs. Setting it up carefully prevents crashes and slowdowns.

💼 Career

DevOps engineers must configure Airflow properly to keep data pipelines running smoothly and avoid task overload.

1

Create Airflow Configuration Dictionary

Create a dictionary called airflow_config with these exact entries: 'executor': 'LocalExecutor', 'sql_alchemy_conn': 'postgresql+psycopg2://user:password@localhost/airflow', and 'dags_folder': '/usr/local/airflow/dags'.

Apache Airflow

# Create the airflow_config dictionary with the specified keys and values
# Your code here

Hint

Use curly braces to create a dictionary and include the exact keys and values as strings.

2

Add Maximum Active Tasks Variable

Add a variable called max_active_tasks and set it to 5 to limit the number of tasks running at the same time.

Apache Airflow

airflow_config = {
    'executor': 'LocalExecutor',
    'sql_alchemy_conn': 'postgresql+psycopg2://user:password@localhost/airflow',
    'dags_folder': '/usr/local/airflow/dags'
}
# Set max_active_tasks to 5
# Your code here

Hint

Just create a variable with the exact name and assign the number 5.

3

Check Active Tasks Against Limit

Write a for loop using the variable task_id to iterate over the list active_tasks = ['task1', 'task2', 'task3', 'task4', 'task5', 'task6']. Inside the loop, check if the number of tasks exceeds max_active_tasks. If yes, break the loop.

Apache Airflow

airflow_config = {
    'executor': 'LocalExecutor',
    'sql_alchemy_conn': 'postgresql+psycopg2://user:password@localhost/airflow',
    'dags_folder': '/usr/local/airflow/dags'
}
max_active_tasks = 5
active_tasks = ['task1', 'task2', 'task3', 'task4', 'task5', 'task6']
# Loop over active_tasks with task_id and break if count exceeds max_active_tasks
# Your code here

Hint

Use a counter variable to track how many tasks you have checked. Stop the loop when the count is greater than max_active_tasks.

4

Print Task Status Message

Write a print statement that outputs exactly: "Too many active tasks running! Limit is 5."

Apache Airflow

airflow_config = {
    'executor': 'LocalExecutor',
    'sql_alchemy_conn': 'postgresql+psycopg2://user:password@localhost/airflow',
    'dags_folder': '/usr/local/airflow/dags'
}
max_active_tasks = 5
active_tasks = ['task1', 'task2', 'task3', 'task4', 'task5', 'task6']
count = 0
for task_id in active_tasks:
    count += 1
    if count > max_active_tasks:
        break
# Print the warning message about too many active tasks
# Your code here

Hint

Use print() with the exact message inside quotes.