Celery executor for distributed execution in Apache Airflow - Mini Project: Build & Apply

Celery Executor Setup for Distributed Task Execution in Airflow

📖 Scenario: You are setting up Apache Airflow to run tasks distributed across multiple worker machines. To do this, you will configure Airflow to use the Celery executor, which allows tasks to be sent to a queue and processed by multiple workers.This setup helps when you want to run many tasks in parallel and spread the work across several computers.

🎯 Goal: Configure Airflow to use the Celery executor by creating the necessary configuration settings. You will define the executor type, set up the broker URL for task messaging, and verify the configuration by printing the executor setting.

📋 What You'll Learn

Create a variable executor and set it to the string 'CeleryExecutor'.

Create a variable broker_url and set it to the string 'redis://localhost:6379/0' to specify the message broker.

Print the value of the executor variable to confirm the setup.

💡 Why This Matters

🌍 Real World

Using Celery executor in Airflow allows companies to run many data processing tasks in parallel across multiple machines, speeding up workflows.

💼 Career

Understanding how to configure distributed executors like Celery is important for DevOps engineers and data engineers managing scalable workflow systems.

1

Define the Airflow executor type

Create a variable called executor and set it to the string 'CeleryExecutor'.

Apache Airflow

# Create a variable called executor and set it to 'CeleryExecutor'
# Your code here

Hint

Use a simple assignment like executor = 'CeleryExecutor'.

2

Set the broker URL for Celery

Create a variable called broker_url and set it to the string 'redis://localhost:6379/0' to specify the Redis broker URL.

Apache Airflow

executor = 'CeleryExecutor'
# Create a variable called broker_url and set it to 'redis://localhost:6379/0'
# Your code here

Hint

Assign the Redis URL string to broker_url.

3

Add a configuration dictionary for Airflow

Create a dictionary called airflow_config with keys 'executor' and 'broker_url' set to the variables executor and broker_url respectively.

Apache Airflow

executor = 'CeleryExecutor'
broker_url = 'redis://localhost:6379/0'
# Create a dictionary airflow_config with keys 'executor' and 'broker_url'
# Your code here

Hint

Use airflow_config = {'executor': executor, 'broker_url': broker_url}.

4

Print the executor configuration

Write a print statement to display the value of airflow_config['executor'].

Apache Airflow

executor = 'CeleryExecutor'
broker_url = 'redis://localhost:6379/0'
airflow_config = {'executor': executor, 'broker_url': broker_url}
# Print the executor value from airflow_config
# Your code here

Hint

Use print(airflow_config['executor']) to show the executor.