Celery executor for distributed execution in Apache Airflow - Step-by-Step Execution

Process Flow - Celery executor for distributed execution

Start Airflow Scheduler

↓

Scheduler sends task to Celery Broker

↓

Celery Broker queues task

↓

Celery Worker picks task from Broker

↓

Worker executes task

↓

Worker sends task result back to Backend

↓

Scheduler updates task status

The Airflow scheduler sends tasks to a message broker. Celery workers pick tasks from the broker, execute them, and send results back. The scheduler updates task status accordingly.

Execution Sample

Apache Airflow

from airflow import DAG
from airflow.operators.bash import BashOperator
from datetime import datetime

dag = DAG('example', start_date=datetime(2024,1,1))
t1 = BashOperator(task_id='print_date', bash_command='date', dag=dag)

Defines a simple Airflow DAG with one task that runs the 'date' command using Celery executor.

Process Table

Step	Component	Action	State Change	Output/Result
1	Scheduler	Detects new task 'print_date'	Task queued to Celery broker	Task 'print_date' sent to broker
2	Celery Broker	Receives task 'print_date'	Task added to queue	Task waiting in queue
3	Celery Worker	Fetches task 'print_date' from broker	Task removed from queue, execution started	Worker starts running 'date' command
4	Celery Worker	Executes 'date' command	Task running	Outputs current date/time
5	Celery Worker	Sends result back to backend	Task marked as completed	Result stored in Airflow metadata DB
6	Scheduler	Updates task status to success	Task state updated	Task 'print_date' marked SUCCESS
7	Scheduler	Checks for next tasks	No more tasks	Scheduler idle

💡 All tasks executed and marked complete; no pending tasks remain

Status Tracker

Variable	Start	After Step 1	After Step 3	After Step 5	Final
Task State	None	Queued	Running	Completed	Success
Broker Queue	Empty	Contains 'print_date'	Empty	Empty	Empty
Worker Status	Idle	Idle	Busy executing 'print_date'	Idle	Idle

Key Moments - 3 Insights

Why does the task state change from 'Queued' to 'Running' only after the worker picks it?

Because the scheduler only queues the task to the broker. The task state changes to 'Running' when a worker actually starts executing it, as shown in steps 2 and 3 of the execution table.

What happens if no worker picks the task from the broker?

The task stays in the broker queue in 'Queued' state and does not progress to 'Running' or 'Completed', as seen in step 2 where the broker holds the task until a worker fetches it.

How does the scheduler know when a task is completed?

The worker sends the result back to the backend, which updates the task status. The scheduler then reads this update and marks the task as 'Success', as shown in steps 5 and 6.

Visual Quiz - 3 Questions

Look at the execution table, at which step does the Celery worker start executing the task?

A Step 3

B Step 2

C Step 4

D Step 5

💡 Hint

Check the 'Action' column for when the worker fetches the task from the broker.

According to the variable tracker, what is the state of the broker queue after step 3?

A Contains multiple tasks

B Empty

C Contains 'print_date'

D Unknown

💡 Hint

Look at the 'Broker Queue' row after 'After Step 3' column.

If the worker fails to send the result back, what will the scheduler's task state remain as?

A Success

B Queued

C Running

D Failed

💡 Hint

Refer to the key moment about how the scheduler updates task status after receiving results.

Concept Snapshot

Celery executor lets Airflow run tasks on multiple workers.
Scheduler sends tasks to a message broker.
Workers pick tasks from the broker and run them.
Workers send results back to update task status.
This enables distributed, parallel task execution.

Full Transcript

This visual execution shows how Airflow uses the Celery executor for distributed task execution. The scheduler detects a new task and sends it to the Celery broker, which queues it. A Celery worker picks the task from the broker and starts executing it. After running the task, the worker sends the result back to the backend. The scheduler then updates the task status to success. Variables like task state, broker queue, and worker status change step-by-step to reflect this flow. Key moments clarify common confusions about task state changes and communication between components. The quiz tests understanding of when tasks move between states and how components interact.

Practice

1. What is the main purpose of using the CeleryExecutor in Apache Airflow?

easy

A. To replace the Airflow webserver with a faster interface

B. To store task logs locally on the scheduler machine

C. To execute tasks only on the local machine sequentially

D. To run tasks on multiple machines simultaneously for better scalability

Solution

1. Step 1: Understand CeleryExecutor's role

   CeleryExecutor allows Airflow to distribute task execution across multiple worker machines.

2. Step 2: Compare options with this role

   Only To run tasks on multiple machines simultaneously for better scalability describes running tasks on many machines at once, which matches CeleryExecutor's purpose.

3. Final Answer:

   To run tasks on multiple machines simultaneously for better scalability -> Option D

4. Quick Check:

   CeleryExecutor = distributed task execution [OK]

Hint: CeleryExecutor means many workers run tasks in parallel [OK]

Common Mistakes:

• Thinking CeleryExecutor runs tasks only locally
• Confusing CeleryExecutor with Airflow webserver
• Assuming CeleryExecutor stores logs locally

2. Which configuration setting in airflow.cfg enables the Celery executor?

easy

A. executor = LocalExecutor

B. executor = CeleryExecutor

C. executor = SequentialExecutor

D. executor = KubernetesExecutor

Solution

1. Step 1: Identify the executor setting in airflow.cfg

   The executor type is set by the executor parameter in airflow.cfg.

2. Step 2: Match the correct executor name for Celery

   To enable Celery executor, the value must be CeleryExecutor.

3. Final Answer:

   executor = CeleryExecutor -> Option B

4. Quick Check:

   executor setting = CeleryExecutor [OK]

Hint: Set executor = CeleryExecutor in airflow.cfg [OK]

Common Mistakes:

• Using LocalExecutor instead of CeleryExecutor
• Misspelling CeleryExecutor in config
• Confusing executor with other unrelated settings

3. Given this Airflow CeleryExecutor setup, what happens when a DAG task is triggered?

1. Scheduler sends task to message broker
2. Worker picks task from broker queue
3. Worker executes task and updates database

medium

A. Task is distributed to a worker and executed asynchronously

B. Task runs only on the scheduler machine

C. Task is executed immediately by the scheduler without queue

D. Task fails because CeleryExecutor does not use a message broker

Solution

1. Step 1: Understand CeleryExecutor task flow

   The scheduler sends tasks to a message broker, which workers listen to and execute asynchronously.

2. Step 2: Match the correct description

   Task is distributed to a worker and executed asynchronously correctly describes the asynchronous distributed execution via workers.

3. Final Answer:

   Task is distributed to a worker and executed asynchronously -> Option A

4. Quick Check:

   CeleryExecutor = async distributed tasks [OK]

Hint: Tasks go to workers via broker, run async [OK]

Common Mistakes:

• Thinking tasks run on scheduler directly
• Ignoring the message broker role
• Assuming synchronous execution

4. You configured Airflow to use CeleryExecutor but tasks are stuck in the queue and never run. What is the most likely cause?

medium

A. The DAG files are missing from the scheduler

B. The scheduler is not running

C. The message broker (e.g., RabbitMQ or Redis) is not running or unreachable

D. The Airflow webserver is down

Solution

1. Step 1: Identify the role of the message broker

   CeleryExecutor relies on a message broker to queue tasks for workers.

2. Step 2: Understand why tasks get stuck

   If the broker is down or unreachable, tasks remain queued and are not delivered to workers.

3. Final Answer:

   The message broker (e.g., RabbitMQ or Redis) is not running or unreachable -> Option C

4. Quick Check:

   Broker down = tasks stuck [OK]

Hint: Check if message broker service is running [OK]

Common Mistakes:

• Blaming scheduler when broker is down
• Ignoring broker connectivity issues
• Assuming webserver affects task execution

5. You want to improve task execution reliability using CeleryExecutor. Which combination of components should you configure to ensure tasks are not lost if a worker crashes?

hard

A. Use a persistent message broker like RabbitMQ and enable result backend in the database

B. Use LocalExecutor and disable the message broker

C. Run all tasks on the scheduler machine without workers

D. Use CeleryExecutor without a result backend

Solution

1. Step 1: Understand reliability needs in CeleryExecutor

   To avoid task loss, the message broker must persist tasks and results must be stored reliably.

2. Step 2: Identify correct components

   RabbitMQ is a persistent broker; enabling a result backend in the database stores task states safely.

3. Final Answer:

   Use a persistent message broker like RabbitMQ and enable result backend in the database -> Option A

4. Quick Check:

   Persistent broker + result backend = reliable tasks [OK]

Hint: Persistent broker + result backend prevent task loss [OK]

Common Mistakes:

• Disabling message broker with CeleryExecutor
• Not enabling result backend causes lost states
• Running tasks only on scheduler reduces reliability