Celery executor for distributed execution in Apache Airflow - Step-by-Step Execution

Process Flow - Celery executor for distributed execution

Start Airflow Scheduler

↓

Scheduler sends task to Celery Broker

↓

Celery Broker queues task

↓

Celery Worker picks task from Broker

↓

Worker executes task

↓

Worker sends task result back to Backend

↓

Scheduler updates task status

The Airflow scheduler sends tasks to a message broker. Celery workers pick tasks from the broker, execute them, and send results back. The scheduler updates task status accordingly.

Execution Sample

Apache Airflow

from airflow import DAG
from airflow.operators.bash import BashOperator
from datetime import datetime

dag = DAG('example', start_date=datetime(2024,1,1))
t1 = BashOperator(task_id='print_date', bash_command='date', dag=dag)

Defines a simple Airflow DAG with one task that runs the 'date' command using Celery executor.

Process Table

Step	Component	Action	State Change	Output/Result
1	Scheduler	Detects new task 'print_date'	Task queued to Celery broker	Task 'print_date' sent to broker
2	Celery Broker	Receives task 'print_date'	Task added to queue	Task waiting in queue
3	Celery Worker	Fetches task 'print_date' from broker	Task removed from queue, execution started	Worker starts running 'date' command
4	Celery Worker	Executes 'date' command	Task running	Outputs current date/time
5	Celery Worker	Sends result back to backend	Task marked as completed	Result stored in Airflow metadata DB
6	Scheduler	Updates task status to success	Task state updated	Task 'print_date' marked SUCCESS
7	Scheduler	Checks for next tasks	No more tasks	Scheduler idle

💡 All tasks executed and marked complete; no pending tasks remain

Status Tracker

Variable	Start	After Step 1	After Step 3	After Step 5	Final
Task State	None	Queued	Running	Completed	Success
Broker Queue	Empty	Contains 'print_date'	Empty	Empty	Empty
Worker Status	Idle	Idle	Busy executing 'print_date'	Idle	Idle

Key Moments - 3 Insights

Why does the task state change from 'Queued' to 'Running' only after the worker picks it?

Because the scheduler only queues the task to the broker. The task state changes to 'Running' when a worker actually starts executing it, as shown in steps 2 and 3 of the execution table.

What happens if no worker picks the task from the broker?

The task stays in the broker queue in 'Queued' state and does not progress to 'Running' or 'Completed', as seen in step 2 where the broker holds the task until a worker fetches it.

How does the scheduler know when a task is completed?

The worker sends the result back to the backend, which updates the task status. The scheduler then reads this update and marks the task as 'Success', as shown in steps 5 and 6.

Visual Quiz - 3 Questions

Look at the execution table, at which step does the Celery worker start executing the task?

A Step 3

B Step 2

C Step 4

D Step 5

💡 Hint

Check the 'Action' column for when the worker fetches the task from the broker.

According to the variable tracker, what is the state of the broker queue after step 3?

A Contains multiple tasks

B Empty

C Contains 'print_date'

D Unknown

💡 Hint

Look at the 'Broker Queue' row after 'After Step 3' column.

If the worker fails to send the result back, what will the scheduler's task state remain as?

A Success

B Queued

C Running

D Failed

💡 Hint

Refer to the key moment about how the scheduler updates task status after receiving results.

Concept Snapshot

Celery executor lets Airflow run tasks on multiple workers.
Scheduler sends tasks to a message broker.
Workers pick tasks from the broker and run them.
Workers send results back to update task status.
This enables distributed, parallel task execution.

Full Transcript

This visual execution shows how Airflow uses the Celery executor for distributed task execution. The scheduler detects a new task and sends it to the Celery broker, which queues it. A Celery worker picks the task from the broker and starts executing it. After running the task, the worker sends the result back to the backend. The scheduler then updates the task status to success. Variables like task state, broker queue, and worker status change step-by-step to reflect this flow. Key moments clarify common confusions about task state changes and communication between components. The quiz tests understanding of when tasks move between states and how components interact.