Celery executor for distributed execution in Apache Airflow - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

How Celery Executor manages task distribution

In Apache Airflow, the Celery Executor allows distributed task execution. Which statement best describes how the Celery Executor distributes tasks?

A It runs all tasks sequentially on the scheduler machine without using workers.

B It uses a local thread pool to execute tasks concurrently on the same machine.

C It sends tasks to multiple worker nodes via a message broker, allowing parallel execution.

D It stores tasks in a database and executes them only when manually triggered.

💡 Hint

Think about how tasks are sent to different machines to run at the same time.

The Celery Executor sends tasks to multiple worker nodes through a message broker like RabbitMQ or Redis. This allows tasks to run in parallel on different machines, improving scalability.

💻 Command Output

intermediate

Output of Celery worker startup command

What is the expected output when you start an Airflow Celery worker with the command airflow celery worker?

Apache Airflow

airflow celery worker

A

Starting Celery worker with hostname airflow@worker1
[INFO] Ready to accept tasks

B Error: Unknown command 'celery worker'

C Scheduler started successfully

D No module named 'celery'

💡 Hint

Consider what a worker process does when it starts.

When you run airflow celery worker, it starts a Celery worker process that connects to the message broker and waits for tasks. The output shows the worker hostname and readiness.

❓ Configuration

advanced

Correct Celery Executor configuration in airflow.cfg

Which airflow.cfg snippet correctly configures Airflow to use the Celery Executor with Redis as the broker?

A

[core]
executor = CeleryExecutor

[celery]
broker_url = redis://localhost:6379/0
result_backend = db+sqlite:///airflow.db

B

[core]
executor = LocalExecutor

[celery]
broker_url = redis://localhost:6379/0
result_backend = db+sqlite:///airflow.db

C

[core]
executor = CeleryExecutor

[celery]
broker_url = amqp://guest:guest@localhost:5672//
result_backend = db+sqlite:///airflow.db

D

[core]
executor = CeleryExecutor

[celery]
broker_url = redis://localhost:6379
result_backend = redis://localhost:6379/0

💡 Hint

Check the executor type and the broker URL format carefully.

The Celery Executor must be set in the [core] section. The broker_url for Redis includes the database number (/0). The result_backend can be a database URL like SQLite.

❓ Troubleshoot

advanced

Diagnosing Celery worker connection failure

You start a Celery worker but it fails to connect to the message broker. Which log message indicates the cause?

A Task executed successfully

B Connection refused: unable to connect to redis://localhost:6379/0

C Scheduler heartbeat missed

D Database locked error

💡 Hint

Look for connection or network errors in logs.

A 'Connection refused' error means the Celery worker cannot reach the Redis broker, possibly because Redis is not running or the URL is incorrect.

🔀 Workflow

expert

Order of steps to set up Celery Executor in Airflow

Arrange the steps in the correct order to set up the Celery Executor for distributed task execution in Airflow.

A 2,3,1,4

B 2,1,4,3

C 1,2,3,4

D 1,2,4,3

💡 Hint

Think about what must be ready before starting workers and scheduler.

First, the message broker must be installed and running. Then configure Airflow to use CeleryExecutor with the broker URL. Next, start the Celery workers so they can receive tasks. Finally, start the scheduler to send tasks to workers.