Celery executor for distributed execution in Apache Airflow - Interactive Code Practice

Practice - 5 Tasks

1fill in blank

easy

Complete the code to set the executor to Celery in airflow.cfg.

Apache Airflow

[core]
executor = [1]

Drag options to blanks, or click blank then click option'

ACeleryExecutor

BLocalExecutor

CSequentialExecutor

DKubernetesExecutor

Attempts:

3 left

💡 Hint

Common Mistakes

Using LocalExecutor instead of CeleryExecutor disables distributed execution.

Misspelling the executor name causes Airflow to fail starting.

Setting executor = CeleryExecutor enables distributed task execution using Celery workers.

2fill in blank

medium

Complete the Celery broker URL to use Redis as the message broker.

Apache Airflow

broker_url = [1]

Drag options to blanks, or click blank then click option'

Amemory://

Bredis://localhost:6379/0

Csqla+sqlite:///celerydb.sqlite

Damqp://guest:guest@localhost:5672//

Attempts:

3 left

💡 Hint

Common Mistakes

Using AMQP URL when Redis is intended.

Using memory broker disables distributed messaging.

The Redis URL redis://localhost:6379/0 configures Celery to use Redis as the broker.

3fill in blank

hard

Fix the error in the Celery result backend URL to use Redis correctly.

Apache Airflow

result_backend = [1]

Drag options to blanks, or click blank then click option'

Adb+sqlite:///results.sqlite

Bredis://localhost:6379

Camqp://guest:guest@localhost:5672//

Dredis://localhost:6379/0

Attempts:

3 left

💡 Hint

Common Mistakes

Leaving out the database number in Redis URL.

Using AMQP URL for result backend instead of Redis.

The Redis result backend URL must specify the database number, like /0.

4fill in blank

hard

Fill both blanks to configure Celery worker concurrency and log level.

Apache Airflow

[celery]
worker_concurrency = [1]
worker_log_level = [2]

Drag options to blanks, or click blank then click option'

A4

BINFO

CDEBUG

D8

Attempts:

3 left

💡 Hint

Common Mistakes

Setting concurrency too low limits parallelism.

Using DEBUG log level can produce too much output.

Setting worker_concurrency = 8 allows 8 parallel tasks. worker_log_level = INFO sets standard logging verbosity.

5fill in blank

hard

Fill all three blanks to complete the Celery task routing configuration.

Apache Airflow

[celery]
task_routes = {"my_task": {"queue": [1], "routing_key": [2], "exchange": [3]

Drag options to blanks, or click blank then click option'

A"high_priority"

B"high.key"

C"high_exchange"

D"default"

Attempts:

3 left

💡 Hint

Common Mistakes

Using default queue when a specific queue is needed.

Mismatching routing key and exchange names.

Tasks routed to high_priority queue use high.key routing key and high_exchange exchange for prioritized processing.

Practice

1. What is the main purpose of using the CeleryExecutor in Apache Airflow?

easy

A. To replace the Airflow webserver with a faster interface

B. To store task logs locally on the scheduler machine

C. To execute tasks only on the local machine sequentially

D. To run tasks on multiple machines simultaneously for better scalability

Solution

1. Step 1: Understand CeleryExecutor's role

   CeleryExecutor allows Airflow to distribute task execution across multiple worker machines.

2. Step 2: Compare options with this role

   Only To run tasks on multiple machines simultaneously for better scalability describes running tasks on many machines at once, which matches CeleryExecutor's purpose.

3. Final Answer:

   To run tasks on multiple machines simultaneously for better scalability -> Option D

4. Quick Check:

   CeleryExecutor = distributed task execution [OK]

Hint: CeleryExecutor means many workers run tasks in parallel [OK]

Common Mistakes:

• Thinking CeleryExecutor runs tasks only locally
• Confusing CeleryExecutor with Airflow webserver
• Assuming CeleryExecutor stores logs locally

2. Which configuration setting in airflow.cfg enables the Celery executor?

easy

A. executor = LocalExecutor

B. executor = CeleryExecutor

C. executor = SequentialExecutor

D. executor = KubernetesExecutor

Solution

1. Step 1: Identify the executor setting in airflow.cfg

   The executor type is set by the executor parameter in airflow.cfg.

2. Step 2: Match the correct executor name for Celery

   To enable Celery executor, the value must be CeleryExecutor.

3. Final Answer:

   executor = CeleryExecutor -> Option B

4. Quick Check:

   executor setting = CeleryExecutor [OK]

Hint: Set executor = CeleryExecutor in airflow.cfg [OK]

Common Mistakes:

• Using LocalExecutor instead of CeleryExecutor
• Misspelling CeleryExecutor in config
• Confusing executor with other unrelated settings

3. Given this Airflow CeleryExecutor setup, what happens when a DAG task is triggered?

1. Scheduler sends task to message broker
2. Worker picks task from broker queue
3. Worker executes task and updates database

medium

A. Task is distributed to a worker and executed asynchronously

B. Task runs only on the scheduler machine

C. Task is executed immediately by the scheduler without queue

D. Task fails because CeleryExecutor does not use a message broker

Solution

1. Step 1: Understand CeleryExecutor task flow

   The scheduler sends tasks to a message broker, which workers listen to and execute asynchronously.

2. Step 2: Match the correct description

   Task is distributed to a worker and executed asynchronously correctly describes the asynchronous distributed execution via workers.

3. Final Answer:

   Task is distributed to a worker and executed asynchronously -> Option A

4. Quick Check:

   CeleryExecutor = async distributed tasks [OK]

Hint: Tasks go to workers via broker, run async [OK]

Common Mistakes:

• Thinking tasks run on scheduler directly
• Ignoring the message broker role
• Assuming synchronous execution

4. You configured Airflow to use CeleryExecutor but tasks are stuck in the queue and never run. What is the most likely cause?

medium

A. The DAG files are missing from the scheduler

B. The scheduler is not running

C. The message broker (e.g., RabbitMQ or Redis) is not running or unreachable

D. The Airflow webserver is down

Solution

1. Step 1: Identify the role of the message broker

   CeleryExecutor relies on a message broker to queue tasks for workers.

2. Step 2: Understand why tasks get stuck

   If the broker is down or unreachable, tasks remain queued and are not delivered to workers.

3. Final Answer:

   The message broker (e.g., RabbitMQ or Redis) is not running or unreachable -> Option C

4. Quick Check:

   Broker down = tasks stuck [OK]

Hint: Check if message broker service is running [OK]

Common Mistakes:

• Blaming scheduler when broker is down
• Ignoring broker connectivity issues
• Assuming webserver affects task execution

5. You want to improve task execution reliability using CeleryExecutor. Which combination of components should you configure to ensure tasks are not lost if a worker crashes?

hard

A. Use a persistent message broker like RabbitMQ and enable result backend in the database

B. Use LocalExecutor and disable the message broker

C. Run all tasks on the scheduler machine without workers

D. Use CeleryExecutor without a result backend

Solution

1. Step 1: Understand reliability needs in CeleryExecutor

   To avoid task loss, the message broker must persist tasks and results must be stored reliably.

2. Step 2: Identify correct components

   RabbitMQ is a persistent broker; enabling a result backend in the database stores task states safely.

3. Final Answer:

   Use a persistent message broker like RabbitMQ and enable result backend in the database -> Option A

4. Quick Check:

   Persistent broker + result backend = reliable tasks [OK]

Hint: Persistent broker + result backend prevent task loss [OK]

Common Mistakes:

• Disabling message broker with CeleryExecutor
• Not enabling result backend causes lost states
• Running tasks only on scheduler reduces reliability