Celery executor for distributed execution in Apache Airflow - Cheat Sheet & Quick Revision

Recall & Review

beginner

What is the Celery executor in Apache Airflow?

The Celery executor is a way to run Airflow tasks in parallel across multiple worker machines using Celery, a distributed task queue system.

beginner

How does the Celery executor improve task execution in Airflow?

It allows tasks to run on multiple worker nodes simultaneously, improving scalability and fault tolerance compared to running tasks on a single machine.

intermediate

What components are required to use the Celery executor in Airflow?

You need a message broker (like RabbitMQ or Redis), multiple worker nodes running Celery workers, and the Airflow scheduler configured to use the Celery executor.

intermediate

Which configuration file do you modify to enable the Celery executor in Airflow?

You modify the airflow.cfg file, setting the executor parameter to CeleryExecutor and configuring the broker URL and result backend.

beginner

What is the role of the message broker in the Celery executor setup?

The message broker queues tasks and sends them to available worker nodes for execution, acting like a post office delivering task messages.

What does the Celery executor use to distribute tasks across workers?

A A message broker like RabbitMQ or Redis

B A single-threaded scheduler

C Direct database queries

D Local file system locks

The Celery executor uses a message broker such as RabbitMQ or Redis to queue and distribute tasks to workers.

Which Airflow configuration parameter sets the executor to Celery?

A executor = CeleryExecutor

B executor = LocalExecutor

C executor = SequentialExecutor

D executor = KubernetesExecutor

Setting executor = CeleryExecutor in airflow.cfg enables the Celery executor.

What is NOT required to run Airflow with the Celery executor?

A A message broker

B A local-only scheduler

C A web server

D Multiple worker nodes

The scheduler must be running and configured for Celery, but it is not limited to local-only; it works with distributed workers.

What happens if a Celery worker node fails during task execution?

A The scheduler shuts down

B The entire Airflow system stops

C The task is lost permanently

D The task is retried or reassigned to another worker

Celery can retry or reassign tasks if a worker fails, improving fault tolerance.

Which of these is a common message broker used with Celery executor?

A MongoDB

B MySQL

C Redis

D Apache Kafka

Redis is a popular message broker used with Celery for task queuing.

Explain how the Celery executor enables distributed task execution in Airflow.

Think about how tasks move from scheduler to workers.

Describe the key configuration changes needed to switch Airflow to use the Celery executor.

Focus on airflow.cfg and required services.