How to Configure Executor in Apache Airflow

To configure the executor in Airflow, edit the airflow.cfg file and set the executor option under the [core] section to your desired executor type like SequentialExecutor, LocalExecutor, or KubernetesExecutor. Then restart the Airflow services to apply the change.

Syntax

The executor is configured in the airflow.cfg file under the [core] section. The key is executor and its value defines how tasks run.

• SequentialExecutor: Runs tasks one at a time, good for testing.
• LocalExecutor: Runs tasks in parallel on the same machine.
• KubernetesExecutor: Runs tasks in Kubernetes pods for scalability.

ini

[core]
executor = LocalExecutor

Example

This example shows how to set the executor to LocalExecutor in the airflow.cfg file. This allows Airflow to run multiple tasks in parallel on the same machine.

ini

[core]
executor = LocalExecutor

Output

After saving this change and restarting Airflow, tasks will run in parallel locally.

Common Pitfalls

Common mistakes include:

• Not restarting Airflow after changing the executor, so changes don't apply.
• Using SequentialExecutor in production, which limits parallelism.
• Misconfiguring executors that require extra setup, like KubernetesExecutor needing a Kubernetes cluster.

ini

[core]
executor = SequentialExecutor  # Wrong for production

# Correct for production with parallelism
executor = LocalExecutor

Quick Reference

Executor	Description	Use Case
SequentialExecutor	Runs tasks one at a time	Testing or simple setups
LocalExecutor	Runs tasks in parallel on one machine	Small to medium production
CeleryExecutor	Distributes tasks across workers	Large scale with multiple machines
KubernetesExecutor	Runs tasks in Kubernetes pods	Cloud-native scalable setups

Key Takeaways

Set the executor in airflow.cfg under the [core] section using the executor key.

Restart Airflow services after changing the executor to apply changes.

Choose SequentialExecutor for testing and LocalExecutor or others for production.

Executors like KubernetesExecutor require additional infrastructure setup.

Avoid using SequentialExecutor in production to enable parallel task execution.