LocalExecutor in Airflow: What It Is and When to Use

LocalExecutor in Airflow is a way to run multiple tasks in parallel on the same machine using multiple processes. It allows Airflow to execute tasks concurrently locally without needing external systems like Celery or Kubernetes.

How It Works

LocalExecutor works by running multiple task instances at the same time on a single machine using separate processes. Imagine a kitchen where one cook can only prepare one dish at a time (like the default SequentialExecutor). With LocalExecutor, you have several cooks working in parallel in the same kitchen, each preparing a different dish simultaneously.

This executor uses Python's multiprocessing to create separate workers that pick up tasks from the Airflow scheduler and run them independently. It is simpler than distributed executors because it does not require extra infrastructure like message brokers or external worker nodes.

Example

This example shows how to configure Airflow to use LocalExecutor in the airflow.cfg file and a simple DAG that runs two tasks in parallel.

python

[core]
executor = LocalExecutor

from airflow import DAG
from airflow.operators.bash import BashOperator
from datetime import datetime

default_args = {
    'start_date': datetime(2024, 1, 1),
}

dag = DAG('localexecutor_example', default_args=default_args, schedule_interval='@once')

task1 = BashOperator(
    task_id='print_date',
    bash_command='date',
    dag=dag
)

task2 = BashOperator(
    task_id='sleep',
    bash_command='sleep 5',
    dag=dag
)

# Both tasks run in parallel when the DAG is triggered

Output

When triggered, both tasks run at the same time on the local machine, showing the current date and sleeping for 5 seconds concurrently.

When to Use

Use LocalExecutor when you want to run multiple tasks in parallel but do not need a distributed setup. It is ideal for small to medium workloads on a single machine where installing and managing a complex system like Celery or Kubernetes is not necessary.

For example, if you have a development environment or a small production server and want better parallelism than the default SequentialExecutor, LocalExecutor is a good choice. However, for large-scale or multi-node setups, distributed executors are better.

Key Points

• LocalExecutor runs tasks in parallel using multiple processes on one machine.
• It requires no external message broker or worker nodes.
• Good for moderate parallelism without complex infrastructure.
• Configured by setting executor = LocalExecutor in airflow.cfg.
• Not suitable for large distributed environments.

Key Takeaways

LocalExecutor enables parallel task execution on a single machine using multiple processes.

It is simpler than distributed executors because it needs no external brokers or workers.

Configure it by setting executor = LocalExecutor in airflow.cfg.

Best for small to medium workloads needing parallelism without complex setup.

Not recommended for large-scale or multi-node Airflow deployments.