What is Concurrency in Airflow: Explanation and Examples

In Airflow, concurrency refers to the number of tasks that can run at the same time within a DAG or across the system. It controls parallel task execution to optimize resource use and avoid overload.

How It Works

Imagine you have a kitchen where multiple chefs can cook dishes at the same time. Concurrency in Airflow is like deciding how many chefs can work simultaneously without bumping into each other or running out of stove space.

Airflow manages concurrency by setting limits on how many tasks can run in parallel. These limits can be set globally for the whole Airflow system, per DAG (a workflow), or per task. This helps balance workload and system resources, preventing too many tasks from running at once and causing slowdowns or failures.

Example

This example shows how to set concurrency limits in an Airflow DAG to control how many tasks run at the same time.

python

from airflow import DAG
from airflow.operators.bash import BashOperator
from datetime import datetime

with DAG(
    dag_id='concurrency_example',
    start_date=datetime(2024, 1, 1),
    schedule_interval='@daily',
    concurrency=2  # Limit to 2 tasks running at the same time
) as dag:
    task1 = BashOperator(task_id='task1', bash_command='sleep 5')
    task2 = BashOperator(task_id='task2', bash_command='sleep 5')
    task3 = BashOperator(task_id='task3', bash_command='sleep 5')

    task1 >> [task2, task3]

Output

When this DAG runs, only 2 tasks will run at the same time due to the concurrency limit. For example, task1 runs first, then task2 and task3 run together after task1 finishes.

When to Use

Use concurrency settings in Airflow when you want to control how many tasks run simultaneously to avoid overloading your system or external services. For example:

• If your database can only handle a few connections at once, limit concurrency to prevent errors.
• When running resource-heavy tasks, limit concurrency to keep your server stable.
• To speed up workflows by running multiple tasks in parallel but within safe limits.

Key Points

• Concurrency controls how many tasks run at the same time in Airflow.
• It can be set globally, per DAG, or per task.
• Helps balance system resources and avoid overload.
• Useful for managing external service limits and resource-heavy tasks.

Key Takeaways

Concurrency in Airflow limits how many tasks run simultaneously to optimize resources.

Set concurrency at the DAG level to control parallel task execution within that workflow.

Use concurrency limits to prevent overloading external systems or your server.

Proper concurrency settings improve workflow efficiency and stability.