High availability configuration in Apache Airflow - Commands & Configuration

Introduction

High availability configuration in Airflow ensures that the workflow system keeps running smoothly even if one part fails. It helps avoid downtime by having backup components ready to take over automatically.

When you want your Airflow scheduler to keep working even if one scheduler instance crashes

When you need your Airflow webserver to be accessible without interruption during maintenance

When you want to prevent losing running tasks if a worker node goes down

When you run Airflow in a production environment with many users and critical workflows

When you want to scale Airflow components across multiple machines for reliability

Config File - airflow.cfg

airflow.cfg

[core]
executor = CeleryExecutor
sql_alchemy_conn = postgresql+psycopg2://airflow:airflow@postgres/airflow
load_examples = False

[webserver]
web_server_port = 8080
web_server_worker_timeout = 120

[scheduler]
max_threads = 2
scheduler_heartbeat_sec = 5
num_runs = -1

[celery]
broker_url = redis://redis:6379/0
result_backend = db+postgresql://airflow:airflow@postgres/airflow
worker_concurrency = 4

[logging]
remote_logging = False

[metrics]
enable = True

[database]
sql_alchemy_pool_size = 5
sql_alchemy_max_overflow = 10

[secrets]
backend = airflow.providers.hashicorp.secrets.vault.VaultBackend

[ha]
enable = True

This airflow.cfg file configures Airflow for high availability:

• executor = CeleryExecutor: Runs tasks distributed across multiple worker nodes.
• sql_alchemy_conn: Connects to a PostgreSQL database shared by all components.
• broker_url and result_backend: Use Redis and PostgreSQL for messaging and task results.
• scheduler settings: Allow multiple scheduler threads and continuous running.
• ha.enable = True: Enables high availability features in Airflow.

This setup allows multiple schedulers and workers to run simultaneously, so if one fails, others continue working.

Commands

Initializes the Airflow metadata database. This sets up the tables Airflow needs to track workflows and tasks.

Terminal

airflow db init

Expected Output

INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.

INFO  [alembic.runtime.migration] Will assume transactional DDL.

INFO  [alembic.runtime.migration] Running upgrade head

INFO  [alembic.runtime.migration] Upgrade done.

Creates an admin user to log into the Airflow web interface.

Terminal

airflow users create --username admin --firstname Admin --lastname User --role Admin --email [e-mail] --password admin123

Expected Output

Created user admin

→

--username - Set the login username

→

--role - Assign user role for permissions

Starts the Airflow scheduler in the background. Multiple schedulers can run for high availability.

Terminal

airflow scheduler &

Expected Output

No output (command runs silently)

Starts a Celery worker in the background to execute tasks. Multiple workers improve reliability and scalability.

Terminal

airflow celery worker &

Expected Output

No output (command runs silently)

Starts the Airflow webserver on port 8080 in the background, providing the user interface.

Terminal

airflow webserver -p 8080 &

Expected Output

[2024-06-01 12:00:00,000] {webserver} INFO - Starting web server on port 8080

→

-p - Specify the port number

Checks the health of the Airflow webserver and scheduler to confirm high availability is active.

Terminal

curl -s http://localhost:8080/health | jq .

Expected Output

{

  "metadatabase": {

    "status": "healthy"

  },

  "scheduler": {

    "status": "healthy"

  },

  "webserver": {

    "status": "healthy"

  }

}

Key Concept

If you remember nothing else from this pattern, remember: running multiple schedulers and workers with a shared database and message broker keeps Airflow running smoothly even if one component fails.

Common Mistakes

Using the SequentialExecutor instead of CeleryExecutor for high availability

SequentialExecutor runs tasks one at a time on a single machine, so it cannot handle multiple workers or schedulers.

Set executor = CeleryExecutor in airflow.cfg to enable distributed task execution.

Not configuring a shared database and message broker for all Airflow components

Without a shared backend, schedulers and workers cannot coordinate, causing failures and inconsistent state.

Use a central PostgreSQL database and Redis broker configured in airflow.cfg for all nodes.

Starting only one scheduler or worker process

Single processes create a single point of failure, defeating high availability.

Run multiple scheduler and worker processes on different machines or containers.

Summary

Initialize the Airflow database to prepare metadata storage.

Create an admin user to access the Airflow web interface.

Start multiple schedulers and Celery workers to enable high availability.

Run the Airflow webserver to provide the user interface.

Verify the health of Airflow components to confirm high availability is active.

Practice

1. What is the main purpose of configuring high availability in Apache Airflow?

easy

A. To reduce the number of DAGs in the system

B. To keep Airflow running even if some components fail

C. To speed up the Airflow UI loading time

D. To limit the number of users accessing Airflow

Solution

1. Step 1: Understand high availability concept

   High availability means the system keeps working even if parts fail.

2. Step 2: Apply to Airflow context

   In Airflow, this means tasks and services continue running despite failures.

3. Final Answer:

   To keep Airflow running even if some components fail -> Option B

4. Quick Check:

   High availability = system stays up despite failures [OK]

Hint: High availability means no downtime during failures [OK]

Common Mistakes:

• Confusing high availability with performance optimization
• Thinking it limits users or DAGs
• Assuming it only affects the UI

2. Which executor is commonly used in Airflow for high availability setups?

easy

A. SequentialExecutor

B. DebugExecutor

C. LocalExecutor

D. CeleryExecutor

Solution

1. Step 1: Recall executor types in Airflow

   SequentialExecutor runs tasks one by one, LocalExecutor runs tasks locally in parallel.

2. Step 2: Identify executor for distributed task handling

   CeleryExecutor uses workers and message brokers for distributed task execution, suitable for high availability.

3. Final Answer:

   CeleryExecutor -> Option D

4. Quick Check:

   Distributed task handling = CeleryExecutor [OK]

Hint: CeleryExecutor enables distributed task execution [OK]

Common Mistakes:

• Choosing SequentialExecutor which is single-threaded
• Confusing LocalExecutor with distributed execution
• Selecting DebugExecutor which is for testing only

3. Given this Airflow configuration snippet for high availability, what is the role of Redis?

[celery]
broker_url = redis://localhost:6379/0

medium

A. Stores task results permanently

B. Manages the Airflow metadata database

C. Acts as a message broker between scheduler and workers

D. Runs the Airflow webserver

Solution

1. Step 1: Identify Redis role in CeleryExecutor

   Redis is used as a message broker to pass task messages between scheduler and workers.

2. Step 2: Differentiate Redis from other components

   Redis is not for storing results permanently or running webserver or metadata DB.

3. Final Answer:

   Acts as a message broker between scheduler and workers -> Option C

4. Quick Check:

   Redis = message broker in CeleryExecutor [OK]

Hint: Redis brokers messages between scheduler and workers [OK]

Common Mistakes:

• Thinking Redis stores metadata or results
• Confusing Redis with PostgreSQL database
• Assuming Redis runs the webserver

4. You configured Airflow with CeleryExecutor but tasks are not being distributed to workers. Which configuration mistake could cause this?

medium

A. Using SQLite as the metadata database

B. Setting broker_url to a Redis server

C. Running multiple scheduler instances

D. Enabling high availability mode in webserver

Solution

1. Step 1: Understand metadata database requirements

   CeleryExecutor requires a robust metadata database like PostgreSQL, not SQLite.

2. Step 2: Identify why SQLite causes issues

   SQLite locks the database and does not support concurrent writes well, blocking task distribution.

3. Final Answer:

   Using SQLite as the metadata database -> Option A

4. Quick Check:

   SQLite blocks concurrency, breaks CeleryExecutor [OK]

Hint: Use PostgreSQL, not SQLite, for CeleryExecutor metadata DB [OK]

Common Mistakes:

• Assuming Redis broker_url is wrong when it is correct
• Thinking multiple schedulers cause task distribution failure
• Confusing webserver settings with task distribution

5. To achieve high availability for Airflow's scheduler, which setup is recommended?

hard

A. Run multiple scheduler instances with PostgreSQL backend and CeleryExecutor

B. Run a single scheduler instance with SQLite backend

C. Run multiple webservers without a scheduler

D. Use LocalExecutor with a single scheduler instance

Solution

1. Step 1: Understand scheduler high availability needs

   Multiple scheduler instances increase reliability and avoid single point of failure.

2. Step 2: Combine with proper backend and executor

   PostgreSQL supports concurrency well; CeleryExecutor enables distributed task execution.

3. Step 3: Eliminate incorrect options

   SQLite and single scheduler limit availability; webservers alone don't schedule tasks; LocalExecutor is not distributed.

4. Final Answer:

   Run multiple scheduler instances with PostgreSQL backend and CeleryExecutor -> Option A

5. Quick Check:

   Multiple schedulers + PostgreSQL + CeleryExecutor = HA scheduler [OK]

Hint: Multiple schedulers + PostgreSQL + CeleryExecutor = high availability [OK]

Common Mistakes:

• Using SQLite which blocks concurrency
• Running only one scheduler instance
• Confusing webserver role with scheduler