High availability configuration in Apache Airflow - Cheat Sheet & Quick Revision

Recall & Review

beginner

What is high availability (HA) in Airflow?

High availability in Airflow means setting up the system so it keeps running without interruption, even if some parts fail. It ensures workflows continue without downtime.

intermediate

Name two key components to configure for Airflow high availability.

1. Multiple Airflow schedulers running in parallel. 2. A highly available metadata database (like PostgreSQL with replication).

intermediate

Why use a message broker like Redis or RabbitMQ in Airflow HA setup?

A message broker helps coordinate tasks between multiple schedulers and workers, ensuring task distribution and avoiding conflicts.

beginner

What role does the metadata database play in Airflow high availability?

The metadata database stores the state of all workflows and tasks. Making it highly available prevents data loss and keeps Airflow running smoothly.

beginner

How does running multiple schedulers improve Airflow availability?

Multiple schedulers share the workload and take over if one fails, so workflows keep running without interruption.

Which component is essential for Airflow high availability to avoid a single point of failure?

A Single worker node

B Single scheduler instance

C Local file storage

D Metadata database with replication

A replicated metadata database prevents data loss and downtime if one database instance fails.

What is the purpose of running multiple Airflow schedulers in HA setup?

A To balance task scheduling and provide failover

B To reduce the number of workers needed

C To store logs more efficiently

D To increase the number of DAGs

Multiple schedulers share the scheduling load and provide backup if one scheduler fails.

Which message broker is commonly used in Airflow HA for task coordination?

A MySQL

B Redis

C MongoDB

D SQLite

Redis is a popular message broker used to coordinate tasks between schedulers and workers.

What happens if the Airflow metadata database is not highly available?

A DAGs will execute twice

B Schedulers will run faster

C Airflow may lose task state and stop working properly

D Workers will automatically restart

Without a highly available database, Airflow can lose critical state information causing failures.

Which of these is NOT a benefit of Airflow high availability?

A Elimination of all bugs in DAGs

B Automatic failover of schedulers

C Zero downtime for workflows

D Guaranteed task execution order without conflicts

High availability improves uptime and failover but does not fix bugs in workflow code.

Explain how multiple schedulers and a replicated metadata database work together to provide high availability in Airflow.

Think about how tasks keep running even if one part stops working.

Describe the role of a message broker in an Airflow high availability setup.

It's like a traffic controller for tasks.