High availability configuration in Apache Airflow - Mini Project: Build & Apply

High Availability Configuration in Apache Airflow

📖 Scenario: You are setting up Apache Airflow for a company that needs its workflows to run without interruption. To achieve this, you will configure Airflow for high availability (HA) so that if one scheduler or webserver fails, another can take over seamlessly.

🎯 Goal: Build a simple Airflow configuration that enables high availability by setting up multiple schedulers and webservers with a shared metadata database.

📋 What You'll Learn

Create an Airflow configuration dictionary with basic settings

Add a configuration variable to enable multiple schedulers

Configure the webserver to run with multiple workers

Print the final Airflow configuration dictionary

💡 Why This Matters

🌍 Real World

High availability ensures that Airflow workflows keep running even if one scheduler or webserver fails, which is critical for business processes that depend on timely data pipelines.

💼 Career

DevOps engineers and data engineers often configure Airflow for high availability to maintain reliable and fault-tolerant workflow orchestration in production environments.

1

Create the initial Airflow configuration dictionary

Create a dictionary called airflow_config with these exact entries: "core": {"executor": "LocalExecutor", "sql_alchemy_conn": "sqlite:///airflow.db"} and "webserver": {"web_server_port": 8080}.

Apache Airflow

# Create the airflow_config dictionary with core and webserver sections
# Your code here

Hint

Use a dictionary with keys core and webserver. Each key maps to another dictionary with the specified settings.

2

Enable multiple schedulers for high availability

Add a new key "scheduler" to the airflow_config dictionary with the entry "num_runs": 0 to enable continuous running schedulers.

Apache Airflow

airflow_config = {
    "core": {
        "executor": "LocalExecutor",
        "sql_alchemy_conn": "sqlite:///airflow.db"
    },
    "webserver": {
        "web_server_port": 8080
    }
}
# Add scheduler configuration to airflow_config
# Your code here

Hint

Add a new dictionary entry for scheduler with num_runs set to 0 to keep schedulers running continuously.

3

Configure the webserver for multiple workers

Add the key "workers" with value 4 inside the "webserver" dictionary in airflow_config to allow multiple webserver workers.

Apache Airflow

airflow_config = {
    "core": {
        "executor": "LocalExecutor",
        "sql_alchemy_conn": "sqlite:///airflow.db"
    },
    "webserver": {
        "web_server_port": 8080
    },
    "scheduler": {
        "num_runs": 0
    }
}
# Add workers = 4 inside the webserver dictionary
# Your code here

Hint

Inside the webserver dictionary, add "workers": 4 to allow four webserver workers.

4

Print the final Airflow configuration

Write a print statement to display the airflow_config dictionary.

Apache Airflow

airflow_config = {
    "core": {
        "executor": "LocalExecutor",
        "sql_alchemy_conn": "sqlite:///airflow.db"
    },
    "webserver": {
        "web_server_port": 8080,
        "workers": 4
    },
    "scheduler": {
        "num_runs": 0
    }
}
# Print the airflow_config dictionary
# Your code here

Hint

Use print(airflow_config) to display the dictionary.