Airflow architecture (scheduler, webserver, executor, metadata DB) - Mini Project: Build & Apply

Understanding Airflow Architecture Components

📖 Scenario: You are working in a team that uses Apache Airflow to automate workflows. To better understand how Airflow works, you want to explore its main components: the scheduler, webserver, executor, and metadata database.

🎯 Goal: Build a simple Python script that defines the main Airflow components as variables, configures a basic executor type, and prints out the Airflow architecture setup summary.

📋 What You'll Learn

Create variables for Airflow components with exact names and values

Add a configuration variable for the executor type

Use a dictionary to represent the Airflow architecture setup

Print the final Airflow architecture dictionary

💡 Why This Matters

🌍 Real World

Understanding Airflow's architecture helps teams configure and troubleshoot workflow automation systems effectively.

💼 Career

Knowledge of Airflow components is essential for roles like DevOps engineers, data engineers, and workflow automation specialists.

1

Define Airflow core components

Create four string variables with these exact names and values: scheduler set to "Airflow Scheduler", webserver set to "Airflow Webserver", executor set to "SequentialExecutor", and metadata_db set to "PostgreSQL Metadata Database".

Apache Airflow

# Your code here

Need a hint?

Use simple assignment statements to create variables with the exact names and values.

2

Add executor configuration

Create a string variable called executor_type and set it to "LocalExecutor" to represent a different executor configuration.

Apache Airflow

scheduler = "Airflow Scheduler"
webserver = "Airflow Webserver"
executor = "SequentialExecutor"
metadata_db = "PostgreSQL Metadata Database"
# Your code here

Need a hint?

Just add one more variable assignment for executor_type.

3

Create Airflow architecture dictionary

Create a dictionary called airflow_architecture with keys "Scheduler", "Webserver", "Executor", and "Metadata DB". Assign the variables scheduler, webserver, executor_type, and metadata_db as their respective values.

Apache Airflow

scheduler = "Airflow Scheduler"
webserver = "Airflow Webserver"
executor = "SequentialExecutor"
metadata_db = "PostgreSQL Metadata Database"
executor_type = "LocalExecutor"
# Your code here

Need a hint?

Use a dictionary literal with the exact keys and assign the existing variables as values.

4

Print Airflow architecture summary

Write a print statement to display the airflow_architecture dictionary.

Apache Airflow

scheduler = "Airflow Scheduler"
webserver = "Airflow Webserver"
executor = "SequentialExecutor"
metadata_db = "PostgreSQL Metadata Database"
executor_type = "LocalExecutor"
airflow_architecture = {
    "Scheduler": scheduler,
    "Webserver": webserver,
    "Executor": executor_type,
    "Metadata DB": metadata_db
}
# Your code here

Need a hint?

Use print(airflow_architecture) to show the dictionary content.