Airflow architecture (scheduler, webserver, executor, metadata DB) - Commands & Configuration

Introduction

Airflow helps you run and manage tasks automatically. It uses parts like the scheduler to plan tasks, the webserver to show status, the executor to run tasks, and a database to keep track of everything.

When you want to run data processing jobs on a schedule without doing it manually

When you need to see the progress and results of your tasks in a web browser

When you want to run many tasks in parallel efficiently

When you want to keep a record of all task runs and their status

When you want to retry failed tasks automatically

Config File - airflow.cfg

airflow.cfg

[core]
executor = LocalExecutor
sql_alchemy_conn = sqlite:///airflow.db

[scheduler]
job_heartbeat_sec = 5
scheduler_heartbeat_sec = 5

[webserver]
web_server_port = 8080

[database]
load_examples = False

This configuration file sets up Airflow's main parts:

• executor: decides how tasks run; here it runs locally in parallel.
• sql_alchemy_conn: connection to the metadata database; here it uses a local SQLite file.
• scheduler: controls how often the scheduler checks for new tasks.
• webserver: sets the port for the web interface.
• database: controls database-related settings.

Commands

This command creates the metadata database and tables to store task and workflow information.

Terminal

airflow db init

Expected Output

INFO  [alembic.runtime.migration] Context impl SQLiteImpl.

INFO  [alembic.runtime.migration] Will assume non-transactional DDL.

INFO  [alembic.runtime.migration] Running upgrade  -> head

INFO  [alembic.runtime.migration] Upgrade done.

Starts the scheduler which looks for tasks to run and sends them to the executor.

Terminal

airflow scheduler

Expected Output

[2024-06-01 12:00:00,000] {scheduler_job.py:123} INFO - Starting the scheduler

[2024-06-01 12:00:05,000] {scheduler_job.py:456} INFO - Scheduler heartbeat

Starts the webserver so you can open the Airflow UI in your browser on port 8080.

Terminal

airflow webserver -p 8080

Expected Output

[2024-06-01 12:00:00,000] {webserver.py:78} INFO - Starting web server on port 8080

[2024-06-01 12:00:00,100] {webserver.py:90} INFO - Webserver started

→

-p - Sets the port number for the webserver

Lists all tasks in the DAG named 'example_dag' to check what tasks are available to run.

Terminal

airflow tasks list example_dag

Expected Output

task_1

task_2

task_3

Key Concept

If you remember nothing else from this pattern, remember: Airflow uses the scheduler to plan tasks, the executor to run them, the webserver to show status, and the metadata database to keep track of everything.

Common Mistakes

Not initializing the metadata database before starting Airflow components

Airflow needs the database to store task info; without it, components fail to start or work properly

Always run 'airflow db init' before starting the scheduler or webserver

Running the webserver on a port already in use

The webserver will fail to start and show an error if the port is busy

Choose a free port with '-p' flag or stop the process using the port

Using SequentialExecutor in production expecting parallel task runs

SequentialExecutor runs tasks one by one, so parallelism is not possible

Use LocalExecutor or CeleryExecutor for parallel task execution

Summary

Initialize the metadata database with 'airflow db init' to prepare Airflow for use.

Start the scheduler to plan and send tasks to run automatically.

Run the webserver to access the Airflow UI and monitor tasks.

Use the metadata database to keep track of task states and history.

Practice

1. Which Airflow component is responsible for deciding when and which tasks to run?

easy

A. Scheduler

B. Webserver

C. Executor

D. Metadata Database

Solution

1. Step 1: Understand the role of the scheduler

   The scheduler checks the workflow definitions and decides the timing and order of task execution.

2. Step 2: Differentiate from other components

   The executor runs tasks, the webserver shows status, and the metadata DB stores info. Only the scheduler plans tasks.

3. Final Answer:

   Scheduler -> Option A

4. Quick Check:

   Task planning = Scheduler [OK]

Hint: Scheduler plans tasks; executor runs them [OK]

Common Mistakes:

• Confusing executor with scheduler
• Thinking webserver runs tasks
• Assuming metadata DB schedules tasks

2. Which of the following is the correct way to start the Airflow webserver using the CLI?

easy

A. airflow webserver -p

B. airflow webserver start

C. airflow start webserver

D. airflow webserver

Solution

1. Step 1: Recall the Airflow CLI command for webserver

   The correct command to start the webserver is simply airflow webserver.

2. Step 2: Analyze other options

   airflow webserver start adds an unnecessary 'start', airflow start webserver reverses the order, and airflow webserver -p adds a port flag which is valid but incomplete without specifying the port number explicitly.

3. Final Answer:

   airflow webserver -> Option D

4. Quick Check:

   Start webserver = airflow webserver [OK]

Hint: Use 'airflow webserver' to start the UI [OK]

Common Mistakes:

• Adding extra words like 'start'
• Reversing command order
• Incorrect or incomplete flags

3. Given the following Airflow setup: The scheduler triggers a task, the executor runs it, and the metadata DB records the status. What will happen if the metadata DB is down?

medium

A. Tasks will run but status won't be saved

B. Scheduler will stop scheduling tasks

C. Webserver will crash immediately

D. Executor will ignore tasks

Solution

1. Step 1: Understand metadata DB role

   The metadata DB stores task states and workflow info. If it is down, Airflow components cannot read or write task status.

2. Step 2: Impact on scheduler and executor

   The scheduler depends on the metadata DB to know which tasks to schedule. Without DB access, it cannot schedule new tasks. The executor waits for tasks from the scheduler.

3. Final Answer:

   Scheduler will stop scheduling tasks -> Option B

4. Quick Check:

   Metadata DB down stops scheduler [OK]

Hint: Metadata DB failure stops scheduler [OK]

Common Mistakes:

• Thinking tasks run without DB
• Assuming webserver crashes immediately
• Believing executor ignores tasks

4. You notice that tasks are not running even though the webserver shows them as scheduled. Which component should you check first to debug this issue?

medium

A. Executor

B. Scheduler

C. Metadata Database

D. Webserver

Solution

1. Step 1: Analyze symptoms

   Tasks are marked scheduled in the webserver UI but are not running. This means scheduling is done but execution is not happening.

2. Step 2: Identify responsible component

   The executor runs the tasks. If tasks are scheduled but not running, the executor might be stuck or misconfigured.

3. Final Answer:

   Executor -> Option A

4. Quick Check:

   Tasks not running? Check executor [OK]

Hint: Scheduled but not running? Check executor first [OK]

Common Mistakes:

• Checking scheduler when scheduling works
• Restarting webserver unnecessarily
• Ignoring executor logs

5. In a distributed Airflow setup, which component must be shared and accessible by both the scheduler and executor to coordinate task execution properly?

hard

A. Webserver

B. Local task logs

C. Metadata Database

D. Scheduler's local cache

Solution

1. Step 1: Understand distributed setup needs

   In distributed Airflow, multiple executors and schedulers may run on different machines.

2. Step 2: Identify shared resource for coordination

   The metadata database stores task states and is the central point both scheduler and executor use to coordinate which tasks to run and their status.

3. Step 3: Exclude other options

   The webserver is for UI only, local logs are not shared, and scheduler's local cache is not accessible by executors.

4. Final Answer:

   Metadata Database -> Option C

5. Quick Check:

   Shared coordination = Metadata DB [OK]

Hint: Metadata DB is central shared store in distributed Airflow [OK]

Common Mistakes:

• Thinking webserver shares task state
• Assuming local logs are shared
• Believing scheduler cache syncs executors