Airflow architecture (scheduler, webserver, executor, metadata DB) - Step-by-Step Execution

Process Flow - Airflow architecture (scheduler, webserver, executor, metadata DB)

User submits DAG

↓

Webserver receives DAG

↓

Scheduler scans DAGs

↓

Scheduler triggers tasks

↓

Executor runs tasks

↓

Tasks update Metadata DB

↓

Webserver reads Metadata DB for UI

This flow shows how Airflow components interact: user submits DAGs to the webserver, scheduler scans and triggers tasks, executor runs them, and metadata DB stores state for the webserver UI.

Execution Sample

Apache Airflow

# Simplified Airflow flow
webserver.start()
scheduler.scan_dags()
scheduler.trigger_tasks()
executor.run_tasks()
metadata_db.update_state()
webserver.show_ui()

This code simulates the main Airflow components working together to run workflows and update the UI.

Process Table

Step	Component	Action	Result/State Change
1	User	Submits DAG file	DAG file available to Webserver and Scheduler
2	Webserver	Receives DAG and serves UI	DAG visible in Airflow UI
3	Scheduler	Scans DAG files	Identifies tasks to schedule
4	Scheduler	Triggers task instances	Tasks queued for execution
5	Executor	Runs tasks	Tasks execute on workers
6	Tasks	Update Metadata DB	Task states updated (running, success, failed)
7	Webserver	Reads Metadata DB	UI shows updated task status
8	Scheduler	Repeats scanning and triggering	Continuous workflow management
9	Exit	No new tasks to schedule	Waits for next cycle

💡 Scheduler waits when no tasks are ready; cycle repeats continuously

Status Tracker

Variable	Start	After Step 3	After Step 4	After Step 6	Final
DAG files	Empty	Loaded	Loaded	Loaded	Loaded
Task queue	Empty	Empty	Queued	Running/Completed	Empty or updated
Metadata DB	Empty	Empty	Empty	Updated with task states	Updated
UI display	Empty	Shows DAGs	Shows DAGs	Shows running tasks	Shows updated task states

Key Moments - 3 Insights

Why does the scheduler scan DAG files repeatedly?

The scheduler scans DAG files continuously (see Step 3 and Step 8) to detect new or updated workflows and to trigger tasks as needed.

How does the webserver know the current status of tasks?

The webserver reads the Metadata DB (Step 7) where task states are updated by running tasks, so it can show accurate status in the UI.

What role does the executor play in task execution?

The executor runs the actual tasks on workers (Step 5), changing their state which is then recorded in the Metadata DB.

Visual Quiz - 3 Questions

Look at the execution table, at which step does the scheduler trigger tasks?

A Step 3

B Step 5

C Step 4

D Step 7

💡 Hint

Check the 'Action' column for when tasks are queued by the scheduler.

According to the variable tracker, what is the state of the task queue after Step 6?

A Running/Completed

B Empty

C Queued

D Loaded

💡 Hint

Look at the 'Task queue' row under 'After Step 6' in the variable tracker.

If the Metadata DB did not update after task execution, what would the webserver show?

A No DAGs available

B Outdated or no task status

C Updated task status

D Scheduler status

💡 Hint

Refer to Step 7 where the webserver reads the Metadata DB for UI updates.

Concept Snapshot

Airflow architecture:
- Webserver: serves UI and DAGs
- Scheduler: scans DAGs, triggers tasks
- Executor: runs tasks on workers
- Metadata DB: stores task and DAG states
Scheduler and executor work continuously to manage workflows.

Full Transcript

Airflow architecture involves four main parts working together. The user submits DAG files which the webserver receives and shows in the UI. The scheduler scans these DAGs repeatedly to find tasks to run. When tasks are ready, the scheduler triggers them and the executor runs them on workers. As tasks run, they update their status in the metadata database. The webserver reads this database to show the current status of workflows and tasks. This cycle repeats continuously to manage workflows smoothly.

Practice

1. Which Airflow component is responsible for deciding when and which tasks to run?

easy

A. Scheduler

B. Webserver

C. Executor

D. Metadata Database

Solution

1. Step 1: Understand the role of the scheduler

   The scheduler checks the workflow definitions and decides the timing and order of task execution.

2. Step 2: Differentiate from other components

   The executor runs tasks, the webserver shows status, and the metadata DB stores info. Only the scheduler plans tasks.

3. Final Answer:

   Scheduler -> Option A

4. Quick Check:

   Task planning = Scheduler [OK]

Hint: Scheduler plans tasks; executor runs them [OK]

Common Mistakes:

• Confusing executor with scheduler
• Thinking webserver runs tasks
• Assuming metadata DB schedules tasks

2. Which of the following is the correct way to start the Airflow webserver using the CLI?

easy

A. airflow webserver -p

B. airflow webserver start

C. airflow start webserver

D. airflow webserver

Solution

1. Step 1: Recall the Airflow CLI command for webserver

   The correct command to start the webserver is simply airflow webserver.

2. Step 2: Analyze other options

   airflow webserver start adds an unnecessary 'start', airflow start webserver reverses the order, and airflow webserver -p adds a port flag which is valid but incomplete without specifying the port number explicitly.

3. Final Answer:

   airflow webserver -> Option D

4. Quick Check:

   Start webserver = airflow webserver [OK]

Hint: Use 'airflow webserver' to start the UI [OK]

Common Mistakes:

• Adding extra words like 'start'
• Reversing command order
• Incorrect or incomplete flags

3. Given the following Airflow setup: The scheduler triggers a task, the executor runs it, and the metadata DB records the status. What will happen if the metadata DB is down?

medium

A. Tasks will run but status won't be saved

B. Scheduler will stop scheduling tasks

C. Webserver will crash immediately

D. Executor will ignore tasks

Solution

1. Step 1: Understand metadata DB role

   The metadata DB stores task states and workflow info. If it is down, Airflow components cannot read or write task status.

2. Step 2: Impact on scheduler and executor

   The scheduler depends on the metadata DB to know which tasks to schedule. Without DB access, it cannot schedule new tasks. The executor waits for tasks from the scheduler.

3. Final Answer:

   Scheduler will stop scheduling tasks -> Option B

4. Quick Check:

   Metadata DB down stops scheduler [OK]

Hint: Metadata DB failure stops scheduler [OK]

Common Mistakes:

• Thinking tasks run without DB
• Assuming webserver crashes immediately
• Believing executor ignores tasks

4. You notice that tasks are not running even though the webserver shows them as scheduled. Which component should you check first to debug this issue?

medium

A. Executor

B. Scheduler

C. Metadata Database

D. Webserver

Solution

1. Step 1: Analyze symptoms

   Tasks are marked scheduled in the webserver UI but are not running. This means scheduling is done but execution is not happening.

2. Step 2: Identify responsible component

   The executor runs the tasks. If tasks are scheduled but not running, the executor might be stuck or misconfigured.

3. Final Answer:

   Executor -> Option A

4. Quick Check:

   Tasks not running? Check executor [OK]

Hint: Scheduled but not running? Check executor first [OK]

Common Mistakes:

• Checking scheduler when scheduling works
• Restarting webserver unnecessarily
• Ignoring executor logs

5. In a distributed Airflow setup, which component must be shared and accessible by both the scheduler and executor to coordinate task execution properly?

hard

A. Webserver

B. Local task logs

C. Metadata Database

D. Scheduler's local cache

Solution

1. Step 1: Understand distributed setup needs

   In distributed Airflow, multiple executors and schedulers may run on different machines.

2. Step 2: Identify shared resource for coordination

   The metadata database stores task states and is the central point both scheduler and executor use to coordinate which tasks to run and their status.

3. Step 3: Exclude other options

   The webserver is for UI only, local logs are not shared, and scheduler's local cache is not accessible by executors.

4. Final Answer:

   Metadata Database -> Option C

5. Quick Check:

   Shared coordination = Metadata DB [OK]

Hint: Metadata DB is central shared store in distributed Airflow [OK]

Common Mistakes:

• Thinking webserver shares task state
• Assuming local logs are shared
• Believing scheduler cache syncs executors