What is Apache Airflow - Visual Explanation

Process Flow - What is Apache Airflow

Start Airflow Scheduler

↓

Scheduler checks DAGs

↓

Identify tasks to run

↓

Trigger task execution

↓

Workers run tasks

↓

Tasks update status

↓

Scheduler monitors progress

↓

Repeat for next schedule

Airflow runs by scheduling workflows (DAGs), triggering tasks, and monitoring their execution continuously.

Execution Sample

Apache Airflow

from airflow import DAG
from airflow.operators.bash import BashOperator
from datetime import datetime

dag = DAG('hello_world', start_date=datetime(2024,1,1))
hello = BashOperator(task_id='say_hello', bash_command='echo Hello Airflow', dag=dag)

Defines a simple Airflow workflow that runs a bash command to print 'Hello Airflow'.

Process Table

Step	Action	Evaluation	Result
1	Scheduler starts and reads DAG file	DAG 'hello_world' found	DAG registered in Airflow
2	Scheduler checks if task 'say_hello' should run	Start date reached	Task scheduled for execution
3	Worker picks up 'say_hello' task	Runs bash command	Outputs 'Hello Airflow'
4	Task completes	Success status recorded	Task marked as done
5	Scheduler waits for next run	No new tasks ready	Idle until next schedule

💡 No more tasks to run at this time; scheduler waits for next scheduled run.

Status Tracker

Variable	Start	After Step 1	After Step 2	After Step 3	After Step 4	Final
DAG 'hello_world'	Not loaded	Loaded	Loaded	Loaded	Loaded	Loaded
Task 'say_hello' status	None	None	Scheduled	Running	Success	Success

Key Moments - 2 Insights

Why does the scheduler keep running even after tasks complete?

The scheduler continuously monitors DAGs to trigger tasks at their scheduled times, as shown in step 5 where it waits for the next run.

What happens if the start_date of a DAG is in the future?

The scheduler will not schedule tasks until the start_date is reached, so no tasks run before that time (see step 2 evaluation).

Visual Quiz - 3 Questions

Look at the execution table, what is the status of the task 'say_hello' after step 3?

A Scheduled

B Running

C Success

D None

💡 Hint

Check the 'Task status' in variable_tracker after Step 3.

At which step does the scheduler mark the task as completed successfully?

A Step 4

B Step 3

C Step 2

D Step 5

💡 Hint

Look for 'Task completes' and 'Success status recorded' in execution_table.

If the DAG start_date was set to a future date, what would change in the execution table?

A Task would be scheduled immediately

B Scheduler would skip loading the DAG

C Task would not be scheduled until start_date is reached

D Task would run twice

💡 Hint

Refer to step 2 evaluation about start_date condition.

Concept Snapshot

Apache Airflow is a tool to schedule and run workflows called DAGs.
It uses a scheduler to trigger tasks at set times.
Workers execute tasks and report status.
The scheduler keeps running to manage ongoing workflows.
DAGs define tasks and their order.
Start_date controls when tasks begin running.

Full Transcript

Apache Airflow is a system that helps you run and manage workflows automatically. It uses a scheduler that reads workflow definitions called DAGs. The scheduler checks when tasks should run and tells workers to execute them. Workers run the tasks and report back if they succeeded or failed. The scheduler keeps running to handle tasks as they become ready. A simple example is a DAG that runs a bash command to print 'Hello Airflow'. The scheduler loads this DAG, schedules the task when the start date arrives, the worker runs the command, and the task completes successfully. The scheduler then waits for the next scheduled run. This continuous cycle allows Airflow to manage complex workflows reliably.

Practice

1. What is the main purpose of Apache Airflow?

easy

A. To monitor network traffic

B. To store large amounts of data

C. To create user interfaces for websites

D. To automate and schedule tasks in a specific order

Solution

1. Step 1: Understand Apache Airflow's role

   Apache Airflow is designed to automate and schedule tasks, not to store data or create interfaces.

2. Step 2: Identify the correct purpose

   It manages workflows by running tasks in a defined order automatically.

3. Final Answer:

   To automate and schedule tasks in a specific order -> Option D

4. Quick Check:

   Automation and scheduling = A [OK]

Hint: Airflow schedules and runs tasks automatically [OK]

Common Mistakes:

• Confusing Airflow with data storage tools
• Thinking Airflow builds websites
• Assuming Airflow monitors networks

2. Which of the following is the correct way to define a DAG in Apache Airflow using Python?

easy

A. dag = DAG('my_dag', schedule_interval='@daily')

B. dag = DAG('my_dag' schedule_interval='@daily')

C. dag = DAG['my_dag', schedule_interval='@daily']

D. dag = DAG{'my_dag', schedule_interval='@daily'}

Solution

1. Step 1: Check Python syntax for function calls

   Function calls use parentheses with arguments separated by commas.

2. Step 2: Identify correct syntax

   dag = DAG('my_dag', schedule_interval='@daily') uses parentheses and commas correctly; others have missing commas or wrong brackets.

3. Final Answer:

   dag = DAG('my_dag', schedule_interval='@daily') -> Option A

4. Quick Check:

   Function call syntax = A [OK]

Hint: Use parentheses and commas for function arguments [OK]

Common Mistakes:

• Missing commas between arguments
• Using square or curly brackets instead of parentheses
• Incorrect argument separation

3. Given the following Airflow DAG snippet, what will be the order of task execution?

from airflow import DAG
from airflow.operators.dummy import DummyOperator
from datetime import datetime

dag = DAG('example_dag', start_date=datetime(2023,1,1))

t1 = DummyOperator(task_id='task1', dag=dag)
t2 = DummyOperator(task_id='task2', dag=dag)
t3 = DummyOperator(task_id='task3', dag=dag)

t1 >> t2 >> t3

medium

A. All tasks run at the same time

B. task3 runs, then task2, then task1

C. task1 runs, then task2, then task3

D. task2 runs first, then task1, then task3

Solution

1. Step 1: Understand the '>>' operator in Airflow

   The '>>' operator sets task dependencies, meaning left task runs before right task.

2. Step 2: Trace the task order

   t1 >> t2 >> t3 means task1 runs first, then task2, then task3.

3. Final Answer:

   task1 runs, then task2, then task3 -> Option C

4. Quick Check:

   Dependency chain = task1 -> task2 -> task3 [OK]

Hint: The '>>' operator means 'runs before' [OK]

Common Mistakes:

• Thinking tasks run in reverse order
• Assuming tasks run in parallel without dependencies
• Misreading the '>>' operator

4. You wrote this Airflow DAG code but it fails to run:

from airflow import DAG
from airflow.operators.dummy import DummyOperator
from datetime import datetime

dag = DAG('my_dag', start_date=datetime(2023,1,1))

t1 = DummyOperator(task_id='task1')
t2 = DummyOperator(task_id='task2')

t1 >> t2

What is the error?

medium

A. Task IDs must be integers

B. Missing DAG parameter in DummyOperator tasks

C. start_date must be a string

D. Wrong import for DummyOperator

Solution

1. Step 1: Check task definitions

   Tasks need to be linked to a DAG by passing the 'dag' parameter.

2. Step 2: Identify missing 'dag' argument

   t1 and t2 are missing 'dag=dag', so Airflow doesn't know which DAG they belong to.

3. Final Answer:

   Missing DAG parameter in DummyOperator tasks -> Option B

4. Quick Check:

   Tasks need DAG assigned = D [OK]

Hint: Always assign tasks to a DAG with dag=dag [OK]

Common Mistakes:

• Forgetting to pass dag=dag to tasks
• Incorrect imports causing errors
• Wrong data types for start_date or task_id

5. You want to create an Airflow workflow that retries a task up to 3 times if it fails. Which DAG task parameter should you set?

hard

A. retries=3

B. retry_count=3

C. max_retry=3

D. retry_limit=3

Solution

1. Step 1: Recall Airflow retry parameter

   Airflow uses 'retries' to specify how many times to retry a failed task.

2. Step 2: Match parameter name

   The correct parameter is 'retries', not retry_count, max_retry, or retry_limit.

3. Final Answer:

   retries=3 -> Option A

4. Quick Check:

   Retries parameter controls retry count = B [OK]

Hint: Use 'retries' to set retry attempts in Airflow tasks [OK]

Common Mistakes:

• Using incorrect parameter names like retry_count
• Confusing retry count with timeout
• Not setting retries at all