DAG concept (Directed Acyclic Graph) in Apache Airflow - Step-by-Step Execution

Process Flow - DAG concept (Directed Acyclic Graph)

Start DAG Definition

↓

Define Tasks (Nodes)

↓

Set Dependencies (Edges)

↓

Check for Cycles?

Yes→Error: Cycle Detected

No↓

Schedule DAG Run

↓

Execute Tasks in Order

↓

DAG Run Complete

This flow shows how a DAG is defined with tasks and dependencies, checked for cycles, then scheduled and executed in order.

Execution Sample

Apache Airflow

from airflow import DAG
from airflow.operators.dummy import DummyOperator
from datetime import datetime

dag = DAG('example_dag', start_date=datetime(2024,1,1))

t1 = DummyOperator(task_id='task1', dag=dag)
t2 = DummyOperator(task_id='task2', dag=dag)
t3 = DummyOperator(task_id='task3', dag=dag)

t1 >> t2 >> t3

Defines a DAG with three tasks where task1 runs before task2, which runs before task3.

Process Table

Step	Action	DAG State	Dependencies	Notes
1	Create DAG object	DAG 'example_dag' created	No dependencies yet	DAG initialized with start_date 2024-01-01
2	Create task1	Tasks: task1	None	Task1 added to DAG
3	Create task2	Tasks: task1, task2	None	Task2 added to DAG
4	Create task3	Tasks: task1, task2, task3	None	Task3 added to DAG
5	Set dependency: task1 >> task2	Tasks: task1, task2, task3	task1 -> task2	task2 depends on task1
6	Set dependency: task2 >> task3	Tasks: task1, task2, task3	task1 -> task2 -> task3	task3 depends on task2
7	Check for cycles	No cycles detected	task1 -> task2 -> task3	DAG is acyclic
8	Schedule DAG run	DAG scheduled	task1 -> task2 -> task3	DAG ready to execute
9	Execute task1	task1 completed	task1 -> task2 -> task3	First task runs
10	Execute task2	task2 completed	task1 -> task2 -> task3	Second task runs after task1
11	Execute task3	task3 completed	task1 -> task2 -> task3	Final task runs after task2
12	DAG run complete	All tasks completed	task1 -> task2 -> task3	DAG execution finished

💡 All tasks executed in order, no cycles found, DAG run completed successfully

Status Tracker

Variable	Start	After Step 2	After Step 3	After Step 4	After Step 5	After Step 6	Final
DAG Object	None	Created	Created	Created	Created	Created	Created
Tasks	None	task1	task1, task2	task1, task2, task3	task1, task2, task3	task1, task2, task3	task1, task2, task3
Dependencies	None	None	None	None	task1 -> task2	task1 -> task2 -> task3	task1 -> task2 -> task3

Key Moments - 3 Insights

Why can't a DAG have cycles in Airflow?

Because cycles would cause tasks to wait on each other forever, preventing the DAG from completing. The execution_table step 7 shows the cycle check that ensures the DAG is acyclic.

What does the '>>' operator do between tasks?

It sets the order of execution, making the left task a dependency of the right task. See execution_table steps 5 and 6 where dependencies are set using '>>'.

Can tasks run in parallel in a DAG?

Yes, tasks without dependencies between them can run at the same time. In this example, tasks are chained, so they run sequentially, but if tasks had no dependencies, they could run in parallel.

Visual Quiz - 3 Questions

Look at the execution table, what is the dependency state after step 5?

A task2 -> task3

B task1 -> task2

C task1 -> task3

D No dependencies

💡 Hint

Check the 'Dependencies' column in execution_table row with Step 5

At which step does the DAG confirm it has no cycles?

A Step 7

B Step 8

C Step 6

D Step 9

💡 Hint

Look for the step labeled 'Check for cycles' in the execution_table

If task2 had no dependency on task1, how would the execution order change?

A task3 would run before task2

B task2 would run before task1

C task1 and task2 could run in parallel

D No change, tasks run sequentially

💡 Hint

Refer to key_moments about parallel execution and dependencies

Concept Snapshot

DAG (Directed Acyclic Graph) in Airflow defines tasks and their order.
Tasks are nodes; dependencies are edges.
Cycles are not allowed to prevent infinite waits.
Use '>>' to set task order (e.g., task1 >> task2).
Airflow schedules and runs tasks respecting dependencies.
Tasks without dependencies can run in parallel.

Full Transcript

A DAG in Airflow is a set of tasks connected by dependencies that form a directed graph with no cycles. We start by creating a DAG object, then define tasks as nodes. Dependencies between tasks are set using operators like '>>' to specify order. Airflow checks the DAG to ensure no cycles exist, because cycles would cause tasks to wait forever. Once validated, the DAG is scheduled and tasks execute in order, respecting dependencies. Tasks without dependencies can run at the same time. This example shows three tasks where task1 runs first, then task2, then task3, completing the DAG run successfully.