Creating a basic DAG file in Apache Airflow - Visual Walkthrough

Process Flow - Creating a basic DAG file

Import Airflow modules

↓

Define default_args dict

↓

Create DAG object with args

↓

Define tasks as Operators

↓

Set task dependencies

↓

Save DAG file in airflow/dags folder

This flow shows the steps to create a DAG file: import modules, set default arguments, create the DAG, define tasks, set their order, and save the file.

Execution Sample

Apache Airflow

from airflow import DAG
from airflow.providers.bash.operators.bash import BashOperator
from datetime import datetime

with DAG('my_first_dag', start_date=datetime(2024, 1, 1), schedule_interval='@daily') as dag:
    task1 = BashOperator(task_id='print_date', bash_command='date')

This code creates a DAG named 'my_first_dag' that runs daily and has one task that prints the date.

Process Table

Step	Action	Evaluation	Result
1	Import airflow modules	Modules imported	DAG, BashOperator available
2	Define DAG with name 'my_first_dag', start_date 2024-01-01, schedule '@daily'	DAG object created	DAG 'my_first_dag' ready
3	Define task1 as BashOperator with task_id 'print_date' and command 'date'	Task created	Task 'print_date' ready
4	Set task dependencies (none here)	No dependencies set	Single task DAG
5	Save file in airflow/dags folder	File saved	DAG available to Airflow scheduler

💡 All steps completed; DAG file is ready and recognized by Airflow

Status Tracker

Variable	Start	After Step 2	After Step 3	Final
dag	None	DAG object with name 'my_first_dag'	Same DAG object	Same DAG object
task1	None	None	BashOperator task 'print_date'	Same task object

Key Moments - 3 Insights

Why do we need to import specific modules like DAG and BashOperator?

Because these modules provide the classes and functions needed to create and define the DAG and its tasks, as shown in execution_table step 1.

What does the 'with DAG(...) as dag:' syntax do?

It creates the DAG object and allows tasks defined inside the block to be automatically associated with this DAG, as seen in execution_table step 2 and 3.

Why is setting task dependencies important even if not shown here?

Dependencies define the order tasks run. Without them, tasks run independently. This example has one task, so no dependencies are needed, as noted in step 4.

Visual Quiz - 3 Questions

Look at the execution table, what is the state of 'task1' after step 3?

A Task 'print_date' not created yet

B Task 'print_date' deleted

C Task 'print_date' created as BashOperator

D Task 'print_date' running

💡 Hint

Check the 'Result' column in row for step 3 in execution_table

At which step is the DAG object created?

A Step 1

B Step 2

C Step 3

D Step 4

💡 Hint

Look at the 'Action' and 'Result' columns in execution_table for step 2

If you add a second task and set it to run after 'task1', which step would change?

A Step 4 - Set task dependencies

B Step 2 - Define DAG

C Step 1 - Import modules

D Step 5 - Save file

💡 Hint

Setting task order is done in step 4 according to execution_table

Concept Snapshot

Creating a basic Airflow DAG file:
1. Import DAG and Operators
2. Define DAG with name, start_date, schedule
3. Define tasks inside DAG context
4. Set task dependencies
5. Save file in airflow/dags folder
Airflow scheduler detects and runs the DAG

Full Transcript

To create a basic Airflow DAG file, first import the necessary modules like DAG and BashOperator. Then define the DAG object with a name, start date, and schedule interval. Inside the DAG context, define tasks such as BashOperator tasks. Optionally set dependencies between tasks to control execution order. Finally, save the file in the airflow/dags folder so Airflow can find and run it. This process ensures your workflow is automated and scheduled.

Practice

1. What is the main purpose of a DAG file in Apache Airflow?

easy

A. To configure the Airflow webserver settings

B. To store logs generated by tasks

C. To install Python packages required by tasks

D. To define tasks and their schedule for workflow automation

Solution

1. Step 1: Understand what a DAG file represents

   A DAG file in Airflow is a Python script that defines a Directed Acyclic Graph, which represents tasks and their dependencies.

2. Step 2: Identify the role of the DAG file

   The DAG file schedules and organizes tasks to automate workflows, not for logging, configuration, or package installation.

3. Final Answer:

   To define tasks and their schedule for workflow automation -> Option D

4. Quick Check:

   DAG file purpose = Define tasks and schedule [OK]

Hint: DAG files define workflows, not configs or logs [OK]

Common Mistakes:

• Confusing DAG file with Airflow config files
• Thinking DAG files store logs
• Assuming DAG files install packages

2. Which of the following is the correct way to import the DAG class in an Airflow DAG file?

easy

A. import DAG from airflow.models

B. import airflow.DAG

C. from airflow.models import DAG

D. from airflow import DAG

Solution

1. Step 1: Recall the correct import syntax in Python

   Python imports use 'from module import class' syntax for specific classes.

2. Step 2: Match with Airflow's module structure

   The recommended import used in official Airflow documentation is 'from airflow.models import DAG'.

3. Final Answer:

   from airflow.models import DAG -> Option C

4. Quick Check:

   Correct import syntax = from airflow.models import DAG [OK]

Hint: Use 'from airflow.models import DAG' to import DAG class [OK]

Common Mistakes:

• Using incorrect import syntax
• Trying to import DAG directly from airflow
• Using invalid Python import statements

3. Given the following DAG snippet, what will be the task_id of the BashOperator task?

from airflow import DAG
from airflow.operators.bash import BashOperator
from datetime import datetime

default_args = {'start_date': datetime(2024, 1, 1)}
with DAG('example_dag', default_args=default_args, schedule_interval='@daily') as dag:
    task = BashOperator(task_id='print_date', bash_command='date')

medium

A. print_date

B. task

C. example_dag

D. bash_command

Solution

1. Step 1: Identify the task_id parameter in BashOperator

   The BashOperator is created with task_id='print_date', which names the task.

2. Step 2: Understand task_id usage

   task_id is the unique identifier for the task in the DAG, so the task's id is 'print_date'.

3. Final Answer:

   print_date -> Option A

4. Quick Check:

   task_id value = print_date [OK]

Hint: Look for task_id argument in operator constructor [OK]

Common Mistakes:

• Confusing variable name with task_id
• Using DAG id as task_id
• Mistaking bash_command as task_id

4. Identify the error in this DAG file snippet:

from airflow import DAG
from airflow.operators.bash import BashOperator
from datetime import datetime

with DAG('my_dag', start_date=datetime(2024, 1, 1), schedule_interval='@daily') as dag:
    task1 = BashOperator(task_id='task1', bash_command='echo Hello')
    task2 = BashOperator(task_id='task2', bash_command='echo World')
    task1 >> task2

print('DAG created')

medium

A. No error, DAG is correctly defined

B. Missing default_args dictionary for start_date

C. Incorrect import of BashOperator

D. Using print statement inside DAG context

Solution

1. Step 1: Check imports and DAG definition

   Imports are correct, DAG is created with start_date and schedule_interval directly in constructor.

2. Step 2: Verify task creation and dependencies

   Tasks are created with valid task_ids and bash commands, and dependency is set with 'task1 >> task2'.

3. Step 3: Confirm print statement usage

   Print outside DAG context is allowed and does not affect DAG creation.

4. Final Answer:

   No error, DAG is correctly defined -> Option A

5. Quick Check:

   DAG syntax correct = No error [OK]

Hint: Check if DAG and tasks follow Airflow syntax rules [OK]

Common Mistakes:

• Assuming default_args is mandatory
• Thinking print inside DAG causes error
• Confusing import paths for operators

5. You want to create a DAG that runs a task every Monday at 7 AM starting from March 1, 2024. Which schedule_interval and start_date combination is correct in the DAG definition?

hard

A. schedule_interval='@weekly', start_date=datetime(2024, 3, 4)

B. schedule_interval='0 7 * * 1', start_date=datetime(2024, 3, 1)

C. schedule_interval='@daily', start_date=datetime(2024, 3, 1, 7, 0)

D. schedule_interval='0 7 * * 0', start_date=datetime(2024, 3, 1)

Solution

1. Step 1: Determine the cron expression for Monday 7 AM

   Cron '0 7 * * 1' runs every Monday at 7:00 AM (1=Monday).

2. Step 2: Validate start_date

   March 1, 2024 (Friday) is before the first Monday (March 4), so first run on March 4, 7 AM.

3. Step 3: Eliminate other options

   A: '@weekly'='0 0 * * 0' (Sunday midnight), wrong day/time. C: daily. D: Sunday (0).

4. Final Answer:

   schedule_interval='0 7 * * 1', start_date=datetime(2024, 3, 1) -> Option B

5. Quick Check:

   Monday 7AM cron = '0 7 * * 1' [OK]

Hint: Use cron '0 7 * * 1' for Mon 7AM [OK]

Common Mistakes:

• Setting start_date on wrong weekday
• Confusing cron day numbers (0=Sunday, 1=Monday)
• Using daily schedule for weekly task