Default args and DAG parameters in Apache Airflow - Commands & Configuration

Introduction

When you create workflows in Airflow, you often want to set common settings for all tasks, like retries or start dates. Default arguments let you set these once and reuse them. DAG parameters help you define the workflow's schedule and behavior.

When you want all tasks in a workflow to retry on failure without setting retries for each task.

When you need to set a common start date for all tasks in a workflow.

When you want to define the schedule interval for running your workflow automatically.

When you want to control how many past runs Airflow keeps track of for your workflow.

When you want to pass parameters to your workflow to customize its behavior.

Config File - my_dag.py

my_dag.py

from airflow import DAG
from airflow.operators.bash import BashOperator
from datetime import datetime, timedelta

default_args = {
    'owner': 'airflow',
    'depends_on_past': False,
    'start_date': datetime(2024, 6, 1),
    'email_on_failure': False,
    'email_on_retry': False,
    'retries': 1,
    'retry_delay': timedelta(minutes=5),
}

dag = DAG(
    'example_default_args_dag',
    default_args=default_args,
    description='A simple DAG with default args',
    schedule_interval=timedelta(days=1),
    catchup=False,
    max_active_runs=1
)

t1 = BashOperator(
    task_id='print_date',
    bash_command='date',
    dag=dag
)

t2 = BashOperator(
    task_id='sleep',
    bash_command='sleep 5',
    dag=dag
)

t1 >> t2

default_args: Sets common task parameters like owner, retries, and start date.

DAG: Defines the workflow with a name, default args, schedule, and behavior controls.

BashOperator tasks: Simple tasks that run shell commands, inheriting default args.

Commands

Lists all DAGs currently available in Airflow to verify the new DAG is recognized.

Terminal

airflow dags list

Expected Output

example_default_args_dag

Triggers the DAG manually to run the workflow immediately.

Terminal

airflow dags trigger example_default_args_dag

Expected Output

Created <DagRun example_default_args_dag @ 2024-06-01T00:00:00+00:00: manual__2024-06-01T00:00:00+00:00, externally triggered: True>

Lists all tasks in the DAG to confirm the tasks defined with default args are present.

Terminal

airflow tasks list example_default_args_dag

Expected Output

print_date

sleep

Runs the 'print_date' task for the given date without scheduling, useful for testing task behavior with default args.

Terminal

airflow tasks test example_default_args_dag print_date 2024-06-01

Expected Output

[2024-06-01 00:00:00,000] {bash.py:123} INFO - Running command: date

[2024-06-01 00:00:00,100] {bash.py:130} INFO - Output:

Wed Jun  1 00:00:00 UTC 2024

[2024-06-01 00:00:00,200] {taskinstance.py:1234} INFO - Task succeeded

Key Concept

If you remember nothing else from this pattern, remember: default_args let you set common task settings once to keep your DAG clean and consistent.

Common Mistakes

Not setting a start_date in default_args or setting it to a future date.

Airflow will not run the DAG because it doesn't know when to start or waits for the start date to arrive.

Always set start_date to a past or current date to enable DAG scheduling.

Defining tasks without passing the DAG object or default_args.

Tasks won't be linked to the DAG and may not inherit default settings, causing errors or unexpected behavior.

Always pass the DAG object when creating tasks to ensure proper linkage and inheritance.

Setting retries to 0 or forgetting retry_delay in default_args.

Tasks won't retry on failure, which may cause workflow failures without attempts to recover.

Set retries to at least 1 and define retry_delay to control retry timing.

Summary

Define default_args to set common task parameters like retries and start date.

Create a DAG with default_args and schedule_interval to control workflow timing.

Define tasks linked to the DAG to inherit default settings and run commands.

Use airflow CLI commands to list, trigger, and test DAGs and tasks.

Practice

1. What is the main purpose of default_args in an Airflow DAG?

easy

A. To define the schedule interval of the DAG

B. To specify the Python version used by the DAG

C. To set common parameters like retries and owner for all tasks

D. To list all the tasks in the DAG

Solution

1. Step 1: Understand the role of default_args

   default_args is a dictionary used to set common settings for tasks, such as retries, owner, start date, etc.

2. Step 2: Differentiate from DAG parameters

   DAG parameters like schedule_interval control when the DAG runs, not task defaults.

3. Final Answer:

   To set common parameters like retries and owner for all tasks -> Option C

4. Quick Check:

   default_args = common task settings [OK]

Hint: default_args sets task defaults, not DAG schedule [OK]

Common Mistakes:

• Confusing default_args with DAG schedule_interval
• Thinking default_args lists tasks
• Assuming default_args sets Python version

2. Which of the following is the correct way to define default_args with retries set to 3 and owner as 'airflow'?

easy

A. default_args = {'retries': 3, 'owner': 'airflow'}

B. default_args = ['retries=3', 'owner=airflow']

C. default_args = (retries=3, owner='airflow')

D. default_args = retries:3, owner:'airflow'

Solution

1. Step 1: Recall Python dictionary syntax

   Python dictionaries use curly braces with key:value pairs, keys as strings and values properly quoted if strings.

2. Step 2: Check each option's syntax

   default_args = {'retries': 3, 'owner': 'airflow'} uses correct dictionary syntax. Options A, B, and D use invalid Python syntax for dictionaries.

3. Final Answer:

   default_args = {'retries': 3, 'owner': 'airflow'} -> Option A

4. Quick Check:

   Python dict = curly braces with key:value [OK]

Hint: default_args is a Python dict with keys and values [OK]

Common Mistakes:

• Using list or tuple instead of dict
• Missing quotes around string keys or values
• Using colon instead of equal sign inside dict

3. Given this DAG snippet:

default_args = {'owner': 'admin', 'retries': 2}
with DAG('example_dag', default_args=default_args, schedule_interval='@daily') as dag:
    task1 = BashOperator(task_id='task1', bash_command='echo Hello')
    task2 = BashOperator(task_id='task2', bash_command='echo World', retries=5)

What is the number of retries for task2?

medium

A. 5

B. 2

C. 0

D. Undefined, causes error

Solution

1. Step 1: Understand default_args and task-level override

   default_args sets retries=2 for all tasks unless overridden.

2. Step 2: Check task2 retries parameter

   task2 explicitly sets retries=5, which overrides the default.

3. Final Answer:

   5 -> Option A

4. Quick Check:

   Task-level retries override default_args [OK]

Hint: Task args override default_args if set [OK]

Common Mistakes:

• Assuming default_args always applies
• Thinking task2 inherits retries=2
• Confusing retries with schedule_interval

4. You wrote this DAG code:

default_args = {'owner': 'user', 'start_date': '2023-01-01'}
dag = DAG('my_dag', default_args=default_args, schedule_interval='@hourly')

But Airflow throws an error about start_date. What is the likely problem?

medium

A. owner cannot be 'user', must be 'airflow'

B. schedule_interval '@hourly' is invalid syntax

C. default_args must be passed as a list, not dict

D. start_date must be a datetime object, not a string

Solution

1. Step 1: Check start_date type requirement

   Airflow requires start_date to be a datetime object, not a string.

2. Step 2: Validate other parameters

   schedule_interval='@hourly' is valid, default_args is a dict, and owner can be any string.

3. Final Answer:

   start_date must be a datetime object, not a string -> Option D

4. Quick Check:

   start_date = datetime, not string [OK]

Hint: Use datetime objects for start_date in default_args [OK]

Common Mistakes:

• Passing start_date as string instead of datetime
• Misunderstanding schedule_interval format
• Thinking owner must be 'airflow'

5. You want a DAG to run every Monday at 9 AM starting from March 1, 2024, with tasks retrying twice on failure and owned by 'data_team'. Which default_args and DAG parameters are correct?

hard

A. default_args = {'owner': 'data_team', 'retries': 2, 'start_date': '2024-03-01 09:00'}\ndag = DAG('weekly_dag', default_args=default_args, schedule_interval='0 9 * * 1')

B. default_args = {'owner': 'data_team', 'retries': 2, 'start_date': datetime(2024, 3, 1)}\ndag = DAG('weekly_dag', default_args=default_args, schedule_interval='0 9 * * 1')

C. default_args = {'owner': 'data_team', 'retries': 2, 'start_date': datetime(2024, 3, 1, 9, 0)}\ndag = DAG('weekly_dag', default_args=default_args, schedule_interval='@daily')

D. default_args = {'owner': 'data_team', 'retries': 2, 'start_date': '2024-03-01 09:00'}\ndag = DAG('weekly_dag', default_args=default_args, schedule_interval='@weekly')

Solution

1. Step 1: Set start_date with datetime object

   Start date must be a datetime object; time can be set to midnight if not specified.

2. Step 2: Use correct cron for Monday 9 AM

   Cron '0 9 * * 1' means 9 AM every Monday, matching requirement.

3. Step 3: Check retries and owner in default_args

   Retries=2 and owner='data_team' are correctly set.

4. Final Answer:

   default_args with datetime(2024, 3, 1) and schedule_interval='0 9 * * 1' -> Option B

5. Quick Check:

   start_date datetime + cron for Monday 9 AM [OK]

Hint: Use cron '0 9 * * 1' for Monday 9 AM schedule [OK]

Common Mistakes:

• Using string for start_date
• Using '@weekly' which runs Sunday midnight
• Setting schedule_interval to '@daily' instead of Monday