Default args and DAG parameters in Apache Airflow - Step-by-Step Execution

Process Flow - Default args and DAG parameters

Define default_args dict

↓

Create DAG with default_args

↓

Define tasks using DAG

↓

Run DAG

↓

Tasks inherit default_args

↓

Override parameters if needed

↓

DAG executes with parameters

This flow shows how default arguments are set once and then used by all tasks in the DAG unless overridden.

Execution Sample

Apache Airflow

from datetime import datetime
from airflow import DAG

default_args = {
    'owner': 'airflow',
    'start_date': datetime(2024, 1, 1),
    'retries': 1
}

dag = DAG('example_dag', default_args=default_args, schedule_interval='@daily')

This code sets default arguments and creates a DAG that uses them for its tasks.

Process Table

Step	Action	Parameter Set	Value	Effect
1	Define default_args	owner	airflow	Sets task owner to airflow
2	Define default_args	start_date	2024-01-01	Tasks start from this date
3	Define default_args	retries	1	Tasks retry once on failure
4	Create DAG with default_args	default_args	default_args dict	DAG uses these defaults
5	Define task1 without override	owner	airflow	Inherits from default_args
6	Define task1 without override	retries	1	Inherits from default_args
7	Define task2 with override	retries	3	Overrides default retries
8	Run DAG	task1 retries	1	Task1 retries once
9	Run DAG	task2 retries	3	Task2 retries thrice
10	End	-	-	Execution complete

💡 All tasks run with parameters from default_args unless overridden explicitly.

Status Tracker

Variable	Start	After Step 5	After Step 7	Final
owner	undefined	airflow	airflow	airflow
start_date	undefined	2024-01-01	2024-01-01	2024-01-01
retries	undefined	1	3 (task2 override)	varies by task

Key Moments - 2 Insights

Why does task2 retry 3 times but task1 only once?

Because task2 explicitly overrides the 'retries' parameter (see execution_table step 7), while task1 uses the default from default_args (step 5).

What happens if a parameter is missing in default_args?

If a parameter is missing in default_args and not set in the task, Airflow uses its internal defaults or may raise an error depending on the parameter.

Visual Quiz - 3 Questions

Look at the execution table, what is the 'owner' value for task1 at step 5?

A admin

B airflow

C user

D undefined

💡 Hint

Check the 'Parameter Set' and 'Value' columns at step 5 in the execution_table.

At which step does task2 override the default 'retries' value?

A Step 5

B Step 6

C Step 7

D Step 8

💡 Hint

Look for the step mentioning 'Define task2 with override' in the execution_table.

If we remove 'retries' from default_args, what happens to task1 retries?

A Retries use Airflow default

B Retries become 3

C Retries become 0

D Task fails to run

💡 Hint

Refer to key_moments about missing parameters and Airflow's internal defaults.

Concept Snapshot

default_args is a dictionary of parameters set once for a DAG.
Tasks inherit these parameters unless overridden.
Common keys: owner, start_date, retries.
DAG uses default_args to apply consistent settings.
Overrides allow task-specific behavior.
This simplifies DAG management and reduces repetition.

Full Transcript

In Airflow, default_args is a dictionary that holds common parameters like owner, start_date, and retries. When you create a DAG, you pass default_args so all tasks inside inherit these settings automatically. This means you don't have to repeat the same parameters for every task. However, you can override any parameter for a specific task if needed. For example, if default_args sets retries to 1, but a task needs 3 retries, you override it in that task. This approach keeps your DAG clean and consistent. The execution table shows each step: defining default_args, creating the DAG, defining tasks with or without overrides, and running the DAG with the final parameters applied. Remember, if a parameter is missing in default_args and not set in a task, Airflow uses its own default values or may raise an error depending on the parameter.

Practice

1. What is the main purpose of default_args in an Airflow DAG?

easy

A. To define the schedule interval of the DAG

B. To specify the Python version used by the DAG

C. To set common parameters like retries and owner for all tasks

D. To list all the tasks in the DAG

Solution

1. Step 1: Understand the role of default_args

   default_args is a dictionary used to set common settings for tasks, such as retries, owner, start date, etc.

2. Step 2: Differentiate from DAG parameters

   DAG parameters like schedule_interval control when the DAG runs, not task defaults.

3. Final Answer:

   To set common parameters like retries and owner for all tasks -> Option C

4. Quick Check:

   default_args = common task settings [OK]

Hint: default_args sets task defaults, not DAG schedule [OK]

Common Mistakes:

• Confusing default_args with DAG schedule_interval
• Thinking default_args lists tasks
• Assuming default_args sets Python version

2. Which of the following is the correct way to define default_args with retries set to 3 and owner as 'airflow'?

easy

A. default_args = {'retries': 3, 'owner': 'airflow'}

B. default_args = ['retries=3', 'owner=airflow']

C. default_args = (retries=3, owner='airflow')

D. default_args = retries:3, owner:'airflow'

Solution

1. Step 1: Recall Python dictionary syntax

   Python dictionaries use curly braces with key:value pairs, keys as strings and values properly quoted if strings.

2. Step 2: Check each option's syntax

   default_args = {'retries': 3, 'owner': 'airflow'} uses correct dictionary syntax. Options A, B, and D use invalid Python syntax for dictionaries.

3. Final Answer:

   default_args = {'retries': 3, 'owner': 'airflow'} -> Option A

4. Quick Check:

   Python dict = curly braces with key:value [OK]

Hint: default_args is a Python dict with keys and values [OK]

Common Mistakes:

• Using list or tuple instead of dict
• Missing quotes around string keys or values
• Using colon instead of equal sign inside dict

3. Given this DAG snippet:

default_args = {'owner': 'admin', 'retries': 2}
with DAG('example_dag', default_args=default_args, schedule_interval='@daily') as dag:
    task1 = BashOperator(task_id='task1', bash_command='echo Hello')
    task2 = BashOperator(task_id='task2', bash_command='echo World', retries=5)

What is the number of retries for task2?

medium

A. 5

B. 2

C. 0

D. Undefined, causes error

Solution

1. Step 1: Understand default_args and task-level override

   default_args sets retries=2 for all tasks unless overridden.

2. Step 2: Check task2 retries parameter

   task2 explicitly sets retries=5, which overrides the default.

3. Final Answer:

   5 -> Option A

4. Quick Check:

   Task-level retries override default_args [OK]

Hint: Task args override default_args if set [OK]

Common Mistakes:

• Assuming default_args always applies
• Thinking task2 inherits retries=2
• Confusing retries with schedule_interval

4. You wrote this DAG code:

default_args = {'owner': 'user', 'start_date': '2023-01-01'}
dag = DAG('my_dag', default_args=default_args, schedule_interval='@hourly')

But Airflow throws an error about start_date. What is the likely problem?

medium

A. owner cannot be 'user', must be 'airflow'

B. schedule_interval '@hourly' is invalid syntax

C. default_args must be passed as a list, not dict

D. start_date must be a datetime object, not a string

Solution

1. Step 1: Check start_date type requirement

   Airflow requires start_date to be a datetime object, not a string.

2. Step 2: Validate other parameters

   schedule_interval='@hourly' is valid, default_args is a dict, and owner can be any string.

3. Final Answer:

   start_date must be a datetime object, not a string -> Option D

4. Quick Check:

   start_date = datetime, not string [OK]

Hint: Use datetime objects for start_date in default_args [OK]

Common Mistakes:

• Passing start_date as string instead of datetime
• Misunderstanding schedule_interval format
• Thinking owner must be 'airflow'

5. You want a DAG to run every Monday at 9 AM starting from March 1, 2024, with tasks retrying twice on failure and owned by 'data_team'. Which default_args and DAG parameters are correct?

hard

A. default_args = {'owner': 'data_team', 'retries': 2, 'start_date': '2024-03-01 09:00'}\ndag = DAG('weekly_dag', default_args=default_args, schedule_interval='0 9 * * 1')

B. default_args = {'owner': 'data_team', 'retries': 2, 'start_date': datetime(2024, 3, 1)}\ndag = DAG('weekly_dag', default_args=default_args, schedule_interval='0 9 * * 1')

C. default_args = {'owner': 'data_team', 'retries': 2, 'start_date': datetime(2024, 3, 1, 9, 0)}\ndag = DAG('weekly_dag', default_args=default_args, schedule_interval='@daily')

D. default_args = {'owner': 'data_team', 'retries': 2, 'start_date': '2024-03-01 09:00'}\ndag = DAG('weekly_dag', default_args=default_args, schedule_interval='@weekly')

Solution

1. Step 1: Set start_date with datetime object

   Start date must be a datetime object; time can be set to midnight if not specified.

2. Step 2: Use correct cron for Monday 9 AM

   Cron '0 9 * * 1' means 9 AM every Monday, matching requirement.

3. Step 3: Check retries and owner in default_args

   Retries=2 and owner='data_team' are correctly set.

4. Final Answer:

   default_args with datetime(2024, 3, 1) and schedule_interval='0 9 * * 1' -> Option B

5. Quick Check:

   start_date datetime + cron for Monday 9 AM [OK]

Hint: Use cron '0 9 * * 1' for Monday 9 AM schedule [OK]

Common Mistakes:

• Using string for start_date
• Using '@weekly' which runs Sunday midnight
• Setting schedule_interval to '@daily' instead of Monday