Task documentation and tags in Apache Airflow - Step-by-Step Execution

Process Flow - Task documentation and tags

Define Task

↓

Add Documentation

↓

Add Tags

↓

Run DAG

↓

View Docs & Tags in UI

This flow shows how to define a task, add documentation and tags, then run the DAG to see docs and tags in Airflow UI.

Execution Sample

Apache Airflow

from airflow import DAG
from airflow.operators.bash import BashOperator
from datetime import datetime

dag = DAG('example_dag', start_date=datetime(2024,1,1))

task = BashOperator(
    task_id='print_date',
    bash_command='date',
    dag=dag,
    doc_md='''This task prints the current date.''',
    tags=['example', 'date']
)

Defines a BashOperator task with documentation and tags inside a DAG.

Process Table

Step	Action	Code Line	Result
1	Import modules	from airflow import DAG from airflow.operators.bash import BashOperator from datetime import datetime	Modules ready
2	Create DAG object	dag = DAG('example_dag', start_date=datetime(2024,1,1))	DAG 'example_dag' created
3	Define task with doc and tags	task = BashOperator(..., doc_md='This task prints the current date.', tags=['example', 'date'])	Task 'print_date' created with docs and tags
4	Run DAG in Airflow UI	airflow dags trigger example_dag	DAG run started, task executes
5	View task docs in UI	Open Airflow UI > DAGs > example_dag > print_date > Documentation tab	Documentation visible: 'This task prints the current date.'
6	View task tags in UI	Open Airflow UI > DAGs > example_dag > Tags column	Tags visible: 'example', 'date'
7	End	N/A	Task docs and tags successfully applied and visible

💡 All steps completed, task documentation and tags are set and visible in Airflow UI

Status Tracker

Variable	Start	After Step 2	After Step 3	Final
dag	None	DAG object created	DAG object unchanged	DAG object unchanged
task	None	None	BashOperator task with doc and tags	Task ready to run with docs and tags

Key Moments - 3 Insights

Why does the task documentation not appear if I forget to use 'doc_md'?

The execution_table row 3 shows that documentation is added via 'doc_md'. Without it, Airflow has no text to display in the docs tab.

Can I add multiple tags to a task and how are they shown?

Yes, as shown in row 3 and 6, tags are a list of strings. They appear in the Airflow UI tags column separated by commas.

Does adding tags affect task execution?

No, tags are metadata only. The execution_table row 4 shows task runs normally regardless of tags.

Visual Quiz - 3 Questions

Look at the execution_table, at which step is the task created with documentation and tags?

A Step 2

B Step 4

C Step 3

D Step 5

💡 Hint

Check the 'Action' column in execution_table row 3 for task creation details.

According to variable_tracker, what is the value of 'task' after step 3?

A DAG object

B BashOperator task with doc and tags

C None

D String with documentation text

💡 Hint

Look at variable_tracker row for 'task' under 'After Step 3' column.

If you remove the 'tags' list from the task, what changes in the execution_table?

A Step 6 will show no tags in UI

B Step 3 will not create the task

C Step 4 will fail to run the DAG

D Step 5 will not show documentation

💡 Hint

Refer to execution_table rows 3 and 6 about tags presence and UI display.

Concept Snapshot

Task documentation and tags in Airflow:
- Use 'doc_md' parameter in task to add markdown docs.
- Use 'tags' parameter as list of strings for metadata.
- Docs appear in Airflow UI under task Documentation tab.
- Tags appear in DAGs list and task details.
- Tags do not affect task execution.

Full Transcript

This visual execution shows how to add documentation and tags to an Airflow task. First, we import necessary modules and create a DAG object. Then, we define a BashOperator task with the 'doc_md' parameter to add documentation and 'tags' parameter to add tags. When the DAG runs, the task executes normally. In the Airflow UI, the documentation is visible under the task's Documentation tab, and tags appear in the DAGs list and task details. Tags are metadata only and do not affect execution. The variable tracker confirms the task object holds the documentation and tags after creation. Key moments clarify common confusions about missing docs, multiple tags, and execution impact. The quiz tests understanding of when docs and tags are set and their UI visibility.

Practice

1. What is the main purpose of adding documentation to an Airflow task using doc_md?

easy

A. To explain what the task does for better understanding

B. To change the task execution order

C. To assign tags for task grouping

D. To increase task execution speed

Solution

1. Step 1: Understand the role of doc_md in Airflow

   doc_md is used to add markdown documentation to a task, explaining its purpose and behavior.

2. Step 2: Differentiate documentation from other features

   Documentation does not affect execution order, tags, or speed; it only improves clarity.

3. Final Answer:

   To explain what the task does for better understanding -> Option A

4. Quick Check:

   Task documentation = Explanation [OK]

Hint: Documentation explains task purpose, not execution or tags [OK]

Common Mistakes:

• Confusing documentation with tags
• Thinking documentation changes task order
• Assuming documentation affects performance

2. Which of the following is the correct way to add tags to an Airflow task in Python code?

easy

A. task = PythonOperator(task_id='task1', python_callable=my_func, tag='tag1')

B. task = PythonOperator(task_id='task1', python_callable=my_func, tags=['tag1', 'tag2'])

C. task = PythonOperator(task_id='task1', python_callable=my_func, tags='tag1,tag2')

D. task = PythonOperator(task_id='task1', python_callable=my_func, labels=['tag1', 'tag2'])

Solution

1. Step 1: Recall the correct parameter for tags

   Airflow tasks accept a tags parameter as a list of strings to assign multiple tags.

2. Step 2: Check syntax correctness

   task = PythonOperator(task_id='task1', python_callable=my_func, tags=['tag1', 'tag2']) uses tags=['tag1', 'tag2'] which is the correct syntax. Other options use incorrect parameter names or types.

3. Final Answer:

   task = PythonOperator(task_id='task1', python_callable=my_func, tags=['tag1', 'tag2']) -> Option B

4. Quick Check:

   Tags parameter = list of strings [OK]

Hint: Use tags as a list with key 'tags' in operator [OK]

Common Mistakes:

• Using 'tag' instead of 'tags'
• Passing tags as a comma string
• Using 'labels' instead of 'tags'

3. Given this Airflow task code snippet, what will be the value of task.doc_md?

task = PythonOperator(
    task_id='example_task',
    python_callable=lambda: print('Hello'),
    doc_md="""This task prints a greeting message."""
)

medium

A. None

B. "Prints a message"

C. SyntaxError

D. "This task prints a greeting message."

Solution

1. Step 1: Identify the doc_md assignment

   The task is created with doc_md set to a triple-quoted string describing the task.

2. Step 2: Understand doc_md stores the documentation string

   The doc_md attribute holds the exact string provided, so it will be "This task prints a greeting message.".

3. Final Answer:

   "This task prints a greeting message." -> Option D

4. Quick Check:

   doc_md value = given string [OK]

Hint: doc_md stores the exact documentation string given [OK]

Common Mistakes:

• Assuming doc_md is None if not printed
• Confusing doc_md with task output
• Expecting syntax error from triple quotes

4. You wrote this Airflow task but it fails to assign tags properly:

task = PythonOperator(
    task_id='task_fail',
    python_callable=my_func,
    tag=['data', 'etl']
)

What is the error and how to fix it?

medium

A. Parameter should be 'tags' not 'tag'; fix by changing to tags=['data', 'etl']

B. PythonOperator does not support tags; remove the tag parameter

C. Tags must be a string, not a list; fix by tags='data,etl'

D. The task_id is invalid; change task_id to 'task_fail_1'

Solution

1. Step 1: Identify the incorrect parameter name

   The code uses tag instead of the correct tags parameter for assigning tags.

2. Step 2: Correct the parameter name and keep the list format

   Changing tag=['data', 'etl'] to tags=['data', 'etl'] fixes the issue.

3. Final Answer:

   Parameter should be 'tags' not 'tag'; fix by changing to tags=['data', 'etl'] -> Option A

4. Quick Check:

   Correct parameter name = tags [OK]

Hint: Use 'tags' plural, not 'tag' singular [OK]

Common Mistakes:

• Using 'tag' instead of 'tags'
• Passing tags as a string instead of list
• Blaming task_id for tag issues

5. You want to organize your Airflow DAG tasks by adding documentation and tags. Which code snippet correctly adds both a markdown description and multiple tags to a task?

hard

A. task = PythonOperator(task_id='task1', python_callable=my_func, doc_md=['Runs ETL process'], tags='etl,daily')

B. task = PythonOperator(task_id='task1', python_callable=my_func, documentation='Runs ETL process', tag=['etl', 'daily'])

C. task = PythonOperator(task_id='task1', python_callable=my_func, doc_md='Runs ETL process', tags=['etl', 'daily'])

D. task = PythonOperator(task_id='task1', python_callable=my_func, doc_md='Runs ETL process', labels=['etl', 'daily'])

Solution

1. Step 1: Verify correct parameter names and types

   doc_md should be a string with markdown text, and tags should be a list of strings.

2. Step 2: Check each option for correctness

   task = PythonOperator(task_id='task1', python_callable=my_func, doc_md='Runs ETL process', tags=['etl', 'daily']) uses doc_md='Runs ETL process' (string) and tags=['etl', 'daily'] (list), which is correct. Others use wrong parameter names or types.

3. Final Answer:

   task = PythonOperator(task_id='task1', python_callable=my_func, doc_md='Runs ETL process', tags=['etl', 'daily']) -> Option C

4. Quick Check:

   doc_md = string, tags = list [OK]

Hint: Use doc_md as string and tags as list for both features [OK]

Common Mistakes:

• Using 'documentation' instead of 'doc_md'
• Passing tags as string instead of list
• Using 'labels' instead of 'tags'