Task documentation and tags in Apache Airflow - Commands & Configuration

Introduction

When you create tasks in Airflow, you want to explain what each task does and organize them well. Task documentation helps others understand your workflow. Tags help group and find tasks easily in the Airflow UI.

When you want to explain the purpose of a task clearly for your team.

When you have many tasks and want to filter or search them by category.

When you want to add extra notes or instructions to a task for future reference.

When you want to organize tasks by type, like 'data-processing' or 'notification'.

When you want to improve the readability and maintainability of your DAGs.

Config File - my_dag.py

my_dag.py

from airflow import DAG
from airflow.operators.python import PythonOperator
from datetime import datetime

def greet():
    print('Hello from the task!')

def farewell():
    print('Goodbye from the task!')

with DAG(
    dag_id='example_task_docs_tags',
    start_date=datetime(2024, 1, 1),
    schedule_interval='@daily',
    catchup=False
) as dag:

    greet_task = PythonOperator(
        task_id='greet',
        python_callable=greet,
        doc_md='''
        ### Greet Task
        This task prints a greeting message to the logs.
        It helps verify the DAG is running correctly.
        ''',
        tags=['greeting', 'example']
    )

    farewell_task = PythonOperator(
        task_id='farewell',
        python_callable=farewell,
        doc_md='''
        ### Farewell Task
        This task prints a farewell message to the logs.
        It runs after the greet task.
        ''',
        tags=['farewell', 'example']
    )

    greet_task >> farewell_task

This DAG file defines two tasks using PythonOperator.

doc_md adds markdown documentation visible in the Airflow UI for each task.

tags assign labels to tasks for easy filtering and grouping in the UI.

The tasks are connected so that farewell_task runs after greet_task.

Commands

List all DAGs available in Airflow to confirm your DAG is recognized.

Terminal

airflow dags list

Expected Output

example_task_docs_tags

List all tasks in the example_task_docs_tags DAG to see their IDs and confirm they are loaded.

Terminal

airflow tasks list example_task_docs_tags

Expected Output

greet

farewell

Run the greet task manually for the date 2024-01-01 to see its output and verify it works.

Terminal

airflow tasks test example_task_docs_tags greet 2024-01-01

Expected Output

[2024-01-01 00:00:00,000] {python.py:114} INFO - Hello from the task!

[2024-01-01 00:00:00,001] {taskinstance.py:1234} INFO - Task exited with return code 0

Run the farewell task manually for the date 2024-01-01 to see its output and verify it works.

Terminal

airflow tasks test example_task_docs_tags farewell 2024-01-01

Expected Output

[2024-01-01 00:00:00,000] {python.py:114} INFO - Goodbye from the task!

[2024-01-01 00:00:00,001] {taskinstance.py:1234} INFO - Task exited with return code 0

Key Concept

If you remember nothing else, remember: task documentation explains what a task does, and tags help you find and organize tasks easily in the Airflow UI.

Common Mistakes

Not adding doc_md to tasks or leaving it empty

Without documentation, others cannot understand the task's purpose, making maintenance harder.

Always add clear doc_md strings to explain what each task does.

Using inconsistent or no tags on tasks

Without tags, filtering and grouping tasks in the UI becomes difficult, especially in large DAGs.

Use meaningful and consistent tags to group related tasks.

Adding tags or doc_md outside the task definition

Tags and documentation must be inside the task operator to be recognized by Airflow.

Always include doc_md and tags as parameters inside the task operator constructor.

Summary

Define task documentation using the doc_md parameter to explain task purpose.

Use tags parameter to label tasks for easy filtering in the Airflow UI.

Verify tasks and DAGs using airflow CLI commands like dags list and tasks test.

Practice

1. What is the main purpose of adding documentation to an Airflow task using doc_md?

easy

A. To explain what the task does for better understanding

B. To change the task execution order

C. To assign tags for task grouping

D. To increase task execution speed

Solution

1. Step 1: Understand the role of doc_md in Airflow

   doc_md is used to add markdown documentation to a task, explaining its purpose and behavior.

2. Step 2: Differentiate documentation from other features

   Documentation does not affect execution order, tags, or speed; it only improves clarity.

3. Final Answer:

   To explain what the task does for better understanding -> Option A

4. Quick Check:

   Task documentation = Explanation [OK]

Hint: Documentation explains task purpose, not execution or tags [OK]

Common Mistakes:

• Confusing documentation with tags
• Thinking documentation changes task order
• Assuming documentation affects performance

2. Which of the following is the correct way to add tags to an Airflow task in Python code?

easy

A. task = PythonOperator(task_id='task1', python_callable=my_func, tag='tag1')

B. task = PythonOperator(task_id='task1', python_callable=my_func, tags=['tag1', 'tag2'])

C. task = PythonOperator(task_id='task1', python_callable=my_func, tags='tag1,tag2')

D. task = PythonOperator(task_id='task1', python_callable=my_func, labels=['tag1', 'tag2'])

Solution

1. Step 1: Recall the correct parameter for tags

   Airflow tasks accept a tags parameter as a list of strings to assign multiple tags.

2. Step 2: Check syntax correctness

   task = PythonOperator(task_id='task1', python_callable=my_func, tags=['tag1', 'tag2']) uses tags=['tag1', 'tag2'] which is the correct syntax. Other options use incorrect parameter names or types.

3. Final Answer:

   task = PythonOperator(task_id='task1', python_callable=my_func, tags=['tag1', 'tag2']) -> Option B

4. Quick Check:

   Tags parameter = list of strings [OK]

Hint: Use tags as a list with key 'tags' in operator [OK]

Common Mistakes:

• Using 'tag' instead of 'tags'
• Passing tags as a comma string
• Using 'labels' instead of 'tags'

3. Given this Airflow task code snippet, what will be the value of task.doc_md?

task = PythonOperator(
    task_id='example_task',
    python_callable=lambda: print('Hello'),
    doc_md="""This task prints a greeting message."""
)

medium

A. None

B. "Prints a message"

C. SyntaxError

D. "This task prints a greeting message."

Solution

1. Step 1: Identify the doc_md assignment

   The task is created with doc_md set to a triple-quoted string describing the task.

2. Step 2: Understand doc_md stores the documentation string

   The doc_md attribute holds the exact string provided, so it will be "This task prints a greeting message.".

3. Final Answer:

   "This task prints a greeting message." -> Option D

4. Quick Check:

   doc_md value = given string [OK]

Hint: doc_md stores the exact documentation string given [OK]

Common Mistakes:

• Assuming doc_md is None if not printed
• Confusing doc_md with task output
• Expecting syntax error from triple quotes

4. You wrote this Airflow task but it fails to assign tags properly:

task = PythonOperator(
    task_id='task_fail',
    python_callable=my_func,
    tag=['data', 'etl']
)

What is the error and how to fix it?

medium

A. Parameter should be 'tags' not 'tag'; fix by changing to tags=['data', 'etl']

B. PythonOperator does not support tags; remove the tag parameter

C. Tags must be a string, not a list; fix by tags='data,etl'

D. The task_id is invalid; change task_id to 'task_fail_1'

Solution

1. Step 1: Identify the incorrect parameter name

   The code uses tag instead of the correct tags parameter for assigning tags.

2. Step 2: Correct the parameter name and keep the list format

   Changing tag=['data', 'etl'] to tags=['data', 'etl'] fixes the issue.

3. Final Answer:

   Parameter should be 'tags' not 'tag'; fix by changing to tags=['data', 'etl'] -> Option A

4. Quick Check:

   Correct parameter name = tags [OK]

Hint: Use 'tags' plural, not 'tag' singular [OK]

Common Mistakes:

• Using 'tag' instead of 'tags'
• Passing tags as a string instead of list
• Blaming task_id for tag issues

5. You want to organize your Airflow DAG tasks by adding documentation and tags. Which code snippet correctly adds both a markdown description and multiple tags to a task?

hard

A. task = PythonOperator(task_id='task1', python_callable=my_func, doc_md=['Runs ETL process'], tags='etl,daily')

B. task = PythonOperator(task_id='task1', python_callable=my_func, documentation='Runs ETL process', tag=['etl', 'daily'])

C. task = PythonOperator(task_id='task1', python_callable=my_func, doc_md='Runs ETL process', tags=['etl', 'daily'])

D. task = PythonOperator(task_id='task1', python_callable=my_func, doc_md='Runs ETL process', labels=['etl', 'daily'])

Solution

1. Step 1: Verify correct parameter names and types

   doc_md should be a string with markdown text, and tags should be a list of strings.

2. Step 2: Check each option for correctness

   task = PythonOperator(task_id='task1', python_callable=my_func, doc_md='Runs ETL process', tags=['etl', 'daily']) uses doc_md='Runs ETL process' (string) and tags=['etl', 'daily'] (list), which is correct. Others use wrong parameter names or types.

3. Final Answer:

   task = PythonOperator(task_id='task1', python_callable=my_func, doc_md='Runs ETL process', tags=['etl', 'daily']) -> Option C

4. Quick Check:

   doc_md = string, tags = list [OK]

Hint: Use doc_md as string and tags as list for both features [OK]

Common Mistakes:

• Using 'documentation' instead of 'doc_md'
• Passing tags as string instead of list
• Using 'labels' instead of 'tags'