Task documentation and tags in Apache Airflow - Mini Project: Build & Apply

Task documentation and tags

📖 Scenario: You are managing workflows using Apache Airflow. You want to organize your tasks better by adding clear documentation and tags. This helps your team understand what each task does and makes it easier to find tasks by category.

🎯 Goal: Learn how to add documentation and tags to Airflow tasks in a DAG to improve clarity and organization.

📋 What You'll Learn

Create an Airflow DAG with at least one task

Add documentation to the task using the doc_md attribute

Add tags to the task using the tags parameter

Print the task documentation and tags to verify

💡 Why This Matters

🌍 Real World

In real projects, documenting tasks and tagging them helps teams understand workflows quickly and find tasks by category or purpose.

💼 Career

Knowing how to document and tag Airflow tasks is useful for data engineers and DevOps professionals managing complex pipelines.

1

Create a simple Airflow DAG with one task

Create an Airflow DAG called example_dag with a single PythonOperator task named print_hello that runs a function say_hello which prints "Hello Airflow".

Apache Airflow

# Your code here to create example_dag and print_hello task

Hint

Use DAG to create the workflow and PythonOperator for the task.

2

Add documentation to the task

Add a documentation string to the print_hello task using the doc_md attribute. Set it to """This task prints a hello message to the logs.""".

Apache Airflow

from airflow import DAG
from airflow.operators.python import PythonOperator
from datetime import datetime

def say_hello():
    print("Hello Airflow")

example_dag = DAG(
    dag_id='example_dag',
    start_date=datetime(2024, 1, 1),
    schedule_interval='@daily'
)

print_hello = PythonOperator(
    task_id='print_hello',
    python_callable=say_hello,
    dag=example_dag
)

# Add documentation to print_hello task below

Hint

Use the doc_md attribute on the task object to add markdown documentation.

3

Add tags to the task

Add tags to the print_hello task by adding the parameter tags=['example', 'greeting'] when creating the PythonOperator.

Apache Airflow

from airflow import DAG
from airflow.operators.python import PythonOperator
from datetime import datetime

def say_hello():
    print("Hello Airflow")

example_dag = DAG(
    dag_id='example_dag',
    start_date=datetime(2024, 1, 1),
    schedule_interval='@daily'
)

# Add tags=['example', 'greeting'] to print_hello task below
print_hello = PythonOperator(
    task_id='print_hello',
    python_callable=say_hello,
    dag=example_dag
)

print_hello.doc_md = """This task prints a hello message to the logs."""

Hint

Add the tags parameter inside the PythonOperator constructor.

4

Print the task documentation and tags

Print the doc_md and tags attributes of the print_hello task using two separate print() statements.

Apache Airflow

from airflow import DAG
from airflow.operators.python import PythonOperator
from datetime import datetime

def say_hello():
    print("Hello Airflow")

example_dag = DAG(
    dag_id='example_dag',
    start_date=datetime(2024, 1, 1),
    schedule_interval='@daily'
)

print_hello = PythonOperator(
    task_id='print_hello',
    python_callable=say_hello,
    dag=example_dag,
    tags=['example', 'greeting']
)

print_hello.doc_md = """This task prints a hello message to the logs."""

# Print the documentation and tags below

Hint

Use print(print_hello.doc_md) and print(print_hello.tags).

Practice

1. What is the main purpose of adding documentation to an Airflow task using doc_md?

easy

A. To explain what the task does for better understanding

B. To change the task execution order

C. To assign tags for task grouping

D. To increase task execution speed

Solution

1. Step 1: Understand the role of doc_md in Airflow

   doc_md is used to add markdown documentation to a task, explaining its purpose and behavior.

2. Step 2: Differentiate documentation from other features

   Documentation does not affect execution order, tags, or speed; it only improves clarity.

3. Final Answer:

   To explain what the task does for better understanding -> Option A

4. Quick Check:

   Task documentation = Explanation [OK]

Hint: Documentation explains task purpose, not execution or tags [OK]

Common Mistakes:

• Confusing documentation with tags
• Thinking documentation changes task order
• Assuming documentation affects performance

2. Which of the following is the correct way to add tags to an Airflow task in Python code?

easy

A. task = PythonOperator(task_id='task1', python_callable=my_func, tag='tag1')

B. task = PythonOperator(task_id='task1', python_callable=my_func, tags=['tag1', 'tag2'])

C. task = PythonOperator(task_id='task1', python_callable=my_func, tags='tag1,tag2')

D. task = PythonOperator(task_id='task1', python_callable=my_func, labels=['tag1', 'tag2'])

Solution

1. Step 1: Recall the correct parameter for tags

   Airflow tasks accept a tags parameter as a list of strings to assign multiple tags.

2. Step 2: Check syntax correctness

   task = PythonOperator(task_id='task1', python_callable=my_func, tags=['tag1', 'tag2']) uses tags=['tag1', 'tag2'] which is the correct syntax. Other options use incorrect parameter names or types.

3. Final Answer:

   task = PythonOperator(task_id='task1', python_callable=my_func, tags=['tag1', 'tag2']) -> Option B

4. Quick Check:

   Tags parameter = list of strings [OK]

Hint: Use tags as a list with key 'tags' in operator [OK]

Common Mistakes:

• Using 'tag' instead of 'tags'
• Passing tags as a comma string
• Using 'labels' instead of 'tags'

3. Given this Airflow task code snippet, what will be the value of task.doc_md?

task = PythonOperator(
    task_id='example_task',
    python_callable=lambda: print('Hello'),
    doc_md="""This task prints a greeting message."""
)

medium

A. None

B. "Prints a message"

C. SyntaxError

D. "This task prints a greeting message."

Solution

1. Step 1: Identify the doc_md assignment

   The task is created with doc_md set to a triple-quoted string describing the task.

2. Step 2: Understand doc_md stores the documentation string

   The doc_md attribute holds the exact string provided, so it will be "This task prints a greeting message.".

3. Final Answer:

   "This task prints a greeting message." -> Option D

4. Quick Check:

   doc_md value = given string [OK]

Hint: doc_md stores the exact documentation string given [OK]

Common Mistakes:

• Assuming doc_md is None if not printed
• Confusing doc_md with task output
• Expecting syntax error from triple quotes

4. You wrote this Airflow task but it fails to assign tags properly:

task = PythonOperator(
    task_id='task_fail',
    python_callable=my_func,
    tag=['data', 'etl']
)

What is the error and how to fix it?

medium

A. Parameter should be 'tags' not 'tag'; fix by changing to tags=['data', 'etl']

B. PythonOperator does not support tags; remove the tag parameter

C. Tags must be a string, not a list; fix by tags='data,etl'

D. The task_id is invalid; change task_id to 'task_fail_1'

Solution

1. Step 1: Identify the incorrect parameter name

   The code uses tag instead of the correct tags parameter for assigning tags.

2. Step 2: Correct the parameter name and keep the list format

   Changing tag=['data', 'etl'] to tags=['data', 'etl'] fixes the issue.

3. Final Answer:

   Parameter should be 'tags' not 'tag'; fix by changing to tags=['data', 'etl'] -> Option A

4. Quick Check:

   Correct parameter name = tags [OK]

Hint: Use 'tags' plural, not 'tag' singular [OK]

Common Mistakes:

• Using 'tag' instead of 'tags'
• Passing tags as a string instead of list
• Blaming task_id for tag issues

5. You want to organize your Airflow DAG tasks by adding documentation and tags. Which code snippet correctly adds both a markdown description and multiple tags to a task?

hard

A. task = PythonOperator(task_id='task1', python_callable=my_func, doc_md=['Runs ETL process'], tags='etl,daily')

B. task = PythonOperator(task_id='task1', python_callable=my_func, documentation='Runs ETL process', tag=['etl', 'daily'])

C. task = PythonOperator(task_id='task1', python_callable=my_func, doc_md='Runs ETL process', tags=['etl', 'daily'])

D. task = PythonOperator(task_id='task1', python_callable=my_func, doc_md='Runs ETL process', labels=['etl', 'daily'])

Solution

1. Step 1: Verify correct parameter names and types

   doc_md should be a string with markdown text, and tags should be a list of strings.

2. Step 2: Check each option for correctness

   task = PythonOperator(task_id='task1', python_callable=my_func, doc_md='Runs ETL process', tags=['etl', 'daily']) uses doc_md='Runs ETL process' (string) and tags=['etl', 'daily'] (list), which is correct. Others use wrong parameter names or types.

3. Final Answer:

   task = PythonOperator(task_id='task1', python_callable=my_func, doc_md='Runs ETL process', tags=['etl', 'daily']) -> Option C

4. Quick Check:

   doc_md = string, tags = list [OK]

Hint: Use doc_md as string and tags as list for both features [OK]

Common Mistakes:

• Using 'documentation' instead of 'doc_md'
• Passing tags as string instead of list
• Using 'labels' instead of 'tags'