Task documentation and tags in Apache Airflow - Mini Project: Build & Apply

Task documentation and tags

📖 Scenario: You are managing workflows using Apache Airflow. You want to organize your tasks better by adding clear documentation and tags. This helps your team understand what each task does and makes it easier to find tasks by category.

🎯 Goal: Learn how to add documentation and tags to Airflow tasks in a DAG to improve clarity and organization.

📋 What You'll Learn

Create an Airflow DAG with at least one task

Add documentation to the task using the doc_md attribute

Add tags to the task using the tags parameter

Print the task documentation and tags to verify

💡 Why This Matters

🌍 Real World

In real projects, documenting tasks and tagging them helps teams understand workflows quickly and find tasks by category or purpose.

💼 Career

Knowing how to document and tag Airflow tasks is useful for data engineers and DevOps professionals managing complex pipelines.

1

Create a simple Airflow DAG with one task

Create an Airflow DAG called example_dag with a single PythonOperator task named print_hello that runs a function say_hello which prints "Hello Airflow".

Apache Airflow

# Your code here to create example_dag and print_hello task

Need a hint?

Use DAG to create the workflow and PythonOperator for the task.

2

Add documentation to the task

Add a documentation string to the print_hello task using the doc_md attribute. Set it to """This task prints a hello message to the logs.""".

Apache Airflow

from airflow import DAG
from airflow.operators.python import PythonOperator
from datetime import datetime

def say_hello():
    print("Hello Airflow")

example_dag = DAG(
    dag_id='example_dag',
    start_date=datetime(2024, 1, 1),
    schedule_interval='@daily'
)

print_hello = PythonOperator(
    task_id='print_hello',
    python_callable=say_hello,
    dag=example_dag
)

# Add documentation to print_hello task below

Need a hint?

Use the doc_md attribute on the task object to add markdown documentation.

3

Add tags to the task

Add tags to the print_hello task by adding the parameter tags=['example', 'greeting'] when creating the PythonOperator.

Apache Airflow

from airflow import DAG
from airflow.operators.python import PythonOperator
from datetime import datetime

def say_hello():
    print("Hello Airflow")

example_dag = DAG(
    dag_id='example_dag',
    start_date=datetime(2024, 1, 1),
    schedule_interval='@daily'
)

# Add tags=['example', 'greeting'] to print_hello task below
print_hello = PythonOperator(
    task_id='print_hello',
    python_callable=say_hello,
    dag=example_dag
)

print_hello.doc_md = """This task prints a hello message to the logs."""

Need a hint?

Add the tags parameter inside the PythonOperator constructor.

4

Print the task documentation and tags

Print the doc_md and tags attributes of the print_hello task using two separate print() statements.

Apache Airflow

from airflow import DAG
from airflow.operators.python import PythonOperator
from datetime import datetime

def say_hello():
    print("Hello Airflow")

example_dag = DAG(
    dag_id='example_dag',
    start_date=datetime(2024, 1, 1),
    schedule_interval='@daily'
)

print_hello = PythonOperator(
    task_id='print_hello',
    python_callable=say_hello,
    dag=example_dag,
    tags=['example', 'greeting']
)

print_hello.doc_md = """This task prints a hello message to the logs."""

# Print the documentation and tags below

Need a hint?

Use print(print_hello.doc_md) and print(print_hello.tags).