Debugging with Airflow CLI - Mini Project: Build & Apply

Debugging with Airflow CLI

📖 Scenario: You are a data engineer managing workflows using Apache Airflow. Sometimes, tasks in your workflows fail, and you need to quickly find out what went wrong. Airflow provides a command-line tool (CLI) to help you debug these issues by checking task logs and statuses.

🎯 Goal: Learn how to use the Airflow CLI commands to check the status of a task instance and view its logs for debugging.

📋 What You'll Learn

Use the Airflow CLI commands exactly as specified

Work with a DAG named example_dag

Check the status and logs of a task named example_task

Use the execution date 2024-04-01T00:00:00+00:00

💡 Why This Matters

🌍 Real World

Data engineers and DevOps professionals use Airflow CLI commands daily to monitor and debug workflows without needing to open the Airflow web interface.

💼 Career

Knowing how to quickly check task statuses and logs using the Airflow CLI helps you troubleshoot workflow issues efficiently, a key skill for roles managing data pipelines and automation.

1

Set up the DAG and task names

Create two variables: dag_id with the value 'example_dag' and task_id with the value 'example_task'.

Apache Airflow

# Create variables dag_id and task_id with exact values
# Your code here

Need a hint?

Use simple assignment statements to create the variables with the exact string values.

2

Set the execution date variable

Create a variable called execution_date and set it to the string '2024-04-01T00:00:00+00:00'.

Apache Airflow

dag_id = 'example_dag'
task_id = 'example_task'
# Create variable execution_date with the exact value
# Your code here

Need a hint?

Assign the exact string to the variable execution_date.

3

Write the Airflow CLI command to check task state

Create a variable called check_state_cmd that holds the exact Airflow CLI command string to check the state of the task instance. Use the format: airflow tasks state {dag_id} {task_id} {execution_date} with the variables inserted using an f-string.

Apache Airflow

dag_id = 'example_dag'
task_id = 'example_task'
execution_date = '2024-04-01T00:00:00+00:00'
# Create check_state_cmd with the exact airflow CLI command string using f-string
# Your code here

Need a hint?

Use an f-string to insert the variables into the command string exactly as shown.

4

Print the command to view task logs

Print the exact Airflow CLI command to view the logs of the task instance using the format: airflow tasks logs example_dag example_task 2024-04-01T00:00:00+00:00.

Apache Airflow

dag_id = 'example_dag'
task_id = 'example_task'
execution_date = '2024-04-01T00:00:00+00:00'
check_state_cmd = f"airflow tasks state {dag_id} {task_id} {execution_date}"
# Print the airflow CLI command to view logs exactly as specified
# Your code here

Need a hint?

Use a print statement with an f-string to output the exact command string.