Debugging with Airflow CLI - Commands & Configuration

Introduction

Sometimes your Airflow tasks don't run as expected. Debugging with the Airflow CLI helps you find and fix problems by checking logs and task status directly from the command line.

When a task in your Airflow DAG fails and you want to see the error details quickly.

When you want to check the status of a specific task instance without opening the Airflow web UI.

When you need to clear a failed task to retry it from the command line.

When you want to test if a DAG file is valid before deploying it.

When you want to see the list of all DAGs and their last run status to spot issues.

Commands

This command lists all the DAGs currently available in Airflow. It helps you confirm your DAG is loaded and ready.

Terminal

airflow dags list

Expected Output

dag_id          | filepath

example_dag     | /usr/local/airflow/dags/example_dag.py

my_test_dag     | /usr/local/airflow/dags/my_test_dag.py

This shows all the tasks inside the DAG named 'example_dag'. It helps you know the task names you can debug.

Terminal

airflow tasks list example_dag

Expected Output

task_id

start_task

process_data

end_task

Runs the 'process_data' task from 'example_dag' for the date 2024-06-01 locally without affecting the scheduler. Useful for quick debugging.

Terminal

airflow tasks test example_dag process_data 2024-06-01

Expected Output

[2024-06-01 12:00:00,000] {taskinstance.py:876} INFO - Starting attempt 1 of 1

[2024-06-01 12:00:01,000] {taskinstance.py:900} INFO - Task succeeded

Clears the state of the 'process_data' task for the given date so it can be retried. This is useful if you want to rerun a failed task.

Terminal

airflow tasks clear example_dag process_data --start-date 2024-06-01 --end-date 2024-06-01

Expected Output

Cleared the state of 1 task instance from example_dag.process_data for execution dates between 2024-06-01 00:00:00 and 2024-06-01 00:00:00

→

--start-date - Defines the start date of the task instances to clear

→

--end-date - Defines the end date of the task instances to clear

Shows the logs of the 'process_data' task for the date 2024-06-01. Logs help you understand what happened during task execution.

Terminal

airflow tasks logs example_dag process_data 2024-06-01

Expected Output

[2024-06-01 12:00:00,000] INFO - Running process_data task

[2024-06-01 12:00:01,000] ERROR - Failed to connect to database

[2024-06-01 12:00:02,000] INFO - Task failed

Key Concept

If you remember nothing else from debugging Airflow tasks, remember: use the CLI to check task status, logs, and rerun tasks quickly without the web UI.

Common Mistakes

Trying to run 'airflow tasks test' without specifying the exact execution date.

Airflow needs the execution date to know which task instance to run for testing.

Always provide the execution date in YYYY-MM-DD format when using 'airflow tasks test'.

Clearing tasks without specifying the date range.

This can clear many task instances unintentionally, causing unnecessary reruns.

Use --start-date and --end-date flags to clear only the specific task instances you want.

Checking logs for a task that has never run.

No logs exist yet, so the command will show empty or error messages.

Make sure the task has run at least once before checking logs.

Summary

Use 'airflow dags list' to see all available DAGs.

Use 'airflow tasks list DAG_NAME' to see tasks inside a DAG.

Use 'airflow tasks test' to run a task locally for debugging.

Use 'airflow tasks clear' with date flags to reset task state for retries.

Use 'airflow tasks logs' to view detailed logs of task runs.

Practice

1. What is the primary purpose of the Airflow CLI command airflow tasks list DAG_ID?

easy

A. To show logs of a specific task

B. To run all tasks in a DAG immediately

C. To clear all task instances in a DAG

D. To display all tasks in a specific DAG

Solution

1. Step 1: Understand the command structure

   The command airflow tasks list DAG_ID is used to list tasks related to the DAG identified by DAG_ID.

2. Step 2: Identify the command's function

   This command shows all tasks defined in the DAG, helping users see what tasks exist.

3. Final Answer:

   To display all tasks in a specific DAG -> Option D

4. Quick Check:

   List tasks = Display tasks [OK]

Hint: List tasks to see all steps in a DAG quickly [OK]

Common Mistakes:

• Confusing 'list' with 'run' to execute tasks
• Thinking it clears tasks instead of listing
• Assuming it shows logs instead of task names

2. Which of the following is the correct Airflow CLI command to view logs for a task named task1 in DAG example_dag for execution date 2023-01-01T00:00:00?

easy

A. airflow tasks logs example_dag task1 2023-01-01T00:00:00

B. airflow logs example_dag task1 2023-01-01T00:00:00

C. airflow task logs example_dag task1 2023-01-01T00:00:00

D. airflow tasks log example_dag task1 2023-01-01T00:00:00

Solution

1. Step 1: Recall the correct Airflow CLI syntax for logs

   The correct command to view logs is airflow tasks logs DAG_ID TASK_ID EXECUTION_DATE.

2. Step 2: Match the command with the options

   airflow tasks logs example_dag task1 2023-01-01T00:00:00 matches the exact syntax with correct keywords and parameters.

3. Final Answer:

   airflow tasks logs example_dag task1 2023-01-01T00:00:00 -> Option A

4. Quick Check:

   Correct syntax for logs = airflow tasks logs example_dag task1 2023-01-01T00:00:00 [OK]

Hint: Use 'tasks logs' to view task logs with DAG and date [OK]

Common Mistakes:

• Using 'logs' without 'tasks' keyword
• Misspelling 'tasks' as 'task'
• Using 'log' instead of 'logs'

3. What will be the output of the command airflow tasks clear example_dag -t task1 -s 2023-01-01 -e 2023-01-02?

medium

A. It runs task1 in example_dag for the specified dates

B. It lists all tasks in example_dag between the given dates

C. It clears all task instances of task1 in example_dag between 2023-01-01 and 2023-01-02

D. It shows logs of task1 for the specified dates

Solution

1. Step 1: Understand the 'tasks clear' command

   The command airflow tasks clear removes task instance states, allowing rerun.

2. Step 2: Analyze the options -t, -s, -e

   -t specifies the task, -s and -e specify start and end dates for clearing task instances.

3. Final Answer:

   It clears all task instances of task1 in example_dag between 2023-01-01 and 2023-01-02 -> Option C

4. Quick Check:

   Clear task instances = It clears all task instances of task1 in example_dag between 2023-01-01 and 2023-01-02 [OK]

Hint: Clear tasks with -t for task and -s/-e for date range [OK]

Common Mistakes:

• Confusing clear with run or list commands
• Ignoring date range flags -s and -e
• Assuming it shows logs instead of clearing

4. You run airflow tasks run example_dag task1 2023-01-01 but get an error saying the task instance does not exist. What is the most likely cause?

medium

A. The command syntax is incorrect and missing required flags

B. The DAG is not loaded or paused, so the task instance is missing

C. The execution date format is wrong and not recognized

D. The task1 does not exist in any DAG

Solution

1. Step 1: Understand task instance creation

   Task instances exist only if the DAG is loaded and active (not paused).

2. Step 2: Identify why instance might be missing

   If the DAG is paused or not loaded, the task instance for the date won't exist, causing the error.

3. Final Answer:

   The DAG is not loaded or paused, so the task instance is missing -> Option B

4. Quick Check:

   Missing instance = DAG paused/not loaded [OK]

Hint: Check DAG status; paused DAGs block task instances [OK]

Common Mistakes:

• Assuming syntax error without checking DAG state
• Blaming date format without verifying
• Thinking task1 is missing from DAG

5. You want to debug a failed task task2 in DAG data_pipeline for execution date 2023-06-15T12:00:00. Which sequence of Airflow CLI commands helps you clear the failed task, rerun it, and then view its logs?

hard

A. airflow tasks clear data_pipeline -t task2 -s 2023-06-15T12:00:00 -e 2023-06-15T12:00:00 && airflow tasks run data_pipeline task2 2023-06-15T12:00:00 && airflow tasks logs data_pipeline task2 2023-06-15T12:00:00

B. airflow tasks run data_pipeline task2 2023-06-15T12:00:00 && airflow tasks clear data_pipeline -t task2 -s 2023-06-15T12:00:00 -e 2023-06-15T12:00:00 && airflow tasks logs data_pipeline task2 2023-06-15T12:00:00

C. airflow tasks logs data_pipeline task2 2023-06-15T12:00:00 && airflow tasks clear data_pipeline -t task2 -s 2023-06-15T12:00:00 -e 2023-06-15T12:00:00 && airflow tasks run data_pipeline task2 2023-06-15T12:00:00

D. airflow tasks clear data_pipeline -t task2 && airflow tasks logs data_pipeline task2 2023-06-15T12:00:00 && airflow tasks run data_pipeline task2 2023-06-15T12:00:00

Solution

1. Step 1: Clear the failed task instance for the exact execution date

   Use airflow tasks clear with task name and exact start/end date to reset the failed task.

2. Step 2: Run the cleared task instance

   Use airflow tasks run with DAG, task, and execution date to rerun the task.

3. Step 3: View the logs of the rerun task

   Use airflow tasks logs to check the output and debug information.

4. Final Answer:

   Clear, then run, then logs in sequence with correct dates -> Option A

5. Quick Check:

   Clear -> Run -> Logs = airflow tasks clear data_pipeline -t task2 -s 2023-06-15T12:00:00 -e 2023-06-15T12:00:00 && airflow tasks run data_pipeline task2 2023-06-15T12:00:00 && airflow tasks logs data_pipeline task2 2023-06-15T12:00:00 [OK]

Hint: Clear failed task first, then run, then check logs [OK]

Common Mistakes:

• Running task before clearing causes rerun failure
• Checking logs before rerun shows old failure logs
• Omitting date range in clear command