Debugging with Airflow CLI - Step-by-Step Execution

Process Flow - Debugging with Airflow CLI

Start Airflow CLI

↓

Run 'airflow tasks test'

↓

Task runs locally

↓

Check logs output

↓

Identify error or success

↓

Fix code or config

↓

Repeat test or run DAG

This flow shows how to use the Airflow CLI to run a task locally for debugging, check logs, fix issues, and repeat until the task works.

Execution Sample

Apache Airflow

airflow tasks test example_dag example_task 2024-06-01
# Runs the task locally and shows logs for debugging

This command runs a specific task from a DAG on a given date locally, showing logs to help find errors.

Process Table

Step	Command	Action	Output/Result
1	airflow tasks test example_dag example_task 2024-06-01	Run task locally	Task starts, logs show execution steps
2	Logs output	Check logs for errors or success	Logs show task running, possible error message if failure
3	Fix code/config	Edit DAG or task code based on logs	Code updated to fix error
4	Repeat test command	Run task test again	Task runs successfully, logs confirm no errors
5	Run DAG normally	Trigger DAG run in Airflow UI or CLI	DAG runs with fixed task
6	-	Exit debugging	Task works as expected, debugging complete

💡 Task runs successfully without errors, debugging finished

Status Tracker

Variable	Start	After 1	After 2	After 3	Final
Task Status	Not run	Running	Failed (error found)	Running (after fix)	Success
Logs	Empty	Execution logs appear	Error message visible	Updated logs	Clean logs, no errors

Key Moments - 3 Insights

Why do we run 'airflow tasks test' instead of triggering the whole DAG?

Running 'airflow tasks test' runs only one task locally without scheduling, so you can quickly see logs and debug without waiting for the full DAG run. See execution_table step 1.

What does it mean if logs show an error after running the test command?

It means the task code or configuration has a problem. The logs help identify the exact error so you can fix it before running the full DAG. See execution_table step 2.

Why do we repeat the test command after fixing code?

Repeating the test confirms the fix works and no new errors appear before running the DAG normally. See execution_table step 4.

Visual Quiz - 3 Questions

Look at the execution table, what is the task status after the first test run?

A Failed (error found)

B Running

C Success

D Not run

💡 Hint

Check the 'Task Status' variable in variable_tracker after step 1

At which step does the user fix the code based on logs?

A Step 2

B Step 4

C Step 3

D Step 5

💡 Hint

Look at the 'Action' column in execution_table for the step where code is edited

If the logs still show errors after step 4, what should happen next?

A Fix code again and repeat test

B Run the DAG normally

C Exit debugging

D Ignore errors and continue

💡 Hint

Refer to key_moments about repeating tests after fixes

Concept Snapshot

Use 'airflow tasks test <dag_id> <task_id> <date>' to run a task locally.
Check logs output to find errors.
Fix code or config based on logs.
Repeat test until task succeeds.
Then run DAG normally.
This speeds up debugging without full DAG runs.

Full Transcript

Debugging with Airflow CLI involves running a single task locally using the command 'airflow tasks test' with the DAG id, task id, and execution date. This runs the task immediately and shows logs in the terminal. By reading these logs, you can find errors or confirm success. If errors appear, you fix the code or configuration, then run the test command again to verify the fix. Once the task runs successfully without errors, you can trigger the full DAG run. This method helps debug tasks quickly without waiting for scheduled DAG runs.

Practice

1. What is the primary purpose of the Airflow CLI command airflow tasks list DAG_ID?

easy

A. To show logs of a specific task

B. To run all tasks in a DAG immediately

C. To clear all task instances in a DAG

D. To display all tasks in a specific DAG

Solution

1. Step 1: Understand the command structure

   The command airflow tasks list DAG_ID is used to list tasks related to the DAG identified by DAG_ID.

2. Step 2: Identify the command's function

   This command shows all tasks defined in the DAG, helping users see what tasks exist.

3. Final Answer:

   To display all tasks in a specific DAG -> Option D

4. Quick Check:

   List tasks = Display tasks [OK]

Hint: List tasks to see all steps in a DAG quickly [OK]

Common Mistakes:

• Confusing 'list' with 'run' to execute tasks
• Thinking it clears tasks instead of listing
• Assuming it shows logs instead of task names

2. Which of the following is the correct Airflow CLI command to view logs for a task named task1 in DAG example_dag for execution date 2023-01-01T00:00:00?

easy

A. airflow tasks logs example_dag task1 2023-01-01T00:00:00

B. airflow logs example_dag task1 2023-01-01T00:00:00

C. airflow task logs example_dag task1 2023-01-01T00:00:00

D. airflow tasks log example_dag task1 2023-01-01T00:00:00

Solution

1. Step 1: Recall the correct Airflow CLI syntax for logs

   The correct command to view logs is airflow tasks logs DAG_ID TASK_ID EXECUTION_DATE.

2. Step 2: Match the command with the options

   airflow tasks logs example_dag task1 2023-01-01T00:00:00 matches the exact syntax with correct keywords and parameters.

3. Final Answer:

   airflow tasks logs example_dag task1 2023-01-01T00:00:00 -> Option A

4. Quick Check:

   Correct syntax for logs = airflow tasks logs example_dag task1 2023-01-01T00:00:00 [OK]

Hint: Use 'tasks logs' to view task logs with DAG and date [OK]

Common Mistakes:

• Using 'logs' without 'tasks' keyword
• Misspelling 'tasks' as 'task'
• Using 'log' instead of 'logs'

3. What will be the output of the command airflow tasks clear example_dag -t task1 -s 2023-01-01 -e 2023-01-02?

medium

A. It runs task1 in example_dag for the specified dates

B. It lists all tasks in example_dag between the given dates

C. It clears all task instances of task1 in example_dag between 2023-01-01 and 2023-01-02

D. It shows logs of task1 for the specified dates

Solution

1. Step 1: Understand the 'tasks clear' command

   The command airflow tasks clear removes task instance states, allowing rerun.

2. Step 2: Analyze the options -t, -s, -e

   -t specifies the task, -s and -e specify start and end dates for clearing task instances.

3. Final Answer:

   It clears all task instances of task1 in example_dag between 2023-01-01 and 2023-01-02 -> Option C

4. Quick Check:

   Clear task instances = It clears all task instances of task1 in example_dag between 2023-01-01 and 2023-01-02 [OK]

Hint: Clear tasks with -t for task and -s/-e for date range [OK]

Common Mistakes:

• Confusing clear with run or list commands
• Ignoring date range flags -s and -e
• Assuming it shows logs instead of clearing

4. You run airflow tasks run example_dag task1 2023-01-01 but get an error saying the task instance does not exist. What is the most likely cause?

medium

A. The command syntax is incorrect and missing required flags

B. The DAG is not loaded or paused, so the task instance is missing

C. The execution date format is wrong and not recognized

D. The task1 does not exist in any DAG

Solution

1. Step 1: Understand task instance creation

   Task instances exist only if the DAG is loaded and active (not paused).

2. Step 2: Identify why instance might be missing

   If the DAG is paused or not loaded, the task instance for the date won't exist, causing the error.

3. Final Answer:

   The DAG is not loaded or paused, so the task instance is missing -> Option B

4. Quick Check:

   Missing instance = DAG paused/not loaded [OK]

Hint: Check DAG status; paused DAGs block task instances [OK]

Common Mistakes:

• Assuming syntax error without checking DAG state
• Blaming date format without verifying
• Thinking task1 is missing from DAG

5. You want to debug a failed task task2 in DAG data_pipeline for execution date 2023-06-15T12:00:00. Which sequence of Airflow CLI commands helps you clear the failed task, rerun it, and then view its logs?

hard

A. airflow tasks clear data_pipeline -t task2 -s 2023-06-15T12:00:00 -e 2023-06-15T12:00:00 && airflow tasks run data_pipeline task2 2023-06-15T12:00:00 && airflow tasks logs data_pipeline task2 2023-06-15T12:00:00

B. airflow tasks run data_pipeline task2 2023-06-15T12:00:00 && airflow tasks clear data_pipeline -t task2 -s 2023-06-15T12:00:00 -e 2023-06-15T12:00:00 && airflow tasks logs data_pipeline task2 2023-06-15T12:00:00

C. airflow tasks logs data_pipeline task2 2023-06-15T12:00:00 && airflow tasks clear data_pipeline -t task2 -s 2023-06-15T12:00:00 -e 2023-06-15T12:00:00 && airflow tasks run data_pipeline task2 2023-06-15T12:00:00

D. airflow tasks clear data_pipeline -t task2 && airflow tasks logs data_pipeline task2 2023-06-15T12:00:00 && airflow tasks run data_pipeline task2 2023-06-15T12:00:00

Solution

1. Step 1: Clear the failed task instance for the exact execution date

   Use airflow tasks clear with task name and exact start/end date to reset the failed task.

2. Step 2: Run the cleared task instance

   Use airflow tasks run with DAG, task, and execution date to rerun the task.

3. Step 3: View the logs of the rerun task

   Use airflow tasks logs to check the output and debug information.

4. Final Answer:

   Clear, then run, then logs in sequence with correct dates -> Option A

5. Quick Check:

   Clear -> Run -> Logs = airflow tasks clear data_pipeline -t task2 -s 2023-06-15T12:00:00 -e 2023-06-15T12:00:00 && airflow tasks run data_pipeline task2 2023-06-15T12:00:00 && airflow tasks logs data_pipeline task2 2023-06-15T12:00:00 [OK]

Hint: Clear failed task first, then run, then check logs [OK]

Common Mistakes:

• Running task before clearing causes rerun failure
• Checking logs before rerun shows old failure logs
• Omitting date range in clear command