Log inspection and troubleshooting in Apache Airflow - Step-by-Step Execution

Process Flow - Log inspection and troubleshooting

Trigger DAG Run

↓

Task Starts Execution

↓

Logs Generated

↓

Access Logs via UI or CLI

↓

Inspect Logs for Errors or Info

↓

Identify Issue?

No→Wait or Retry

Yes↓

Apply Fix or Adjust Config

↓

Re-run Task or DAG

↓

Verify Logs Again

↓

END

This flow shows how Airflow tasks generate logs during execution, which you inspect to find issues, then fix and re-run tasks to troubleshoot.

Execution Sample

Apache Airflow

airflow tasks logs example_dag example_task 2024-06-01T00:00:00+00:00

This command fetches and displays logs for a specific task instance in Airflow to inspect its execution details.

Process Table

Step	Action	Command/Log Output	Result/Next Step
1	Trigger DAG run	airflow dags trigger example_dag	DAG run started, tasks queued
2	Task execution starts	Task example_task started at 2024-06-01T00:00:00	Logs begin recording task progress
3	Generate logs	INFO - Starting task INFO - Processing data ERROR - File not found: data.csv	Error detected in logs
4	Inspect logs	airflow tasks logs example_dag example_task 2024-06-01T00:00:00+00:00	Logs show missing file error
5	Identify issue	Missing data.csv file	Decide to fix missing file
6	Fix issue	Upload data.csv to expected location	File available for next run
7	Re-run task	airflow tasks run example_dag example_task 2024-06-01T00:00:00	Task runs successfully
8	Verify logs again	INFO - Starting task INFO - Processing data INFO - Task completed successfully	No errors, troubleshooting complete
9	End	-	Task and DAG run successful

💡 Task logs show error, fix applied, task re-run succeeds, confirming issue resolved

Status Tracker

Variable	Start	After Step 3	After Step 6	After Step 7	Final
Task Status	queued	failed (error in logs)	fixed (file uploaded)	success	success
Log Content	empty	error: File not found	error resolved	success messages	success messages

Key Moments - 3 Insights

Why do we check logs after task execution?

Logs show detailed task progress and errors, helping identify what went wrong (see execution_table step 3 and 4).

What does it mean if logs show 'File not found'?

It means the task failed because a required file was missing; this is the root cause to fix (see execution_table step 3).

Why re-run the task after fixing the issue?

To verify the fix works and the task completes successfully, confirmed by logs showing no errors (see execution_table steps 7 and 8).

Visual Quiz - 3 Questions

Look at the execution table, what error is shown in the logs at step 3?

A File not found: data.csv

B Database connection failed

C Task completed successfully

D Permission denied

💡 Hint

Check the 'Command/Log Output' column at step 3 in the execution_table

At which step does the task status change to 'success'?

A Step 3

B Step 6

C Step 7

D Step 4

💡 Hint

Look at the 'Task Status' row in variable_tracker after each step

If the missing file was not fixed, what would happen at step 7?

A Task would run successfully

B Task would fail again with the same error

C Logs would be empty

D Task would be skipped

💡 Hint

Refer to the error shown at step 3 and the fix applied at step 6 in execution_table

Concept Snapshot

Airflow logs record task execution details.
Use 'airflow tasks logs <dag> <task> <execution_date>' to view logs.
Inspect logs to find errors or info.
Fix issues based on logs, then re-run tasks.
Verify success by checking logs again.
Logs are key for troubleshooting Airflow tasks.

Full Transcript

In Airflow, when you run a DAG, each task generates logs showing what happened during execution. You can fetch these logs using the airflow CLI command for tasks logs. By reading the logs, you can spot errors like missing files or other issues. Once you find the problem, you fix it, for example by uploading a missing file. Then you re-run the task to check if the fix worked. Finally, you look at the logs again to confirm the task completed successfully. This process helps you troubleshoot and fix problems in your Airflow workflows.