Log inspection and troubleshooting in Apache Airflow - Commands & Configuration

Introduction

When something goes wrong in Airflow, you need to find out what happened. Logs show detailed messages from tasks and the scheduler. Inspecting logs helps you fix errors and keep workflows running smoothly.

When a task in your Airflow DAG fails and you want to see why.

When the Airflow scheduler is not triggering tasks as expected.

When you want to check the history of task runs and their status.

When debugging connection or configuration issues in Airflow.

When you want to monitor task progress and resource usage.

Commands

This command fetches and shows the logs of the task named 'example_task' in the DAG 'example_dag' for the run on June 1, 2024. It helps you see detailed output and errors from that task run.

Terminal

airflow tasks logs example_dag example_task 2024-06-01T00:00:00+00:00

Expected Output

2024-06-01 00:00:01,234 INFO - Starting task

2024-06-01 00:00:02,345 INFO - Task is running

2024-06-01 00:00:03,456 ERROR - Task failed due to missing file

Traceback (most recent call last):

  File "/usr/local/airflow/dags/example_task.py", line 10, in execute

    open('data.csv')

FileNotFoundError: [Errno 2] No such file or directory: 'data.csv'

This command lists all the runs of the DAG named 'example_dag'. It helps you find the run IDs and their status to know which runs to inspect further.

Terminal

airflow dags list-runs -d example_dag

Expected Output

run_id               state     execution_date

manual__2024-06-01    failed    2024-06-01 00:00:00+00:00

scheduled__2024-05-31 success   2024-05-31 00:00:00+00:00

This command shows the current state of the task 'example_task' in the DAG 'example_dag' for the specified run. It tells you if the task succeeded, failed, or is still running.

Terminal

airflow tasks state example_dag example_task 2024-06-01T00:00:00+00:00

Expected Output

failed

Key Concept

If you remember nothing else from this pattern, remember: task logs are your first and best source to understand why Airflow tasks fail or behave unexpectedly.

Common Mistakes

Trying to check logs for a task run that does not exist or has a wrong timestamp.

Airflow will not find logs and will show an error or empty output, causing confusion.

Use 'airflow dags list-runs' to find the correct execution date and run ID before checking logs.

Ignoring the error messages in the logs and only looking at task status.

Task status alone does not explain why a failure happened; logs contain the detailed error and traceback.

Always read the full logs to understand the root cause of failures.

Summary

Use 'airflow tasks logs' to see detailed output and errors from a specific task run.

Use 'airflow dags list-runs' to find all runs of a DAG and their statuses.

Use 'airflow tasks state' to quickly check the status of a task in a DAG run.

Practice

1. What is the main purpose of inspecting logs in Airflow tasks?

easy

A. To schedule new DAG runs

B. To change the task's code automatically

C. To understand what happened during task execution step-by-step

D. To delete old DAGs from Airflow

Solution

1. Step 1: Understand what logs represent in Airflow

   Logs record the detailed events and outputs of each task run, showing what happened inside the task.

2. Step 2: Identify the purpose of inspecting logs

   Inspecting logs helps to see the step-by-step execution and find errors or confirm success.

3. Final Answer:

   To understand what happened during task execution step-by-step -> Option C

4. Quick Check:

   Logs show task execution details = A [OK]

Hint: Logs tell the story of task steps and errors [OK]

Common Mistakes:

• Thinking logs change task code
• Confusing logs with scheduling
• Believing logs delete DAGs

2. Which command correctly shows logs for a task named process_data in DAG daily_report for date 2024-06-01?

easy

A. airflow tasks logs daily_report process_data 2024-06-01

B. airflow logs show daily_report process_data 2024-06-01

C. airflow show logs daily_report process_data 2024-06-01

D. airflow task logs daily_report process_data 2024-06-01

Solution

1. Step 1: Recall the correct airflow CLI syntax for logs

   The correct command is airflow tasks logs <dag_id> <task_id> <execution_date>.

2. Step 2: Match the command with given task and date

   airflow tasks logs daily_report process_data 2024-06-01 matches the syntax exactly with correct order and spelling.

3. Final Answer:

   airflow tasks logs daily_report process_data 2024-06-01 -> Option A

4. Quick Check:

   Correct airflow logs command = A [OK]

Hint: Use 'airflow tasks logs DAG task date' exactly [OK]

Common Mistakes:

• Swapping 'tasks' and 'logs' words
• Using 'show' or 'task' incorrectly
• Wrong command order

3. You run airflow tasks logs sales_report generate_summary 2024-06-02 and see an error about missing file. What does this log output indicate?

medium

A. The task succeeded without issues

B. The DAG did not run on that date

C. The Airflow scheduler is down

D. The task failed because it could not find a required file

Solution

1. Step 1: Analyze the log error message about missing file

   An error about a missing file means the task tried to access a file that does not exist or is not reachable.

2. Step 2: Understand the implication for task status

   This error causes the task to fail because it cannot complete its work without the file.

3. Final Answer:

   The task failed because it could not find a required file -> Option D

4. Quick Check:

   Missing file error means task failure = C [OK]

Hint: Missing file error means task failure [OK]

Common Mistakes:

• Assuming task succeeded despite error
• Confusing DAG run status with task logs
• Thinking scheduler status is shown in task logs

4. You try to view logs with airflow tasks logs my_dag my_task 2024-06-05 but get a 'No logs found' message. What is the most likely cause?

medium

A. The DAG is paused

B. The task has not run yet for that date

C. Airflow is not installed

D. The command syntax is incorrect

Solution

1. Step 1: Understand what 'No logs found' means in Airflow

   This message usually means no logs exist because the task has not executed for the given date.

2. Step 2: Check other options for plausibility

   The command syntax is correct, Airflow installed is unrelated, and paused DAGs still keep logs from past runs.

3. Final Answer:

   The task has not run yet for that date -> Option B

4. Quick Check:

   No logs means no task run yet = D [OK]

Hint: No logs means task not run yet for that date [OK]

Common Mistakes:

• Assuming command syntax error
• Thinking Airflow is missing
• Confusing paused DAG with no logs

5. You want to troubleshoot why your Airflow task data_cleanup in DAG weekly_maintenance fails intermittently. Which approach best helps you find the root cause using logs?

hard

A. Check logs for multiple recent runs to find patterns of failure

B. Delete all logs to force fresh logging

C. Restart Airflow scheduler without checking logs

D. Change the DAG schedule to run more often

Solution

1. Step 1: Understand the goal of troubleshooting intermittent failures

   Intermittent failures require comparing multiple runs to see when and why failures happen.

2. Step 2: Identify the best log inspection method

   Checking logs from several recent runs helps spot patterns or recurring errors causing failure.

3. Final Answer:

   Check logs for multiple recent runs to find patterns of failure -> Option A

4. Quick Check:

   Compare multiple logs to find failure patterns = B [OK]

Hint: Compare logs from several runs to spot failure patterns [OK]

Common Mistakes:

• Deleting logs loses important info
• Restarting scheduler ignores root cause
• Changing schedule doesn't fix errors