Log inspection and troubleshooting in Apache Airflow - Commands & Configuration

Introduction

When something goes wrong in Airflow, you need to find out what happened. Logs show detailed messages from tasks and the scheduler. Inspecting logs helps you fix errors and keep workflows running smoothly.

When a task in your Airflow DAG fails and you want to see why.

When the Airflow scheduler is not triggering tasks as expected.

When you want to check the history of task runs and their status.

When debugging connection or configuration issues in Airflow.

When you want to monitor task progress and resource usage.

Commands

This command fetches and shows the logs of the task named 'example_task' in the DAG 'example_dag' for the run on June 1, 2024. It helps you see detailed output and errors from that task run.

Terminal

airflow tasks logs example_dag example_task 2024-06-01T00:00:00+00:00

Expected Output

2024-06-01 00:00:01,234 INFO - Starting task

2024-06-01 00:00:02,345 INFO - Task is running

2024-06-01 00:00:03,456 ERROR - Task failed due to missing file

Traceback (most recent call last):

  File "/usr/local/airflow/dags/example_task.py", line 10, in execute

    open('data.csv')

FileNotFoundError: [Errno 2] No such file or directory: 'data.csv'

This command lists all the runs of the DAG named 'example_dag'. It helps you find the run IDs and their status to know which runs to inspect further.

Terminal

airflow dags list-runs -d example_dag

Expected Output

run_id               state     execution_date

manual__2024-06-01    failed    2024-06-01 00:00:00+00:00

scheduled__2024-05-31 success   2024-05-31 00:00:00+00:00

This command shows the current state of the task 'example_task' in the DAG 'example_dag' for the specified run. It tells you if the task succeeded, failed, or is still running.

Terminal

airflow tasks state example_dag example_task 2024-06-01T00:00:00+00:00

Expected Output

failed

Key Concept

If you remember nothing else from this pattern, remember: task logs are your first and best source to understand why Airflow tasks fail or behave unexpectedly.

Common Mistakes

Trying to check logs for a task run that does not exist or has a wrong timestamp.

Airflow will not find logs and will show an error or empty output, causing confusion.

Use 'airflow dags list-runs' to find the correct execution date and run ID before checking logs.

Ignoring the error messages in the logs and only looking at task status.

Task status alone does not explain why a failure happened; logs contain the detailed error and traceback.

Always read the full logs to understand the root cause of failures.

Summary

Use 'airflow tasks logs' to see detailed output and errors from a specific task run.

Use 'airflow dags list-runs' to find all runs of a DAG and their statuses.

Use 'airflow tasks state' to quickly check the status of a task in a DAG run.