Log inspection and troubleshooting in Apache Airflow - Cheat Sheet & Quick Revision

Recall & Review

beginner

What is the primary purpose of log inspection in Airflow?

Log inspection helps you understand what happened during a task run, identify errors, and find clues to fix issues.

beginner

Where can you find task logs in the Airflow web UI?

In the Airflow web UI, click on a DAG run, then select a task instance, and click the 'Log' tab to see the logs for that task.

intermediate

What does a 'Task instance failed' message in logs usually indicate?

It means the task did not complete successfully, often due to an error in the code, missing resources, or external system failures.

intermediate

How can you access Airflow logs from the command line?

You can access logs by navigating to the logs folder on the Airflow server, usually under AIRFLOW_HOME/logs, or use the CLI command: airflow tasks logs <dag_id> <task_id> <execution_date>.

advanced

Why is it important to check both scheduler and worker logs when troubleshooting Airflow?

Scheduler logs show DAG scheduling and triggering info, while worker logs show task execution details. Checking both helps find where the problem lies.

Where do you find task logs in the Airflow web UI?

A Under the task instance's 'Log' tab

B In the DAG details page only

C In the Airflow configuration file

D In the database directly

Task logs are accessible by clicking on a task instance and then the 'Log' tab in the Airflow web UI.

What does a 'Task instance failed' log message usually mean?

A The task completed successfully

B The task encountered an error during execution

C The task did not run at all

D The task was skipped intentionally

A 'Task instance failed' message indicates the task ran but encountered an error.

Which Airflow component's logs help you understand DAG scheduling issues?

A Worker logs

B Database logs

C Webserver logs

D Scheduler logs

Scheduler logs show how DAGs are scheduled and triggered.

How can you view Airflow task logs from the command line?

A Using 'airflow tasks logs <dag_id> <task_id> <execution_date>'

B Using 'airflow webserver start'

C Using 'airflow scheduler status'

D Using 'airflow db check'

The CLI command 'airflow tasks logs' shows task logs for a specific DAG, task, and execution date.

Why check both scheduler and worker logs when troubleshooting?

A Scheduler logs show task execution details

B Worker logs show DAG scheduling info

C Both logs together help locate the problem source

D Only worker logs are useful

Scheduler logs show scheduling info; worker logs show task execution. Both are needed to find issues.

Explain how to find and interpret task logs in Airflow for troubleshooting.

Think about both UI and command line ways to see logs.

Describe why checking both scheduler and worker logs is important when troubleshooting Airflow issues.

Consider the roles of scheduler and worker in Airflow.