Log inspection and troubleshooting in Apache Airflow - Practice Problems & Coding Challenges

Challenge - 5 Problems

💻 Command Output

intermediate

Identify the output of the Airflow CLI log command

You run the command airflow tasks logs example_dag example_task 2024-06-01. What will this command output?

A The status summary of all tasks in <code>example_dag</code> for June 1, 2024.

B A list of all DAGs available in Airflow.

C An error message saying the task does not exist.

D The logs of the task <code>example_task</code> from the DAG <code>example_dag</code> for the execution date June 1, 2024.

💡 Hint

The command airflow tasks logs shows logs for a specific task instance.

The airflow tasks logs command fetches logs for a specific task in a DAG for a given execution date. It does not list DAGs or task statuses.

❓ Troubleshoot

intermediate

Troubleshoot missing logs in Airflow UI

You notice that the Airflow web UI shows No logs found for a task that ran successfully. What is the most likely cause?

A The task logs are stored remotely but the Airflow UI is not configured to access the remote storage.

B The task never ran, so no logs exist.

C The Airflow scheduler is down, so logs cannot be displayed.

D The DAG file has syntax errors preventing log generation.

💡 Hint

Check where Airflow stores logs and if the UI can access that location.

If logs are stored remotely (e.g., S3, GCS) but the UI is not configured with the right connection, it will show no logs even if the task ran.

❓ Configuration

advanced

Configure Airflow to store logs remotely on Amazon S3

Which configuration snippet correctly sets Airflow to store logs on Amazon S3?

A

[core]
remote_logging = True
remote_log_conn_id = MyS3Conn
remote_base_log_folder = s3://my-airflow-logs

B

[logging]
remote_logging = False
remote_log_conn_id = MyS3Conn
remote_base_log_folder = s3://my-airflow-logs

C

[logging]
remote_logging = True
remote_log_conn_id = MyS3Conn
remote_base_log_folder = s3://my-airflow-logs

D

[logging]
remote_logging = True
remote_log_conn_id = MyLocalConn
remote_base_log_folder = /var/log/airflow

💡 Hint

Remote logging settings belong under the [logging] section.

To enable remote logging to S3, set remote_logging = True under [logging], specify the S3 connection ID, and the S3 bucket path.

🔀 Workflow

advanced

Steps to troubleshoot a failing Airflow task using logs

What is the correct order of steps to troubleshoot a failing Airflow task using logs?

A 2,1,3,4

B 1,2,3,4

C 1,3,2,4

D 3,1,2,4

💡 Hint

Start by looking at logs, then find the error, then fix it.

The logical troubleshooting flow is to first check logs, find the error, understand the cause, then fix and rerun.

✅ Best Practice

expert

Best practice for managing Airflow logs in a production environment

Which option is the best practice for managing Airflow logs in a production environment with many DAGs and tasks?

A Use remote logging to a centralized storage like S3 or GCS and configure log rotation.

B Disable logging to save disk space and rely on task status only.

C Store logs locally on the Airflow worker nodes to reduce latency.

D Manually copy logs from workers to a shared folder daily.

💡 Hint

Think about scalability, reliability, and ease of access.

Remote logging to centralized storage with rotation is scalable, reliable, and makes logs accessible from anywhere.