Log inspection and troubleshooting in Apache Airflow - Mini Project: Build & Apply

Log inspection and troubleshooting

📖 Scenario: You are a data engineer managing workflows using Apache Airflow. Sometimes, tasks fail and you need to check the logs to find out what went wrong. This project will guide you through inspecting Airflow task logs and identifying errors.

🎯 Goal: Learn how to locate and read Airflow task logs to troubleshoot failures effectively.

📋 What You'll Learn

Access to Airflow logs directory

Basic understanding of Airflow task and DAG names

Ability to use command line tools to read log files

💡 Why This Matters

🌍 Real World

In real Airflow setups, logs help data engineers quickly find why a task failed and fix issues to keep data pipelines running smoothly.

💼 Career

Knowing how to inspect and troubleshoot Airflow logs is a key skill for data engineers and DevOps professionals managing workflow automation.

1

Identify the log file path

Create a variable called log_path and set it to the string "/usr/local/airflow/logs/example_dag/example_task/2024-06-01T00:00:00+00:00/1.log" which is the exact path to the log file for a specific Airflow task run.

Apache Airflow

# Create the log_path variable with the exact log file path
# Your code here

Need a hint?

The log path includes the DAG name, task name, execution date, and try number.

2

Set a keyword to search for errors

Create a variable called error_keyword and set it to the string "ERROR" which will be used to find error lines in the log file.

Apache Airflow

log_path = "/usr/local/airflow/logs/example_dag/example_task/2024-06-01T00:00:00+00:00/1.log"
# Create the error_keyword variable with the value "ERROR"
# Your code here

Need a hint?

Use uppercase "ERROR" to match typical log error lines.

3

Read the log file and filter error lines

Open the file at log_path, read all lines, and create a list called error_lines that contains only the lines which include the error_keyword string.

Apache Airflow

log_path = "/usr/local/airflow/logs/example_dag/example_task/2024-06-01T00:00:00+00:00/1.log"
error_keyword = "ERROR"
# Open the log file and filter lines containing error_keyword
# Your code here

Need a hint?

Use a list comprehension to filter lines containing the error keyword.

4

Display the error lines

Print the error_lines list to show all error messages found in the log file.

Apache Airflow

log_path = "/usr/local/airflow/logs/example_dag/example_task/2024-06-01T00:00:00+00:00/1.log"
error_keyword = "ERROR"
with open(log_path, 'r') as file:
    error_lines = [line.strip() for line in file if error_keyword in line]
# Print the error_lines list
# Your code here

Need a hint?

Use print(error_lines) to display the list of error lines.