Log inspection and troubleshooting in Apache Airflow - Time & Space Complexity

Time Complexity: Log inspection and troubleshooting

O(n * m)

Understanding Time Complexity

When we inspect logs in Airflow, we want to know how the time to find issues grows as logs get bigger.

We ask: How does the effort to search logs change when there are more log entries?

Scenario Under Consideration

Analyze the time complexity of the following Airflow log inspection code.

from airflow.models import TaskInstance

def inspect_logs(task_instances):
    errors = []
    for ti in task_instances:
        log = ti.log.read()
        if 'ERROR' in log:
            errors.append(ti.task_id)
    return errors
    

This code checks each task instance's log for the word "ERROR" and collects task IDs with errors.

Identify Repeating Operations

Look at what repeats in the code.

• Primary operation: Looping over all task instances and reading their logs.
• How many times: Once for each task instance in the input list.

How Execution Grows With Input

As the number of task instances grows, the time to check logs grows too.

Input Size (n)	Approx. Operations
10	10 log reads and checks
100	100 log reads and checks
1000	1000 log reads and checks

Pattern observation: The work grows directly with the number of task instances.

Final Time Complexity

Time Complexity: O(n * m)

This means the time to inspect logs grows linearly with the number of task instances and the size of each log.

Common Mistake

[X] Wrong: "Checking logs for errors is instant no matter how many tasks there are."

[OK] Correct: Each task's log must be read and checked, so more tasks mean more work and more time.

Interview Connect

Understanding how log inspection time grows helps you design better monitoring and troubleshooting tools in real projects.

Self-Check

"What if we cached logs after the first read? How would that change the time complexity?"