Timeout in Airflow Sensor: What It Is and How It Works

In Airflow, timeout in a sensor defines the maximum time in seconds the sensor will wait for its condition before failing. If the sensor does not meet its condition within this time, it stops waiting and marks the task as failed.

How It Works

Think of an Airflow sensor as a watchful guard waiting for a specific event, like a file arriving or a database update. The timeout is like the guard's patience limit — it tells the sensor how long to keep watching before giving up.

When a sensor starts, it checks its condition repeatedly at intervals defined by poke_interval. If the condition is met, the sensor task succeeds and moves on. But if the condition isn't met within the timeout period, the sensor stops checking and fails the task. This prevents the sensor from waiting forever and helps manage workflow timing.

Example

This example shows a sensor waiting for a file to appear in a directory. The timeout is set to 60 seconds, so the sensor will fail if the file doesn't show up within one minute.

python

from airflow import DAG
from airflow.sensors.filesystem import FileSensor
from airflow.utils.dates import days_ago

with DAG(dag_id='file_sensor_timeout_example', start_date=days_ago(1), schedule_interval=None) as dag:
    wait_for_file = FileSensor(
        task_id='wait_for_file',
        filepath='/tmp/myfile.txt',
        poke_interval=10,  # check every 10 seconds
        timeout=60  # fail if not found in 60 seconds
    )

When to Use

Use timeout in sensors when you want to limit how long your workflow waits for an external event. This is useful to avoid stuck tasks that block your pipeline indefinitely.

For example, if you expect a file to arrive within a certain time window, setting a timeout ensures your workflow fails fast if the file is missing, allowing you to handle the error or alert someone.

Timeouts are also helpful when waiting for external systems that might be down or slow, so your workflow can recover or retry later instead of hanging.

Key Points

• Timeout sets the max wait time in seconds for a sensor.
• If the condition isn't met before timeout, the sensor task fails.
• It prevents infinite waiting and helps manage workflow timing.
• Works together with poke_interval, which controls how often the sensor checks the condition.

Key Takeaways

Timeout limits how long a sensor waits before failing in Airflow.

It helps avoid stuck tasks by stopping sensors that wait too long.

Set timeout based on expected wait times for external events.

Timeout works with poke_interval to control sensor checking frequency.