Poke Mode vs Reschedule Mode in Airflow: Key Differences and Usage

In Airflow, poke mode keeps the sensor task running and checking continuously at intervals, consuming a worker slot the whole time. Reschedule mode frees the worker slot between checks by rescheduling the task, making it more efficient for long waits.

Quick Comparison

Here is a quick side-by-side comparison of poke mode and reschedule mode in Airflow sensors.

Factor	Poke Mode	Reschedule Mode
Task Behavior	Keeps running and checks repeatedly	Stops and reschedules between checks
Worker Slot Usage	Consumes a worker slot continuously	Frees worker slot between checks
Resource Efficiency	Less efficient for long waits	More efficient for long waits
Default Mode	Yes, default for sensors	Optional mode enabled by setting mode='reschedule'
Use Case	Short wait times or quick checks	Long wait times or infrequent checks

Key Differences

Poke mode means the sensor task stays active and keeps checking the condition at fixed intervals without releasing the worker slot. This can block a worker for a long time if the sensor waits for a long event, reducing resource availability.

In contrast, reschedule mode makes the sensor task pause and release the worker slot after each check if the condition is not met. Airflow then reschedules the task to run again later. This approach is more resource-friendly because it frees workers to run other tasks during the wait.

While poke mode is simpler and the default, reschedule mode is recommended for sensors that wait for long periods or when worker slots are limited. You enable reschedule mode by setting mode='reschedule' in the sensor's parameters.

Code Comparison

Example of a sensor using poke mode (default) to wait for a file in S3:

python

from airflow.sensors.s3_key_sensor import S3KeySensor

sensor = S3KeySensor(
    task_id='poke_mode_sensor',
    bucket_name='my-bucket',
    bucket_key='path/to/file.txt',
    poke_interval=30,  # check every 30 seconds
    timeout=600       # timeout after 10 minutes
)

Output

The sensor task runs continuously, checking every 30 seconds until the file appears or timeout.

Reschedule Mode Equivalent

Equivalent sensor using reschedule mode to wait for the same file:

python

from airflow.sensors.s3_key_sensor import S3KeySensor

sensor = S3KeySensor(
    task_id='reschedule_mode_sensor',
    bucket_name='my-bucket',
    bucket_key='path/to/file.txt',
    poke_interval=30,  # check every 30 seconds
    timeout=600,      # timeout after 10 minutes
    mode='reschedule' # enable reschedule mode
)

Output

The sensor task releases the worker slot after each check and is rescheduled until the file appears or timeout.

When to Use Which

Choose poke mode when the wait time is short or the check is very quick, as it is simpler and uses fewer task state transitions.

Choose reschedule mode when the sensor might wait a long time or when worker slots are limited, as it frees up resources and improves cluster efficiency.

In general, prefer reschedule mode for most sensors to avoid blocking workers unnecessarily.

Key Takeaways

Poke mode keeps the sensor task running and occupies a worker slot continuously.

Reschedule mode frees the worker slot between checks by pausing and rescheduling the task.

Use poke mode for short, quick checks and reschedule mode for long waits or limited resources.

Enable reschedule mode by setting mode='reschedule' in the sensor parameters.

Reschedule mode improves resource efficiency and cluster scalability.