ExternalTaskSensor for cross-DAG dependencies in Apache Airflow - Step-by-Step Execution

Process Flow - ExternalTaskSensor for cross-DAG dependencies

Start ExternalTaskSensor

↓

Check if external DAG run exists

↓

Check external task

v +-----> Back to Check↓

Sensor completes

↓

Trigger downstream tasks

↩v +-----> Back to Check

The ExternalTaskSensor waits for a task in another DAG to complete before continuing. It checks repeatedly until the external task finishes, then proceeds.

Execution Sample

Apache Airflow

external_task_sensor = ExternalTaskSensor(
    task_id='wait_for_task',
    external_dag_id='external_dag',
    external_task_id='external_task',
    mode='poke',
    poke_interval=30,
    timeout=600
)

This sensor waits for 'external_task' in 'external_dag' to finish, checking every 30 seconds, timing out after 10 minutes.

Process Table

Step	Check External DAG Run	External Task Status	Action	Sensor State
1	No run found	N/A	Wait and retry after 30s	Waiting
2	Run found	Task running	Wait and retry after 30s	Waiting
3	Run found	Task success	Complete sensor	Success

💡 Sensor completes when external task status is success.

Status Tracker

Variable	Start	After Step 1	After Step 2	After Step 3
external_dag_run_found	False	False	True	True
external_task_status	N/A	N/A	running	success
sensor_state	None	Waiting	Waiting	Success

Key Moments - 2 Insights

Why does the sensor keep waiting even if the external DAG run exists?

Because the external task is still running, not yet successful. The sensor waits until the task status is 'success' as shown in execution_table step 2.

What happens if the external DAG run is not found at all?

The sensor waits and retries checking after the poke interval, as shown in execution_table step 1.

Visual Quiz - 3 Questions

Look at the execution table, what is the sensor state at step 2?

A Waiting

B Success

C Failed

D Skipped

💡 Hint

Check the 'Sensor State' column at step 2 in the execution_table.

At which step does the sensor complete successfully?

A Step 1

B Step 2

C Step 3

D Never

💡 Hint

Look for 'Complete sensor' action and 'Success' state in the execution_table.

If the external task never succeeds, what will happen to the sensor state?

A It will become Success anyway

B It will keep Waiting until timeout

C It will fail immediately

D It will skip

💡 Hint

Refer to the sensor behavior in execution_table steps 1 and 2 and the timeout setting in the code sample.

Concept Snapshot

ExternalTaskSensor waits for a task in another DAG to finish.
It checks repeatedly (poke mode) every poke_interval seconds.
If external task is successful, sensor completes.
If no external DAG run or task not done, sensor waits and retries.
Timeout stops sensor after max wait time.

Full Transcript

The ExternalTaskSensor in Airflow is used to wait for a task in a different DAG to complete before continuing. It works by checking if the external DAG run exists and if the external task has succeeded. If the external DAG run is not found or the task is still running, the sensor waits and retries after a set interval. Once the external task is successful, the sensor completes and allows downstream tasks to proceed. This process helps coordinate workflows across multiple DAGs safely and reliably.