Bird
Raised Fist0
Apache Airflowdevops~5 mins

ExternalTaskSensor for cross-DAG dependencies in Apache Airflow - Time & Space Complexity

Choose your learning style10 modes available
LearnWhyDeepVisualTryChallengeProjectRecallTime
Time Complexity: ExternalTaskSensor for cross-DAG dependencies
O(n)
Understanding Time Complexity

When using ExternalTaskSensor in Airflow, we want to know how waiting for another DAG's task affects execution time.

We ask: How does the sensor's checking grow as the number of dependencies or wait time increases?

Scenario Under Consideration

Analyze the time complexity of this ExternalTaskSensor usage.


from airflow.sensors.external_task import ExternalTaskSensor

sensor = ExternalTaskSensor(
    task_id='wait_for_task',
    external_dag_id='other_dag',
    external_task_id='task_to_wait_for',
    poke_interval=60,
    timeout=3600
)

This sensor waits for a task in another DAG to complete by checking every 60 seconds, timing out after 1 hour.

Identify Repeating Operations

The sensor repeatedly checks if the external task is done.

  • Primary operation: Periodic database query to check task status.
  • How many times: Number of checks = timeout / poke_interval (e.g., 3600/60 = 60 times).
How Execution Grows With Input

The number of checks grows linearly with the timeout length and inversely with poke interval.

Input Size (timeout in seconds)Approx. Checks
600 (10 min)10 (if poke_interval=60s)
3600 (1 hour)60
7200 (2 hours)120

Pattern observation: More waiting time means more checks, growing in a straight line.

Final Time Complexity

Time Complexity: O(n)

This means the sensor's work grows directly with the number of checks it performs while waiting.

Common Mistake

[X] Wrong: "The sensor checks only once, so time doesn't grow with timeout."

[OK] Correct: The sensor keeps checking repeatedly until the task finishes or timeout, so more wait means more checks.

Interview Connect

Understanding how sensors wait and check helps you design efficient workflows and avoid unnecessary delays in real projects.

Self-Check

"What if we reduce the poke_interval to 10 seconds? How would the time complexity change?"

Practice

(1/5)
1. What is the main purpose of the ExternalTaskSensor in Airflow?
easy
A. To wait for a task in another DAG to complete before proceeding
B. To trigger a DAG run manually
C. To schedule DAGs at specific intervals
D. To monitor system resource usage

Solution

  1. Step 1: Understand the role of ExternalTaskSensor

    The ExternalTaskSensor is designed to pause a task until a specified task in a different DAG finishes successfully.

  2. Step 2: Compare with other options

    Triggering DAGs, scheduling, and monitoring resources are handled by other components, not ExternalTaskSensor.

  3. Final Answer:

    To wait for a task in another DAG to complete before proceeding -> Option A

  4. Quick Check:

    ExternalTaskSensor waits for cross-DAG task completion [OK]
Hint: ExternalTaskSensor waits for tasks in other DAGs [OK]
Common Mistakes:
  • Confusing ExternalTaskSensor with DAG trigger
  • Thinking it schedules DAGs
  • Assuming it monitors system resources
2. Which of the following is the correct way to instantiate an ExternalTaskSensor to wait for task 'task_a' in DAG 'dag_1'?
easy
A. ExternalTaskSensor(task_id='wait_task', dag='dag_1', task='task_a')
B. ExternalTaskSensor(task_id='wait_task', dag_id='dag_1', task_id='task_a')
C. ExternalTaskSensor(task_id='wait_task', external_dag='dag_1', external_task='task_a')
D. ExternalTaskSensor(task_id='wait_task', external_dag_id='dag_1', external_task_id='task_a')

Solution

  1. Step 1: Identify correct parameter names

    The ExternalTaskSensor requires external_dag_id and external_task_id to specify the DAG and task it waits for.

  2. Step 2: Match parameters with options

    Only ExternalTaskSensor(task_id='wait_task', external_dag_id='dag_1', external_task_id='task_a') uses the correct parameter names and syntax.

  3. Final Answer:

    ExternalTaskSensor(task_id='wait_task', external_dag_id='dag_1', external_task_id='task_a') -> Option D

  4. Quick Check:

    Use external_dag_id and external_task_id parameters [OK]
Hint: Use external_dag_id and external_task_id parameters [OK]
Common Mistakes:
  • Using wrong parameter names like dag_id or external_dag
  • Confusing task_id with external_task_id
  • Omitting required parameters
3. Given this snippet in DAG 'dag_2':
wait_for_task = ExternalTaskSensor(
    task_id='wait_for_task_a',
    external_dag_id='dag_1',
    external_task_id='task_a',
    mode='reschedule',
    poke_interval=30
)

wait_for_task.execute(context={})

What happens if task_a in dag_1 has not completed yet?
medium
A. The sensor waits and retries every 30 seconds without blocking a worker slot
B. The sensor immediately fails with an error
C. The sensor completes successfully without waiting
D. The sensor blocks a worker slot and retries every 30 seconds

Solution

  1. Step 1: Understand mode='reschedule'

    In 'reschedule' mode, the sensor frees the worker slot between pokes, retrying after the poke_interval.

  2. Step 2: Analyze behavior when task not complete

    The sensor will keep checking every 30 seconds without blocking a worker slot until the external task finishes.

  3. Final Answer:

    The sensor waits and retries every 30 seconds without blocking a worker slot -> Option A

  4. Quick Check:

    mode='reschedule' frees worker slot during wait [OK]
Hint: mode='reschedule' frees worker slot during wait [OK]
Common Mistakes:
  • Confusing 'reschedule' with 'poke' mode
  • Assuming immediate failure if task incomplete
  • Thinking sensor completes without waiting
4. You wrote this ExternalTaskSensor:
wait_task = ExternalTaskSensor(
    task_id='wait_task',
    external_dag_id='dag_1',
    external_task_id='task_a',
    timeout=300
)

But the sensor never finishes even after task_a completes. What is the likely cause?
medium
A. The timeout is too high
B. The sensor's execution_date does not match the external task's execution date
C. The task_id is incorrect
D. The sensor is missing poke_interval

Solution

  1. Step 1: Understand execution_date matching

    ExternalTaskSensor waits for the external task with the same execution_date by default.

  2. Step 2: Identify mismatch issue

    If the sensor's execution_date differs from the external task's, it will wait forever even if the external task completed.

  3. Final Answer:

    The sensor's execution_date does not match the external task's execution date -> Option B

  4. Quick Check:

    Execution dates must match for ExternalTaskSensor to succeed [OK]
Hint: Check execution_date alignment between DAGs [OK]
Common Mistakes:
  • Assuming timeout controls waiting for task completion
  • Confusing task_id with external_task_id
  • Thinking poke_interval absence causes hang
5. You want to create a cross-DAG dependency where task_b in dag_2 waits for task_a in dag_1 to succeed, but only for the same execution date. Which configuration ensures this behavior and avoids deadlocks?
hard
A. Use ExternalTaskSensor with external_dag_id='dag_1', external_task_id='task_a', and execution_date_fn=lambda x: x - timedelta(days=1)
B. Use ExternalTaskSensor with external_dag_id='dag_1', external_task_id='task_a', and execution_date_fn=lambda x: x
C. Use ExternalTaskSensor with external_dag_id='dag_1', external_task_id='task_a', and default execution_date matching
D. Use ExternalTaskSensor with external_dag_id='dag_1', external_task_id='task_a', and mode='poke' without execution_date control

Solution

  1. Step 1: Understand default execution_date behavior

    By default, ExternalTaskSensor waits for the external task with the same execution_date, ensuring correct cross-DAG sync.

  2. Step 2: Avoid execution_date shifts

    Using execution_date_fn to shift dates can cause deadlocks or missed dependencies.

  3. Step 3: Consider mode

    mode='poke' works but does not affect execution_date matching; default is fine here.

  4. Final Answer:

    Use ExternalTaskSensor with external_dag_id='dag_1', external_task_id='task_a', and default execution_date matching -> Option C

  5. Quick Check:

    Default execution_date matching avoids deadlocks [OK]
Hint: Default execution_date matching avoids deadlocks [OK]
Common Mistakes:
  • Shifting execution_date causing missed dependencies
  • Ignoring execution_date alignment
  • Misusing mode without fixing execution_date
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard