FileSensor for file arrival detection in Apache Airflow - Practice Problems & Coding Challenges

Challenge - 5 Problems

💻 Command Output

intermediate

What does this FileSensor detect?

Given the following Airflow FileSensor configuration, what file event is it waiting for?

Apache Airflow

file_sensor = FileSensor(
    task_id='wait_for_file',
    filepath='/data/input/data_ready.txt',
    poke_interval=30,
    timeout=600
)

A It waits until the file '/data/input/data_ready.txt' is modified before continuing.

B It waits until the file '/data/input/data_ready.txt' is deleted before continuing.

C It waits until the directory '/data/input/' is empty before continuing.

D It waits until the file '/data/input/data_ready.txt' exists before continuing.

💡 Hint

FileSensor checks for the presence of a file at the given path.

FileSensor waits for a file to appear at the specified filepath. It does not detect deletion or modification events.

❓ Configuration

intermediate

Choose the correct FileSensor timeout setting

You want the FileSensor to stop waiting after 5 minutes if the file does not appear. Which timeout value should you set?

A timeout=300

B timeout=5

C timeout=60

D timeout=0

💡 Hint

Timeout is in seconds.

Timeout is specified in seconds. 5 minutes equals 300 seconds.

🔀 Workflow

advanced

How does FileSensor behave with poke_interval?

If a FileSensor has poke_interval=10 and timeout=60, how many times will it check for the file before timing out?

A 6 times

B 10 times

C 60 times

D 1 time

💡 Hint

Divide timeout by poke_interval to find the number of checks.

The sensor checks every 10 seconds up to 60 seconds, so 60/10 = 6 checks.

❓ Troubleshoot

advanced

Why does FileSensor keep timing out without detecting the file?

You configured a FileSensor to watch '/tmp/data.csv' but it always times out. The file is created by another process. What could be a common cause?

A The poke_interval is set too high.

B The FileSensor does not support detecting files.

C The file is created in a different directory than '/tmp'.

D The timeout is set to zero.

💡 Hint

Check the exact file path the other process writes to.

If the file is created in a different directory or path than the one the sensor watches, it will never detect it and time out.

✅ Best Practice

expert

What is the best practice to avoid blocking Airflow workers with FileSensor?

FileSensors can block workers while waiting for files. Which approach is best to avoid this problem?

A Set poke_interval to 1 second to check very frequently.

B Use the 'mode' parameter set to 'reschedule' to free the worker during waiting.

C Increase timeout to a very high value to avoid task failure.

D Run FileSensor tasks on the scheduler instead of workers.

💡 Hint

The 'reschedule' mode releases the worker slot while waiting.

Setting mode='reschedule' makes the sensor task free the worker slot between pokes, improving resource usage.