FileSensor for file arrival detection in Apache Airflow - Practice Problems & Coding Challenges

Challenge - 5 Problems

💻 Command Output

intermediate

What does this FileSensor detect?

Given the following Airflow FileSensor configuration, what file event is it waiting for?

Apache Airflow

file_sensor = FileSensor(
    task_id='wait_for_file',
    filepath='/data/input/data_ready.txt',
    poke_interval=30,
    timeout=600
)

A It waits until the file '/data/input/data_ready.txt' is modified before continuing.

B It waits until the file '/data/input/data_ready.txt' is deleted before continuing.

C It waits until the directory '/data/input/' is empty before continuing.

D It waits until the file '/data/input/data_ready.txt' exists before continuing.

💡 Hint

FileSensor checks for the presence of a file at the given path.

FileSensor waits for a file to appear at the specified filepath. It does not detect deletion or modification events.

❓ Configuration

intermediate

Choose the correct FileSensor timeout setting

You want the FileSensor to stop waiting after 5 minutes if the file does not appear. Which timeout value should you set?

A timeout=300

B timeout=5

C timeout=60

D timeout=0

💡 Hint

Timeout is in seconds.

Timeout is specified in seconds. 5 minutes equals 300 seconds.

🔀 Workflow

advanced

How does FileSensor behave with poke_interval?

If a FileSensor has poke_interval=10 and timeout=60, how many times will it check for the file before timing out?

A 6 times

B 10 times

C 60 times

D 1 time

💡 Hint

Divide timeout by poke_interval to find the number of checks.

The sensor checks every 10 seconds up to 60 seconds, so 60/10 = 6 checks.

❓ Troubleshoot

advanced

Why does FileSensor keep timing out without detecting the file?

You configured a FileSensor to watch '/tmp/data.csv' but it always times out. The file is created by another process. What could be a common cause?

A The poke_interval is set too high.

B The FileSensor does not support detecting files.

C The file is created in a different directory than '/tmp'.

D The timeout is set to zero.

💡 Hint

Check the exact file path the other process writes to.

If the file is created in a different directory or path than the one the sensor watches, it will never detect it and time out.

✅ Best Practice

expert

What is the best practice to avoid blocking Airflow workers with FileSensor?

FileSensors can block workers while waiting for files. Which approach is best to avoid this problem?

A Set poke_interval to 1 second to check very frequently.

B Use the 'mode' parameter set to 'reschedule' to free the worker during waiting.

C Increase timeout to a very high value to avoid task failure.

D Run FileSensor tasks on the scheduler instead of workers.

💡 Hint

The 'reschedule' mode releases the worker slot while waiting.

Setting mode='reschedule' makes the sensor task free the worker slot between pokes, improving resource usage.

Practice

1. What is the main purpose of using a FileSensor in Airflow?

easy

A. To upload files to a remote server

B. To delete a file after processing

C. To wait until a specific file appears before running the next task

D. To monitor CPU usage during task execution

Solution

1. Step 1: Understand FileSensor's role

   A FileSensor is designed to pause a task until a file is detected at a given path.

2. Step 2: Compare options with FileSensor behavior

   Only To wait until a specific file appears before running the next task describes waiting for a file's arrival, which matches FileSensor's purpose.

3. Final Answer:

   To wait until a specific file appears before running the next task -> Option C

4. Quick Check:

   FileSensor waits for file = A [OK]

Hint: FileSensor pauses until file exists, then continues [OK]

Common Mistakes:

• Confusing FileSensor with file deletion or upload
• Thinking it monitors system resources
• Assuming it triggers immediately without waiting

2. Which of the following is the correct way to create a FileSensor in Airflow to check for a file named data.csv in /tmp directory with poke_interval=30 and timeout=300?

easy

A. FileSensor(task_id='check_file', filepath='/tmp/data.csv', poke_interval=30, timeout=300)

B. FileSensor(task_id='check_file', path='/tmp/data.csv', poke=30, timeout=300)

C. FileSensor(task_id='check_file', filepath='/tmp/data.csv', interval=30, timeout=300)

D. FileSensor(task_id='check_file', filepath='/tmp/data.csv', poke_interval=30)

Solution

1. Step 1: Identify correct parameter names

   The FileSensor uses 'filepath' for the file path, 'poke_interval' for checking frequency, and 'timeout' for max wait time.

2. Step 2: Match options with correct parameters

   FileSensor(task_id='check_file', filepath='/tmp/data.csv', poke_interval=30, timeout=300) uses correct parameter names and values. FileSensor(task_id='check_file', filepath='/tmp/data.csv', poke_interval=30) misses 'timeout', which is optional but often needed.

3. Final Answer:

   FileSensor(task_id='check_file', filepath='/tmp/data.csv', poke_interval=30, timeout=300) -> Option A

4. Quick Check:

   Use 'filepath' and 'poke_interval' exactly [OK]

Hint: Use 'filepath' and 'poke_interval' parameters exactly [OK]

Common Mistakes:

• Using 'path' instead of 'filepath'
• Using 'poke' or 'interval' instead of 'poke_interval'
• Omitting required parameters

3. Given this Airflow task using FileSensor:

file_sensor = FileSensor(
    task_id='wait_for_file',
    filepath='/data/input.txt',
    poke_interval=10,
    timeout=60
)

What will happen if the file /data/input.txt does NOT appear within 60 seconds?

medium

A. The task will skip and continue

B. The task will fail with a timeout error

C. The task will keep waiting indefinitely

D. The task will succeed immediately

Solution

1. Step 1: Understand timeout behavior

   The 'timeout' parameter sets the max seconds the sensor waits before failing.

2. Step 2: Apply to given values

   With timeout=60, if the file doesn't appear in 60 seconds, the task fails with a timeout error.

3. Final Answer:

   The task will fail with a timeout error -> Option B

4. Quick Check:

   Timeout reached = task fails [OK]

Hint: Timeout causes failure if file missing too long [OK]

Common Mistakes:

• Assuming task succeeds without file
• Thinking sensor waits forever ignoring timeout
• Believing task skips automatically on timeout

4. You wrote this FileSensor task:

file_sensor = FileSensor(
    task_id='wait_file',
    filepath='/tmp/data.csv',
    poke_interval='30',
    timeout=300
)

But the task fails immediately with a TypeError. What is the likely cause?

medium

A. task_id cannot contain underscores

B. filepath must be a directory, not a file

C. timeout cannot be set to 300

D. poke_interval should be an integer, not a string

Solution

1. Step 1: Check parameter types

   poke_interval expects an integer number of seconds, not a string.

2. Step 2: Identify error cause

   Setting poke_interval='30' (string) causes TypeError because Airflow expects int.

3. Final Answer:

   poke_interval should be an integer, not a string -> Option D

4. Quick Check:

   poke_interval type must be int [OK]

Hint: poke_interval must be int, not quoted string [OK]

Common Mistakes:

• Passing poke_interval as string instead of int
• Misunderstanding filepath type requirements
• Thinking task_id restrictions cause error

5. You want to create a DAG that waits for multiple files file1.csv and file2.csv to appear in /data before starting processing. Which approach correctly uses FileSensor to achieve this?

hard

A. Create two FileSensor tasks for each file, then set a downstream task that depends on both sensors

B. Create one FileSensor with filepath='/data/file1.csv,/data/file2.csv'

C. Use a single FileSensor with filepath='/data' and wait for directory

D. Create one FileSensor for file1.csv and ignore file2.csv

Solution

1. Step 1: Understand FileSensor scope

   FileSensor waits for one file path; it cannot wait for multiple files at once.

2. Step 2: Design DAG dependencies

   Create two FileSensor tasks, each waiting for one file, then a downstream task that runs after both succeed.

3. Final Answer:

   Create two FileSensor tasks for each file, then set a downstream task that depends on both sensors -> Option A

4. Quick Check:

   Multiple files = multiple sensors + downstream task [OK]

Hint: Use one sensor per file, then join downstream [OK]

Common Mistakes:

• Trying to list multiple files in one filepath
• Waiting on directory instead of files
• Ignoring one of the required files