FileSensor for file arrival detection in Apache Airflow - Cheat Sheet & Quick Revision

Recall & Review

beginner

What is a FileSensor in Apache Airflow?

A FileSensor is a special type of sensor operator in Airflow that waits for a file to appear at a specified location before allowing the workflow to continue.

beginner

How does FileSensor detect the arrival of a file?

FileSensor periodically checks the file system path you specify. When the file exists, it triggers the next task in the workflow.

beginner

Which parameter in FileSensor specifies the file path to watch?

The 'filepath' parameter is used to specify the exact path of the file that FileSensor should wait for.

intermediate

What happens if the file does not arrive within the timeout period in FileSensor?

If the file does not appear before the timeout, the FileSensor task fails and the DAG run is marked as failed unless retries are configured.

intermediate

How can you optimize FileSensor to reduce resource usage?

You can set the 'poke_interval' parameter to increase the time between checks, reducing CPU usage and load on the file system.

What does FileSensor in Airflow primarily do?

A Deletes files from a directory

B Processes file contents automatically

C Uploads files to cloud storage

D Waits for a file to appear before continuing

FileSensor waits for a specific file to appear before allowing the workflow to proceed.

Which parameter tells FileSensor which file to watch?

A fileName

B file_path

C filepath

D watch_file

The correct parameter name is 'filepath' to specify the file location.

What does the 'poke_interval' parameter control in FileSensor?

A The file permissions

B How often the sensor checks for the file

C The file size limit

D The maximum time to wait for the file

'poke_interval' sets how frequently the sensor checks for the file's presence.

What happens if the file does not appear before the timeout in FileSensor?

A The task fails and the DAG run fails

B The task succeeds anyway

C The file is created automatically

D The sensor ignores the timeout

If the file does not appear before timeout, the task fails and the DAG run is marked as failed.

Which of these is a good practice to reduce resource use with FileSensor?

A Set poke_interval to a higher value

B Set timeout to zero

C Use multiple FileSensors for the same file

D Disable retries

Increasing poke_interval reduces how often the sensor checks, saving resources.

Explain how FileSensor works in Airflow to detect file arrival.

Think about how Airflow waits for something before moving on.

Describe how to configure FileSensor to avoid high CPU usage during file waiting.

Consider how often the sensor checks for the file.