Handling schema changes in data pipelines in Apache Airflow - Interactive Code Practice

Practice - 5 Tasks

1fill in blank

easy

Complete the code to define a DAG with a daily schedule.

Apache Airflow

with DAG('data_pipeline', schedule_interval=[1], start_date=datetime(2024, 1, 1)) as dag:

Drag options to blanks, or click blank then click option'

A'@monthly'

B'@daily'

C'@weekly'

D'@hourly'

Attempts:

3 left

💡 Hint

Common Mistakes

Using '@hourly' when daily runs are needed.

Forgetting to put quotes around the schedule string.

The schedule_interval set to '@daily' runs the DAG once every day.

2fill in blank

medium

Complete the code to create a task that runs a Python function.

Apache Airflow

task = PythonOperator(task_id='process_data', python_callable=[1], dag=dag)

Drag options to blanks, or click blank then click option'

Aprocess_data_function

B'process_data_function'

Cprocess_data_function()

Dprocess_data_function()()

Attempts:

3 left

💡 Hint

Common Mistakes

Adding parentheses after the function name.

Passing the function name as a string.

The python_callable argument expects a function reference without parentheses.

3fill in blank

hard

Fix the error in the task dependency definition.

Apache Airflow

task1 [1] task2

Drag options to blanks, or click blank then click option'

A->

B><

C=>

D>>

Attempts:

3 left

💡 Hint

Common Mistakes

Using '=>' which is not valid in Airflow.

Using '->' which is not recognized.

The >> operator sets task1 to run before task2 in Airflow.

4fill in blank

hard

Fill both blanks to create a task that checks for a schema file before processing.

Apache Airflow

check_schema = [1](task_id='check_schema_file', filepath='schema.json', dag=dag)
process = [2](task_id='process_data', python_callable=process_data, dag=dag)

Drag options to blanks, or click blank then click option'

AFileSensor

BPythonOperator

CBashOperator

DDummyOperator

Attempts:

3 left

💡 Hint

Common Mistakes

Using BashOperator to check for files instead of FileSensor.

Using DummyOperator which does nothing.

FileSensor waits for the schema file, then PythonOperator runs the processing function.

5fill in blank

hard

Fill both blanks to create a dictionary comprehension that filters columns based on schema.

Apache Airflow

filtered_data = {col:val for col, val in data.items() if col [1] schema and val [2] None}

Drag options to blanks, or click blank then click option'

A:

Bin

Cis not

D==

Attempts:

3 left

💡 Hint

Common Mistakes

Using '=' instead of ':' in dict comprehension.

Using '==' instead of 'is not' for None check.

The comprehension creates a dict with columns and values where the column is in the schema and value is not None.