Handling schema changes in data pipelines in Apache Airflow - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

Why use schema validation in Airflow data pipelines?

In data pipelines managed by Airflow, why is it important to validate schema changes before processing data?

A To prevent pipeline failures caused by unexpected data format changes

B To speed up the execution of Airflow DAGs by skipping data checks

C To automatically fix data errors without human intervention

D To reduce the number of tasks in the DAG by merging schema checks with data loading

💡 Hint

Think about what happens if the data format changes unexpectedly.

Validating schema helps catch unexpected changes early, preventing pipeline crashes and data corruption.

💻 Command Output

intermediate

Output of Airflow task with schema mismatch

What will be the output when an Airflow PythonOperator task runs a function that raises a ValueError due to schema mismatch?

Apache Airflow

def check_schema(data):
    if 'id' not in data:
        raise ValueError('Missing id field')

check_schema({'name': 'Alice'})

A Task retries automatically without error

B Task succeeds and logs 'Schema valid'

C Task skips execution and marks as success

D Task fails with ValueError: Missing id field

💡 Hint

What happens when a Python function raises an exception inside an Airflow task?

Raising ValueError causes the task to fail and Airflow logs the error.

🔀 Workflow

advanced

Best workflow to handle schema changes in Airflow

Which Airflow workflow best handles schema changes in a data pipeline to minimize downtime?

A Add a schema validation task before data processing and alert on failures

B Skip schema validation and rely on downstream tasks to handle errors

C Disable all schema checks to speed up pipeline execution

D Run data processing tasks first, then validate schema after completion

💡 Hint

Consider when to catch schema issues to avoid wasting resources.

Validating schema before processing prevents wasted work and allows quick alerts for fixes.

❓ Troubleshoot

advanced

Troubleshooting schema mismatch errors in Airflow logs

You see this error in Airflow logs: KeyError: 'user_id'. What is the most likely cause?

A The Airflow webserver is not running, so logs are incomplete

B The input data is missing the 'user_id' field expected by the task

C The DAG file has syntax errors causing task failures

D The Airflow scheduler is down and cannot assign tasks

💡 Hint

KeyError usually means a missing dictionary key in Python.

The error indicates the code tried to access a missing field in the input data.

✅ Best Practice

expert

Handling evolving schemas in Airflow pipelines

What is the best practice to handle evolving data schemas in Airflow pipelines to ensure smooth updates?

A Ignore schema changes and fix errors as they appear during pipeline runs

B Hardcode schema fields in tasks and update DAGs only when schema changes

C Implement schema versioning and use conditional branching in DAGs to process different versions

D Disable schema validation to avoid pipeline failures during schema evolution

💡 Hint

Think about how to support multiple schema versions without breaking pipelines.

Schema versioning with conditional DAG logic allows pipelines to adapt to changes smoothly.