Idempotent task design in Apache Airflow - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

Why is idempotency important in Airflow tasks?

In Airflow, tasks may run multiple times due to retries or manual triggers. Why is designing tasks to be idempotent important?

A To prevent Airflow from scheduling tasks automatically

B To make tasks run faster by skipping all processing

C To allow tasks to run only once ever without retries

D To ensure tasks produce the same result and avoid side effects when run multiple times

💡 Hint

Think about what happens if a task runs more than once and changes data each time.

Idempotency ensures that running a task multiple times does not cause unintended changes or duplicate effects. This is crucial in Airflow where retries or manual reruns happen.

💻 Command Output

intermediate

Output of an Airflow task with idempotent file creation

Consider an Airflow PythonOperator task that creates a file only if it does not exist. What will be the output if the task runs twice?

Apache Airflow

import os
file_path = '/tmp/data.txt'
if not os.path.exists(file_path):
    with open(file_path, 'w') as f:
        f.write('Airflow data')
print('File created or already exists')

A

File created
File not created

B

File created or already exists
File created or already exists

C

File created
File created

D File not created

💡 Hint

Check what happens when the file already exists on the second run.

The code checks if the file exists before creating it. On the second run, the file exists, so it skips creation but still prints the message.

❓ Configuration

advanced

Idempotent database update in Airflow task

You want an Airflow task to update a database record only if a specific condition is met, avoiding duplicate updates on retries. Which SQL statement ensures idempotency?

A UPDATE users SET status='active' WHERE id=123 AND status!='active';

B UPDATE users SET status='active' WHERE id=123;

C INSERT INTO users (id, status) VALUES (123, 'active');

D DELETE FROM users WHERE id=123;

💡 Hint

Think about how to avoid changing the record if it already has the desired status.

Option A updates the record only if the status is not already 'active', preventing repeated updates on retries and ensuring idempotency.

❓ Troubleshoot

advanced

Troubleshooting non-idempotent Airflow task causing duplicate data

An Airflow task inserts rows into a table every time it runs, causing duplicates on retries. What is the best way to fix this?

A Modify the task to check if the row exists before inserting

B Increase the retry delay to avoid duplicates

C Disable retries for the task

D Run the task only once manually

💡 Hint

How can you prevent inserting the same data multiple times?

Checking for existing rows before inserting ensures the task does not create duplicates on retries, making it idempotent.

🔀 Workflow

expert

Designing an idempotent Airflow DAG with external API calls

You have an Airflow DAG that calls an external API to create resources. The API does not support idempotency keys. How can you design the DAG to avoid creating duplicate resources on retries?

A Ignore failures and manually clean duplicates later

B Increase the task timeout to avoid retries

C Store API response IDs in XCom and check before retrying to skip duplicate calls

D Use multiple tasks to call the API in parallel

💡 Hint

Think about how to remember what was created in previous attempts.

By storing created resource IDs in XCom, the task can check if the resource was already created and skip duplicate API calls, ensuring idempotency despite the API limitation.