Atomic operations in pipelines in Apache Airflow - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

Understanding Atomicity in Airflow Tasks

Which statement best describes an atomic operation in an Airflow pipeline task?

A A task that completes fully or does not change the system state at all.

B A task that can be split into multiple smaller tasks for parallel execution.

C A task that runs only once regardless of retries or failures.

D A task that always runs on the same worker node.

💡 Hint

Think about what it means for a task to be indivisible and consistent.

An atomic operation means the task either completes fully or leaves no partial changes, ensuring data consistency.

💻 Command Output

intermediate

Output of Task Instance State After Failure and Retry

Given an Airflow task configured with retries=1 and retry_delay=5 minutes, what will be the final state of the task instance if it fails once and then succeeds on retry?

Apache Airflow

from airflow.models import TaskInstance
from airflow.utils.state import State

# Simulate task instance
 ti = TaskInstance(task=None, execution_date=None)

# Initial failure
 ti.set_state(State.FAILED)

# Retry attempt
 ti.set_state(State.SUCCESS)

A State.SKIPPED

B State.FAILED

C State.UP_FOR_RETRY

D State.SUCCESS

💡 Hint

Consider the final state after a successful retry.

The task instance state is SUCCESS after it succeeds on retry, even if it failed initially.

❓ Configuration

advanced

Ensuring Atomicity with XCom in Airflow

Which Airflow task configuration ensures that XCom push operations are atomic and do not cause partial data writes?

A Disable XCom push and use external database for communication.

B Use task instance context manager with commit after XCom push.

C Set retries=0 to avoid multiple XCom pushes.

D Use multiple XCom pushes in parallel tasks.

💡 Hint

Think about how to commit changes atomically within a task.

Using the task instance context manager and committing after XCom push ensures atomic writes.

❓ Troubleshoot

advanced

Diagnosing Partial Data Writes in Airflow Pipeline

An Airflow pipeline task writes data to a database but sometimes leaves partial data after failure. What is the most likely cause?

A The task does not use transactions to wrap database writes.

B The task uses XCom to pass data between tasks.

C The task retries are set too high causing duplicate writes.

D The task runs on multiple workers simultaneously.

💡 Hint

Consider how database writes can be atomic or not.

Without transactions, partial writes can occur if the task fails mid-operation.

🔀 Workflow

expert

Order of Operations for Atomic Pipeline Execution

Arrange the following steps in the correct order to ensure atomic execution of a data processing pipeline task in Airflow:

1. Commit database transaction
2. Process data
3. Start database transaction
4. Push XCom result

A 3,1,2,4

B 1,3,2,4

C 3,2,1,4

D 2,1,3,4

💡 Hint

Think about starting a transaction before processing and committing after all steps.

The correct order is to start the transaction, process data, push results, then commit to ensure atomicity.