Atomic operations in pipelines in Apache Airflow - Step-by-Step Execution

Process Flow - Atomic operations in pipelines

Start Pipeline Task

↓

Begin Atomic Operation

↓

Perform Step 1

↓

Perform Step 2

↓

Check for Errors

↓

Commit

↓

End Atomic Operation

↓

Task Complete

The pipeline task starts an atomic operation where multiple steps run. If all succeed, changes commit; if any fail, rollback happens to keep data safe.

Execution Sample

Apache Airflow

with TaskGroup('atomic_task') as atomic_task:
    step1 = BashOperator(task_id='step1', bash_command='echo Step 1')
    step2 = BashOperator(task_id='step2', bash_command='echo Step 2')
    step1 >> step2

This Airflow task group runs two steps in order, simulating an atomic operation where both must succeed.

Process Table

Step	Action	Result	Next Step
1	Start atomic_task group	Task group initialized	Execute step1
2	Execute step1	Step 1 output: 'Step 1'	Execute step2
3	Execute step2	Step 2 output: 'Step 2'	Check for errors
4	Check for errors	No errors found	Commit changes
5	Commit changes	Changes saved successfully	End atomic_task
6	End atomic_task	Task group completed	Pipeline continues

💡 All steps succeeded, so atomic operation committed and pipeline continues.

Status Tracker

Variable	Start	After Step 1	After Step 2	Final
step1_status	None	Success	Success	Success
step2_status	None	None	Success	Success
atomic_task_status	None	Running	Running	Success

Key Moments - 2 Insights

Why do we need to check for errors after all steps instead of after each step?

Checking after all steps ensures the entire group succeeds before committing, as shown in execution_table row 4. This keeps the operation atomic.

What happens if step2 fails?

If step2 fails, the pipeline triggers a rollback instead of commit, preventing partial changes. This is implied by the 'Check for errors' decision in execution_table row 4.

Visual Quiz - 3 Questions

Look at the execution table, what is the output of step1 at step 2?

A Step 2 output: 'Step 2'

B Step 1 output: 'Step 1'

C Task group initialized

D Changes saved successfully

💡 Hint

Refer to execution_table row 2 under the 'Result' column.

At which step does the pipeline decide to commit changes?

A Step 4

B Step 3

C Step 5

D Step 6

💡 Hint

Look at execution_table row 4 where 'Check for errors' leads to 'Commit changes'.

If step1 failed, how would the atomic_task_status change after Step 1?

A Success

B Running

C Failed

D None

💡 Hint

Check variable_tracker for how statuses update after each step.

Concept Snapshot

Atomic operations in pipelines ensure multiple steps succeed together or fail together.
Use task groups or transactions to group steps.
Check for errors after all steps before committing.
Rollback if any step fails to keep data consistent.
This prevents partial updates and keeps pipelines reliable.

Full Transcript

Atomic operations in pipelines mean running several steps as one unit. If all steps succeed, changes are saved. If any step fails, all changes are undone. In Airflow, this can be done using task groups where steps run in order. After all steps, the pipeline checks for errors. If none, it commits changes; if errors exist, it rolls back. This keeps data safe and consistent. The execution table shows each step's action and result. Variables track step statuses. Key moments clarify why error checking happens after all steps and what happens on failure. The quiz tests understanding of step outputs, commit timing, and failure effects.