Idempotent pipeline design in Apache Spark - Step-by-Step Execution

Concept Flow - Idempotent pipeline design

Start Pipeline

↓

Check if output exists

↓

Skip write

↓

End Pipeline

↓

End Pipeline

The pipeline first checks if the output data already exists. If yes, it skips processing to avoid duplication. If no, it processes and writes the output, ensuring the pipeline can run multiple times without side effects.

Execution Sample

Apache Spark

from pyspark.sql import SparkSession
spark = SparkSession.builder.getOrCreate()
table_name = 'output_table'
if not spark.catalog.tableExists(table_name):
    df = spark.read.csv('/tmp/input.csv', header=True, inferSchema=True)
    df.write.mode('overwrite').saveAsTable(table_name)

This code checks if the output table exists before processing input data and writing output, ensuring idempotency.

Execution Table

Step	Action	Condition	Result	Next Step
1	Check if output table exists	output_table exists?	False	Process data
2	Read input CSV	N/A	DataFrame loaded	Write output
3	Write output with overwrite mode	N/A	Table 'output_table' saved	End pipeline
4	Pipeline ends	N/A	Pipeline completed successfully	N/A

💡 Pipeline stops after writing output once, ensuring no duplicate writes on rerun.

Variable Tracker

Variable	Start	After Step 1	After Step 2	After Step 3	Final
table_name	'output_table'	'output_table'	'output_table'	'output_table'	'output_table'
output_exists	N/A	False	False	False	False
df	N/A	N/A	DataFrame loaded	DataFrame loaded	DataFrame loaded

Key Moments - 2 Insights

Why do we check if the output exists before processing?

Checking output existence prevents re-processing and overwriting data unnecessarily, as shown in execution_table step 1 where the condition decides the pipeline path.

What does 'overwrite' mode do when writing output?

'Overwrite' mode replaces existing output safely, ensuring the pipeline can run multiple times without accumulating duplicates, as seen in step 3.

Visual Quiz - 3 Questions

Look at the execution_table at step 1, what happens if the output table exists?

A The pipeline processes data and writes output anyway

B The pipeline skips processing and writing output

C The pipeline throws an error

D The pipeline deletes the output and starts fresh

💡 Hint

Refer to execution_table step 1 where the condition output_table exists? leads to skipping processing if True.

According to variable_tracker, what is the value of 'output_exists' after step 1?

A False

B True

C N/A

D Undefined

💡 Hint

Check variable_tracker row for 'output_exists' after Step 1 column.

If we remove the output existence check, what would likely happen on rerun?

A The pipeline would fail to write output

B The pipeline would skip processing

C The pipeline would process and overwrite output every time

D The pipeline would run faster

💡 Hint

Consider the role of the existence check in execution_table step 1 and its effect on processing.

Concept Snapshot

Idempotent pipeline design:
- Check if output exists before processing
- Skip processing if output is present
- Use overwrite mode to safely write output
- Ensures safe reruns without duplicates
- Key for reliable, repeatable data pipelines

Full Transcript

Idempotent pipeline design means building data pipelines that can run multiple times without causing duplicate data or errors. The pipeline first checks if the output data already exists. If it does, the pipeline skips processing and writing to avoid duplication. If not, it processes the input data and writes the output using overwrite mode. This approach ensures that rerunning the pipeline does not cause side effects and keeps data consistent. The execution table shows each step: checking output existence, reading input, writing output, and ending. Variable tracking shows how key variables change during execution. This design is important for reliable and repeatable data workflows.

Practice

1. What does it mean for a data pipeline to be idempotent in Apache Spark?

easy

A. It requires manual cleanup after each run.

B. It runs faster each time it is executed.

C. It only processes new data and ignores old data.

D. It produces the same output even if run multiple times with the same input.

Solution

1. Step 1: Understand the meaning of idempotency in data pipelines

   Idempotency means running the pipeline multiple times with the same input should not change the output after the first run.

2. Step 2: Relate to Apache Spark pipeline behavior

   In Spark, an idempotent pipeline ensures consistent results by handling duplicates and overwriting data safely.

3. Final Answer:

   It produces the same output even if run multiple times with the same input. -> Option D

4. Quick Check:

   Idempotent = Same output every run [OK]

Hint: Idempotent means repeat runs give same results [OK]

Common Mistakes:

• Thinking idempotent means faster runs
• Assuming it only processes new data
• Believing manual cleanup is needed

2. Which Spark write mode helps achieve idempotency by replacing existing data safely?

easy

A. append

B. ignore

C. overwrite

D. update

Solution

1. Step 1: Review Spark write modes

   Append adds new data without removing old data, overwrite replaces existing data, ignore skips writing if data exists, update is not a standard Spark write mode.

2. Step 2: Identify which mode replaces data for idempotency

   Overwrite mode replaces the existing data, ensuring the output is consistent after multiple runs.

3. Final Answer:

   overwrite -> Option C

4. Quick Check:

   Overwrite mode = Replace data safely [OK]

Hint: Overwrite mode replaces data for consistent output [OK]

Common Mistakes:

• Choosing append which adds duplicates
• Selecting ignore which skips writes
• Thinking update is a valid write mode

3. Given this Spark code snippet, what will be the output count after running twice?

df = spark.createDataFrame([(1, 'a'), (2, 'b'), (1, 'a')], ['id', 'value'])
dedup_df = df.dropDuplicates()
dedup_df.write.mode('overwrite').save('/tmp/data')
loaded_df = spark.read.load('/tmp/data')
print(loaded_df.count())

medium

A. 2

B. 0

C. 1

D. 3

Solution

1. Step 1: Understand the input data and deduplication

   The input DataFrame has 3 rows with one duplicate (id=1, value='a'). dropDuplicates() removes duplicates, leaving 2 unique rows.

2. Step 2: Analyze the write and read operations

   Using overwrite mode saves the deduplicated data, replacing any previous data. Reading it back will load 2 rows.

3. Final Answer:

   2 -> Option A

4. Quick Check:

   Deduplication + overwrite = 2 rows [OK]

Hint: Deduplicate then overwrite keeps unique rows only [OK]

Common Mistakes:

• Counting duplicates as separate rows
• Assuming append mode adds duplicates
• Ignoring overwrite effect on saved data

4. You have a Spark pipeline that appends data without deduplication. What issue can this cause when rerunning the pipeline?

medium

A. Data duplication leading to incorrect results

B. Pipeline runs faster each time

C. Data is automatically cleaned up

D. Output files are overwritten safely

Solution

1. Step 1: Understand append mode without deduplication

   Appending data adds new rows without removing duplicates, so rerunning the pipeline adds repeated data.

2. Step 2: Identify the impact on results

   Repeated runs cause duplicated data, which can lead to wrong analytics or reports.

3. Final Answer:

   Data duplication leading to incorrect results -> Option A

4. Quick Check:

   Append without deduplication = duplicates [OK]

Hint: Appending without deduplication causes duplicates [OK]

Common Mistakes:

• Thinking pipeline runs faster
• Assuming data cleans automatically
• Believing output files overwrite safely

5. You want to design an idempotent Spark pipeline that processes daily sales data. Which combination ensures idempotency when rerunning for the same day?
1. Read raw daily data
2. Remove duplicates
3. Write results to storage
Choose the best write mode and deduplication approach.

hard

A. Use append mode and no deduplication

B. Use overwrite mode and remove duplicates before writing

C. Use ignore mode and append duplicates

D. Use append mode and remove duplicates after writing

Solution

1. Step 1: Identify deduplication timing

   Removing duplicates before writing ensures only unique data is saved, preventing repeated entries.

2. Step 2: Choose write mode for idempotency

   Overwrite mode replaces existing data for the day, so rerunning does not add duplicates.

3. Final Answer:

   Use overwrite mode and remove duplicates before writing -> Option B

4. Quick Check:

   Deduplicate + overwrite = idempotent pipeline [OK]

Hint: Deduplicate first, then overwrite to ensure idempotency [OK]

Common Mistakes:

• Appending duplicates causing repeated data
• Removing duplicates after writing is too late
• Using ignore mode skips writing data