Idempotent pipeline design in Apache Spark - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

What is the output of this Spark DataFrame operation?

Consider the following Spark code that reads a CSV, drops duplicates, and writes to a parquet file. What will be the count of rows in the output parquet if the input CSV has 100 rows with 10 duplicates?

Apache Spark

from pyspark.sql import SparkSession
spark = SparkSession.builder.appName('Test').getOrCreate()
df = spark.read.csv('input.csv', header=True, inferSchema=True)
df_clean = df.dropDuplicates()
df_clean.write.mode('overwrite').parquet('output.parquet')
output_df = spark.read.parquet('output.parquet')
print(output_df.count())

A 0

B 90

C 110

D 100

💡 Hint

Think about what dropDuplicates() does to the data.

The dropDuplicates() removes duplicate rows, so from 100 rows with 10 duplicates, 90 unique rows remain. The output parquet file will have 90 rows.

❓ data_output

intermediate

What is the resulting DataFrame after idempotent transformation?

Given a Spark DataFrame df with a column value containing integers, the following code applies a transformation twice. What is the content of df_transformed?

Apache Spark

df_transformed = df.withColumn('value', (df['value'] * 2)).withColumn('value', (df_transformed['value'] * 2))
df_transformed.show()

A Values are quadrupled (value * 4)

B Values are doubled (value * 2)

C Values remain unchanged

D Code raises an error

💡 Hint

Each withColumn applies the transformation on the current DataFrame.

Applying the doubling transformation twice multiplies the values by 4 (2 * 2). So values are quadrupled.

🔧 Debug

advanced

Identify the error in this idempotent pipeline step

This Spark pipeline step is intended to be idempotent by writing data only if the output path does not exist. What error will occur when running this code twice?

Apache Spark

import os
output_path = '/data/output'
if not os.path.exists(output_path):
    df.write.parquet(output_path)
else:
    print('Output exists, skipping write')

A SparkException due to output path existing

B FileNotFoundError on second run

C No error, pipeline is idempotent

D PermissionError when checking path

💡 Hint

Check what happens if the output path exists before writing.

The code checks if the output path exists before writing. If it exists, it skips writing. So no error occurs and the pipeline is idempotent.

🚀 Application

advanced

Which approach ensures idempotency in Spark streaming writes?

You want to write streaming data to a sink without duplicates on restart. Which approach below ensures idempotent writes?

A Use append mode with checkpointing enabled

B Use complete output mode without checkpointing

C Use overwrite mode without checkpointing

D Use foreachBatch with idempotent batch writes and checkpointing

💡 Hint

Think about how to avoid duplicates on batch retries.

Using foreachBatch with idempotent batch writes and checkpointing ensures that each batch is processed exactly once, avoiding duplicates on restart.

🧠 Conceptual

expert

Why is idempotency critical in data pipelines?

Which statement best explains the importance of idempotency in data pipelines?

A It allows pipelines to run multiple times without changing the final data state

B It guarantees data is encrypted during processing

C It ensures data is always processed in real-time

D It improves the speed of data processing by caching results

💡 Hint

Think about what happens if a pipeline runs twice with the same input.

Idempotency means running the pipeline multiple times does not change the final data beyond the first run, preventing duplicates or corruption.

Practice

1. What does it mean for a data pipeline to be idempotent in Apache Spark?

easy

A. It requires manual cleanup after each run.

B. It runs faster each time it is executed.

C. It only processes new data and ignores old data.

D. It produces the same output even if run multiple times with the same input.

Solution

1. Step 1: Understand the meaning of idempotency in data pipelines

   Idempotency means running the pipeline multiple times with the same input should not change the output after the first run.

2. Step 2: Relate to Apache Spark pipeline behavior

   In Spark, an idempotent pipeline ensures consistent results by handling duplicates and overwriting data safely.

3. Final Answer:

   It produces the same output even if run multiple times with the same input. -> Option D

4. Quick Check:

   Idempotent = Same output every run [OK]

Hint: Idempotent means repeat runs give same results [OK]

Common Mistakes:

• Thinking idempotent means faster runs
• Assuming it only processes new data
• Believing manual cleanup is needed

2. Which Spark write mode helps achieve idempotency by replacing existing data safely?

easy

A. append

B. ignore

C. overwrite

D. update

Solution

1. Step 1: Review Spark write modes

   Append adds new data without removing old data, overwrite replaces existing data, ignore skips writing if data exists, update is not a standard Spark write mode.

2. Step 2: Identify which mode replaces data for idempotency

   Overwrite mode replaces the existing data, ensuring the output is consistent after multiple runs.

3. Final Answer:

   overwrite -> Option C

4. Quick Check:

   Overwrite mode = Replace data safely [OK]

Hint: Overwrite mode replaces data for consistent output [OK]

Common Mistakes:

• Choosing append which adds duplicates
• Selecting ignore which skips writes
• Thinking update is a valid write mode

3. Given this Spark code snippet, what will be the output count after running twice?

df = spark.createDataFrame([(1, 'a'), (2, 'b'), (1, 'a')], ['id', 'value'])
dedup_df = df.dropDuplicates()
dedup_df.write.mode('overwrite').save('/tmp/data')
loaded_df = spark.read.load('/tmp/data')
print(loaded_df.count())

medium

A. 2

B. 0

C. 1

D. 3

Solution

1. Step 1: Understand the input data and deduplication

   The input DataFrame has 3 rows with one duplicate (id=1, value='a'). dropDuplicates() removes duplicates, leaving 2 unique rows.

2. Step 2: Analyze the write and read operations

   Using overwrite mode saves the deduplicated data, replacing any previous data. Reading it back will load 2 rows.

3. Final Answer:

   2 -> Option A

4. Quick Check:

   Deduplication + overwrite = 2 rows [OK]

Hint: Deduplicate then overwrite keeps unique rows only [OK]

Common Mistakes:

• Counting duplicates as separate rows
• Assuming append mode adds duplicates
• Ignoring overwrite effect on saved data

4. You have a Spark pipeline that appends data without deduplication. What issue can this cause when rerunning the pipeline?

medium

A. Data duplication leading to incorrect results

B. Pipeline runs faster each time

C. Data is automatically cleaned up

D. Output files are overwritten safely

Solution

1. Step 1: Understand append mode without deduplication

   Appending data adds new rows without removing duplicates, so rerunning the pipeline adds repeated data.

2. Step 2: Identify the impact on results

   Repeated runs cause duplicated data, which can lead to wrong analytics or reports.

3. Final Answer:

   Data duplication leading to incorrect results -> Option A

4. Quick Check:

   Append without deduplication = duplicates [OK]

Hint: Appending without deduplication causes duplicates [OK]

Common Mistakes:

• Thinking pipeline runs faster
• Assuming data cleans automatically
• Believing output files overwrite safely

5. You want to design an idempotent Spark pipeline that processes daily sales data. Which combination ensures idempotency when rerunning for the same day?
1. Read raw daily data
2. Remove duplicates
3. Write results to storage
Choose the best write mode and deduplication approach.

hard

A. Use append mode and no deduplication

B. Use overwrite mode and remove duplicates before writing

C. Use ignore mode and append duplicates

D. Use append mode and remove duplicates after writing

Solution

1. Step 1: Identify deduplication timing

   Removing duplicates before writing ensures only unique data is saved, preventing repeated entries.

2. Step 2: Choose write mode for idempotency

   Overwrite mode replaces existing data for the day, so rerunning does not add duplicates.

3. Final Answer:

   Use overwrite mode and remove duplicates before writing -> Option B

4. Quick Check:

   Deduplicate + overwrite = idempotent pipeline [OK]

Hint: Deduplicate first, then overwrite to ensure idempotency [OK]

Common Mistakes:

• Appending duplicates causing repeated data
• Removing duplicates after writing is too late
• Using ignore mode skips writing data