Delta Lake vs Parquet in Apache Spark - Hands-On Comparison

Comparing Delta Lake and Parquet with Apache Spark

📖 Scenario: You work as a data analyst at a company that stores large amounts of data. You want to understand the difference between two popular data storage formats: Delta Lake and Parquet. You will create simple data, save it in both formats, and then read and compare them using Apache Spark.

🎯 Goal: Build a small Apache Spark project that creates a DataFrame, saves it as Delta Lake and Parquet files, reads both back, and shows their contents to compare.

📋 What You'll Learn

Create a Spark DataFrame with sample data

Save the DataFrame as a Parquet file

Save the DataFrame as a Delta Lake table

Read both Parquet and Delta Lake data back

Display the contents of both to compare

💡 Why This Matters

🌍 Real World

Data engineers and analysts often choose between Parquet and Delta Lake for storing big data. Delta Lake adds features like ACID transactions and easier updates, which are important for reliable data pipelines.

💼 Career

Understanding how to use Delta Lake and Parquet with Apache Spark is valuable for roles in data engineering, data analysis, and big data management.

1

Create a Spark DataFrame with sample data

Create a Spark DataFrame called df with these exact rows: (1, 'Alice', 29), (2, 'Bob', 31), and (3, 'Charlie', 25). Use column names 'id', 'name', and 'age'.

Apache Spark

# Create the Spark DataFrame called df with the specified rows and columns
# Your code here

Hint

Use spark.createDataFrame(data, columns) to create the DataFrame.

2

Set file paths for Parquet and Delta Lake storage

Create two string variables: parquet_path set to '/tmp/parquet_data' and delta_path set to '/tmp/delta_data'.

Apache Spark

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName('DeltaVsParquet').getOrCreate()
data = [(1, 'Alice', 29), (2, 'Bob', 31), (3, 'Charlie', 25)]
columns = ['id', 'name', 'age']
df = spark.createDataFrame(data, columns)

# Create parquet_path and delta_path variables with the specified paths
# Your code here

Hint

Assign the exact strings to the variables as shown.

3

Save DataFrame as Parquet and Delta Lake, then read both back

Save df as a Parquet file to parquet_path using df.write.mode('overwrite').parquet(parquet_path). Then save df as a Delta Lake table to delta_path using df.write.format('delta').mode('overwrite').save(delta_path). Finally, read both back into DataFrames called parquet_df and delta_df using spark.read.parquet(parquet_path) and spark.read.format('delta').load(delta_path) respectively.

Apache Spark

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName('DeltaVsParquet').getOrCreate()
data = [(1, 'Alice', 29), (2, 'Bob', 31), (3, 'Charlie', 25)]
columns = ['id', 'name', 'age']
df = spark.createDataFrame(data, columns)

parquet_path = '/tmp/parquet_data'
delta_path = '/tmp/delta_data'

# Save df as Parquet and Delta Lake, then read both back into parquet_df and delta_df
# Your code here

Hint

Use write.mode('overwrite') to save and spark.read to load the data.

4

Display the contents of Parquet and Delta Lake DataFrames

Print the contents of parquet_df and delta_df using parquet_df.show() and delta_df.show().

Apache Spark

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName('DeltaVsParquet').getOrCreate()
data = [(1, 'Alice', 29), (2, 'Bob', 31), (3, 'Charlie', 25)]
columns = ['id', 'name', 'age']
df = spark.createDataFrame(data, columns)

parquet_path = '/tmp/parquet_data'
delta_path = '/tmp/delta_data'

df.write.mode('overwrite').parquet(parquet_path)
df.write.format('delta').mode('overwrite').save(delta_path)

parquet_df = spark.read.parquet(parquet_path)
delta_df = spark.read.format('delta').load(delta_path)

# Print the contents of parquet_df and delta_df
# Your code here

Hint

Use print() and .show() to display the DataFrames.

Practice

1. Which statement best describes the main difference between Delta Lake and Parquet?

easy

A. Delta Lake supports data updates and history tracking, while Parquet is read-only.

B. Parquet supports data updates, but Delta Lake does not.

C. Both Delta Lake and Parquet are read-only file formats.

D. Delta Lake is slower than Parquet for reading data.

Solution

1. Step 1: Understand Parquet's capabilities

   Parquet is a columnar file format optimized for fast read operations but does not support data updates or versioning.

2. Step 2: Understand Delta Lake's enhancements

   Delta Lake builds on Parquet by adding support for data updates, deletes, and tracking changes over time (history).

3. Final Answer:

   Delta Lake supports data updates and history tracking, while Parquet is read-only. -> Option A

4. Quick Check:

   Delta Lake adds update features to Parquet [OK]

Hint: Delta Lake = updates/history; Parquet = read-only [OK]

Common Mistakes:

• Thinking Parquet supports data updates
• Assuming Delta Lake is slower for reads
• Confusing both as identical formats

2. Which of the following is the correct way to write a DataFrame as a Delta Lake table in Apache Spark?

easy

A. df.write.format("parquet").saveAsTable("delta_table")

B. df.write.format("delta").save("path/to/table")

C. df.saveAsTable("delta", "path/to/table")

D. df.write.save("delta", "path/to/table")

Solution

1. Step 1: Recall the syntax for writing Delta tables

   To write a DataFrame as Delta, use df.write.format("delta").save(path).

2. Step 2: Check each option's correctness

   df.write.format("delta").save("path/to/table") matches the correct syntax. df.write.format("parquet").saveAsTable("delta_table") uses Parquet format incorrectly. The remaining options use invalid method signatures.

3. Final Answer:

   df.write.format("delta").save("path/to/table") -> Option B

4. Quick Check:

   Use .write.format("delta") to save Delta tables [OK]

Hint: Use .write.format("delta") to save Delta tables [OK]

Common Mistakes:

• Using saveAsTable with format mismatch
• Confusing Parquet and Delta formats
• Incorrect method chaining

3. Given the following code snippet, what will be the output count after the update?

from delta.tables import DeltaTable

# Load Delta table
deltaTable = DeltaTable.forPath(spark, "/data/delta/events")

# Update rows where eventType is 'click'
deltaTable.update(condition = "eventType = 'click'", set = {"eventType": "click_updated"})

# Count rows with updated eventType
count = spark.read.format("delta").load("/data/delta/events").filter("eventType = 'click_updated'").count()
print(count)

medium

A. Zero, because update does not change data

B. Total number of rows in the table

C. Number of rows where eventType was 'click' before update

D. Number of rows where eventType is 'click_updated' before update

Solution

1. Step 1: Understand the update operation on Delta table

   The update changes all rows with eventType 'click' to 'click_updated'.

2. Step 2: Count rows with eventType 'click_updated'

   After update, the count of 'click_updated' rows equals the original count of 'click' rows.

3. Final Answer:

   Number of rows where eventType was 'click' before update -> Option C

4. Quick Check:

   Update changes 'click' to 'click_updated' rows count [OK]

Hint: Updated rows count = original matching rows count [OK]

Common Mistakes:

• Assuming update does not change data
• Counting total rows instead of filtered
• Confusing pre- and post-update values

4. Identify the error in the following code snippet that tries to update a Parquet file:

parquetPath = "/data/parquet/events"
df = spark.read.format("parquet").load(parquetPath)
df.update(condition = "eventType = 'click'", set = {"eventType": "click_updated"})

medium

A. Parquet format does not support update operation

B. The condition syntax is incorrect

C. The set parameter should be a list, not a dictionary

D. The path variable is not defined

Solution

1. Step 1: Check if Parquet supports update

   Parquet is a static file format and does not support update operations directly.

2. Step 2: Identify the cause of error

   Calling update on a DataFrame loaded from Parquet will cause an error because this method is only available on Delta tables.

3. Final Answer:

   Parquet format does not support update operation -> Option A

4. Quick Check:

   Update works only on Delta, not Parquet [OK]

Hint: Update only works on Delta tables, not Parquet [OK]

Common Mistakes:

• Trying to update Parquet files directly
• Misusing update method on DataFrame
• Ignoring format limitations

5. You have a dataset stored in Parquet format that needs frequent updates and audit history. Which approach is best to handle this requirement efficiently?

hard

A. Store data in JSON files and manually track changes

B. Keep using Parquet and overwrite the entire dataset on each update

C. Use CSV format instead for easier updates

D. Convert the Parquet dataset to Delta Lake format and use Delta's update and versioning features

Solution

1. Step 1: Analyze update and audit needs

   Frequent updates and audit history require a format that supports transactions and versioning.

2. Step 2: Evaluate format options

   Delta Lake supports updates and keeps history, unlike Parquet, CSV, or JSON which lack these features natively.

3. Step 3: Choose the best approach

   Converting Parquet to Delta Lake enables efficient updates and audit tracking without full overwrites.

4. Final Answer:

   Convert the Parquet dataset to Delta Lake format and use Delta's update and versioning features -> Option D

5. Quick Check:

   Delta Lake = updates + history for Parquet data [OK]

Hint: Use Delta Lake for updates and history, not Parquet [OK]

Common Mistakes:

• Overwriting entire Parquet files repeatedly
• Choosing CSV or JSON for update-heavy data
• Ignoring Delta Lake benefits