Delta Lake vs Parquet in Apache Spark - Practice Questions

Challenge - 5 Problems

🧠 Conceptual

intermediate

Difference in Data Update Capability

Which statement correctly describes the difference between Delta Lake and Parquet regarding data updates?

A Delta Lake supports ACID transactions allowing updates and deletes; Parquet files are immutable and do not support direct updates.

B Parquet supports ACID transactions allowing updates and deletes; Delta Lake files are immutable and do not support direct updates.

C Both Delta Lake and Parquet support direct updates and deletes without any additional tools.

D Neither Delta Lake nor Parquet support updates; both require rewriting entire datasets for changes.

💡 Hint

Think about which format allows safe changes to existing data without rewriting everything.

Delta Lake adds a transaction log enabling ACID transactions, so you can update or delete data safely. Parquet files are just data files and are immutable, so you cannot update them directly.

❓ data_output

intermediate

Output of Reading Delta vs Parquet

Given a Delta Lake table and a Parquet file with the same data, what will be the output difference when reading them in Spark?

Apache Spark

from pyspark.sql import SparkSession
spark = SparkSession.builder.getOrCreate()
delta_df = spark.read.format('delta').load('/path/to/delta_table')
parquet_df = spark.read.parquet('/path/to/parquet_file')
delta_count = delta_df.count()
parquet_count = parquet_df.count()
print(delta_count, parquet_count)

A parquet_count will be larger because Parquet stores duplicates by default.

B Both delta_count and parquet_count will be the same if data is identical.

C delta_count will always be larger than parquet_count due to transaction logs.

D delta_count will be zero if the Delta table has no transaction log.

💡 Hint

Consider that both read operations load the data content, but Delta Lake manages metadata differently.

If the data content is the same, both DataFrames will have the same number of rows. Delta Lake uses transaction logs internally but presents the current snapshot of data when read.

🔧 Debug

advanced

Error When Writing to Parquet with Schema Evolution

What error will occur when trying to append data with a new column to an existing Parquet dataset without schema merging?

Apache Spark

df_new = spark.createDataFrame([(1, 'a', 10)], ['id', 'name', 'age'])
df_new.write.mode('append').parquet('/path/to/parquet_dataset')

A AnalysisException: Cannot append to table with different schema without schema merging enabled.

B TypeError due to incompatible data types in the new column.

C FileNotFoundError because the path does not exist.

D No error; Parquet automatically merges schemas on append.

💡 Hint

Think about how Parquet handles schema changes when appending data.

Parquet does not support automatic schema merging on append by default. Spark throws an AnalysisException unless schema merging is explicitly enabled.

🚀 Application

advanced

Choosing Storage Format for Streaming Data

You have a streaming data pipeline that requires exactly-once processing and frequent updates. Which storage format should you choose?

A CSV files, because they are simple and support streaming updates well.

B Parquet, because it is faster for streaming data and supports updates natively.

C Delta Lake, because it supports ACID transactions and efficient streaming writes.

D JSON files, because they handle nested data better than Delta Lake.

💡 Hint

Consider which format supports exactly-once semantics and updates in streaming.

Delta Lake supports ACID transactions and is designed for streaming workloads with exactly-once guarantees. Parquet files are immutable and do not support updates natively.

❓ visualization

expert

Visualizing Data Version History in Delta Lake

Which Spark command will show the version history of a Delta Lake table to help visualize changes over time?

Apache Spark

spark.sql("DESCRIBE HISTORY delta.`/path/to/delta_table`").show()

A spark.sql("SHOW VERSIONS delta.`/path/to/delta_table`").show()

B spark.read.format('delta').option('versionAsOf', 0).load('/path/to/delta_table').show()

C spark.read.format('parquet').load('/path/to/delta_table').show()

D spark.sql("DESCRIBE HISTORY delta.`/path/to/delta_table`").show()

💡 Hint

Think about which command lists the commit history of a Delta table.

The DESCRIBE HISTORY command shows the commit history and metadata changes of a Delta Lake table. Other options either do not exist or do not show version history.

Practice

1. Which statement best describes the main difference between Delta Lake and Parquet?

easy

A. Delta Lake supports data updates and history tracking, while Parquet is read-only.

B. Parquet supports data updates, but Delta Lake does not.

C. Both Delta Lake and Parquet are read-only file formats.

D. Delta Lake is slower than Parquet for reading data.

Solution

1. Step 1: Understand Parquet's capabilities

   Parquet is a columnar file format optimized for fast read operations but does not support data updates or versioning.

2. Step 2: Understand Delta Lake's enhancements

   Delta Lake builds on Parquet by adding support for data updates, deletes, and tracking changes over time (history).

3. Final Answer:

   Delta Lake supports data updates and history tracking, while Parquet is read-only. -> Option A

4. Quick Check:

   Delta Lake adds update features to Parquet [OK]

Hint: Delta Lake = updates/history; Parquet = read-only [OK]

Common Mistakes:

• Thinking Parquet supports data updates
• Assuming Delta Lake is slower for reads
• Confusing both as identical formats

2. Which of the following is the correct way to write a DataFrame as a Delta Lake table in Apache Spark?

easy

A. df.write.format("parquet").saveAsTable("delta_table")

B. df.write.format("delta").save("path/to/table")

C. df.saveAsTable("delta", "path/to/table")

D. df.write.save("delta", "path/to/table")

Solution

1. Step 1: Recall the syntax for writing Delta tables

   To write a DataFrame as Delta, use df.write.format("delta").save(path).

2. Step 2: Check each option's correctness

   df.write.format("delta").save("path/to/table") matches the correct syntax. df.write.format("parquet").saveAsTable("delta_table") uses Parquet format incorrectly. The remaining options use invalid method signatures.

3. Final Answer:

   df.write.format("delta").save("path/to/table") -> Option B

4. Quick Check:

   Use .write.format("delta") to save Delta tables [OK]

Hint: Use .write.format("delta") to save Delta tables [OK]

Common Mistakes:

• Using saveAsTable with format mismatch
• Confusing Parquet and Delta formats
• Incorrect method chaining

3. Given the following code snippet, what will be the output count after the update?

from delta.tables import DeltaTable

# Load Delta table
deltaTable = DeltaTable.forPath(spark, "/data/delta/events")

# Update rows where eventType is 'click'
deltaTable.update(condition = "eventType = 'click'", set = {"eventType": "click_updated"})

# Count rows with updated eventType
count = spark.read.format("delta").load("/data/delta/events").filter("eventType = 'click_updated'").count()
print(count)

medium

A. Zero, because update does not change data

B. Total number of rows in the table

C. Number of rows where eventType was 'click' before update

D. Number of rows where eventType is 'click_updated' before update

Solution

1. Step 1: Understand the update operation on Delta table

   The update changes all rows with eventType 'click' to 'click_updated'.

2. Step 2: Count rows with eventType 'click_updated'

   After update, the count of 'click_updated' rows equals the original count of 'click' rows.

3. Final Answer:

   Number of rows where eventType was 'click' before update -> Option C

4. Quick Check:

   Update changes 'click' to 'click_updated' rows count [OK]

Hint: Updated rows count = original matching rows count [OK]

Common Mistakes:

• Assuming update does not change data
• Counting total rows instead of filtered
• Confusing pre- and post-update values

4. Identify the error in the following code snippet that tries to update a Parquet file:

parquetPath = "/data/parquet/events"
df = spark.read.format("parquet").load(parquetPath)
df.update(condition = "eventType = 'click'", set = {"eventType": "click_updated"})

medium

A. Parquet format does not support update operation

B. The condition syntax is incorrect

C. The set parameter should be a list, not a dictionary

D. The path variable is not defined

Solution

1. Step 1: Check if Parquet supports update

   Parquet is a static file format and does not support update operations directly.

2. Step 2: Identify the cause of error

   Calling update on a DataFrame loaded from Parquet will cause an error because this method is only available on Delta tables.

3. Final Answer:

   Parquet format does not support update operation -> Option A

4. Quick Check:

   Update works only on Delta, not Parquet [OK]

Hint: Update only works on Delta tables, not Parquet [OK]

Common Mistakes:

• Trying to update Parquet files directly
• Misusing update method on DataFrame
• Ignoring format limitations

5. You have a dataset stored in Parquet format that needs frequent updates and audit history. Which approach is best to handle this requirement efficiently?

hard

A. Store data in JSON files and manually track changes

B. Keep using Parquet and overwrite the entire dataset on each update

C. Use CSV format instead for easier updates

D. Convert the Parquet dataset to Delta Lake format and use Delta's update and versioning features

Solution

1. Step 1: Analyze update and audit needs

   Frequent updates and audit history require a format that supports transactions and versioning.

2. Step 2: Evaluate format options

   Delta Lake supports updates and keeps history, unlike Parquet, CSV, or JSON which lack these features natively.

3. Step 3: Choose the best approach

   Converting Parquet to Delta Lake enables efficient updates and audit tracking without full overwrites.

4. Final Answer:

   Convert the Parquet dataset to Delta Lake format and use Delta's update and versioning features -> Option D

5. Quick Check:

   Delta Lake = updates + history for Parquet data [OK]

Hint: Use Delta Lake for updates and history, not Parquet [OK]

Common Mistakes:

• Overwriting entire Parquet files repeatedly
• Choosing CSV or JSON for update-heavy data
• Ignoring Delta Lake benefits