Delta Lake introduction in Apache Spark - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

Output of Delta Lake table creation and query

What is the output of the following Apache Spark code using Delta Lake?

Apache Spark

from pyspark.sql import SparkSession
spark = SparkSession.builder.appName("DeltaExample").getOrCreate()
data = [(1, "apple"), (2, "banana"), (3, "cherry")]
df = spark.createDataFrame(data, ["id", "fruit"])
df.write.format("delta").mode("overwrite").save("/tmp/delta-table")
df2 = spark.read.format("delta").load("/tmp/delta-table")
df2.filter(df2.id > 1).count()

A 2

B 0

C 1

D 3

💡 Hint

Count rows where id is greater than 1.

The original data has 3 rows with ids 1, 2, and 3. Filtering for id > 1 keeps rows with ids 2 and 3, so count is 2.

❓ data_output

intermediate

Result of Delta Lake update operation

Given a Delta Lake table with data [(1, 10), (2, 20), (3, 30)] stored at '/tmp/delta-update', what is the content of the table after running this update code?

Apache Spark

from delta.tables import DeltaTable
from pyspark.sql import SparkSession
spark = SparkSession.builder.appName("DeltaUpdate").getOrCreate()
data = [(1, 10), (2, 20), (3, 30)]
df = spark.createDataFrame(data, ["id", "value"])
df.write.format("delta").mode("overwrite").save("/tmp/delta-update")
deltaTable = DeltaTable.forPath(spark, "/tmp/delta-update")
deltaTable.update(condition = "id == 2", set = {"value": "value + 5"})
deltaTable.toDF().orderBy("id").collect()

A [(1, 10), (2, 20), (3, 35)]

B [(1, 10), (2, 25), (3, 30)]

C [(1, 15), (2, 20), (3, 30)]

D [(1, 10), (2, 5), (3, 30)]

💡 Hint

Only the row with id 2 is updated by adding 5 to its value.

The update changes the value for id 2 from 20 to 25. Other rows remain unchanged.

🔧 Debug

advanced

Identify the error in Delta Lake merge code

What error will this Delta Lake merge code produce?

Apache Spark

from delta.tables import DeltaTable
from pyspark.sql import SparkSession
spark = SparkSession.builder.appName("DeltaMerge").getOrCreate()
data_target = [(1, "a"), (2, "b")]
data_source = [(2, "bb"), (3, "c")]
df_target = spark.createDataFrame(data_target, ["id", "value"])
df_source = spark.createDataFrame(data_source, ["id", "value"])
df_target.write.format("delta").mode("overwrite").save("/tmp/delta-merge")
deltaTable = DeltaTable.forPath(spark, "/tmp/delta-merge")
deltaTable.alias("t").merge(
    df_source.alias("s"),
    "t.id = s.id"
).whenMatchedUpdate(set = {"value": "s.value"})
.whenNotMatchedInsert(values = {"id": "s.id", "value": "s.value"})
.execute()

A SyntaxError due to missing colon in merge condition

B No error, merge executes successfully

C AnalysisException due to missing Delta Lake package

D TypeError because set and values expect column expressions, not strings

💡 Hint

Check the types used in set and values arguments in merge.

The set and values parameters require column expressions, not string literals. Passing strings causes a TypeError.

❓ visualization

advanced

Visualize Delta Lake version history

Which option correctly describes how to visualize the version history of a Delta Lake table?

A Use 'DESCRIBE HISTORY delta.`/path/to/table`' SQL command to get version info and plot with matplotlib

B Run 'SHOW VERSIONS delta.`/path/to/table`' to get history and use seaborn to plot

C Use 'SELECT * FROM delta.`/path/to/table` VERSION AS OF 0' to get history and plot with pandas

D Run 'HISTORY delta.`/path/to/table`' command and plot with plotly

💡 Hint

Delta Lake supports DESCRIBE HISTORY command for version info.

DESCRIBE HISTORY returns the version history of a Delta table. This data can be collected and visualized using Python plotting libraries.

🧠 Conceptual

expert

Understanding Delta Lake ACID guarantees

Which statement best explains how Delta Lake ensures ACID transactions on big data?

A Delta Lake uses a single writer lock to prevent concurrent writes, ensuring durability.

B Delta Lake relies on HDFS replication to guarantee atomic writes and consistency.

C Delta Lake uses a transaction log that records all changes atomically and supports snapshot isolation for readers.

D Delta Lake stores data in a relational database to enforce ACID properties.

💡 Hint

Think about how Delta Lake tracks changes and manages concurrent access.

Delta Lake maintains a transaction log that records every change atomically. This log enables snapshot isolation, so readers see a consistent view even during concurrent writes.

Practice

1. What is the main purpose of Delta Lake in Apache Spark?

easy

A. To provide reliable data storage with ACID transactions

B. To speed up Spark job execution by caching data

C. To visualize data using charts and graphs

D. To replace Spark SQL with a new query language

Solution

1. Step 1: Understand Delta Lake's role

   Delta Lake adds reliability by supporting ACID transactions on big data.

2. Step 2: Compare options

   Speeding up jobs, visualization, or replacing Spark SQL are not Delta Lake's main goals.

3. Final Answer:

   To provide reliable data storage with ACID transactions -> Option A

4. Quick Check:

   Delta Lake = ACID transactions [OK]

Hint: Delta Lake ensures data correctness with ACID [OK]

Common Mistakes:

• Confusing Delta Lake with caching or visualization tools
• Thinking Delta Lake replaces Spark SQL
• Assuming Delta Lake only speeds up queries

2. Which of the following is the correct way to create a Delta table in Spark using PySpark?

easy

A. df.write.format('delta').save('/path/to/delta-table')

B. df.saveAsTable('delta', '/path/to/delta-table')

C. df.write.save('/path/to/delta-table', format='delta')

D. df.createDeltaTable('/path/to/delta-table')

Solution

1. Step 1: Recall Delta table creation syntax

   In PySpark, to save a DataFrame as a Delta table, use df.write.format('delta').save(path).

2. Step 2: Check each option

   df.write.format('delta').save('/path/to/delta-table') matches the correct syntax; others are invalid or incorrect method calls.

3. Final Answer:

   df.write.format('delta').save('/path/to/delta-table') -> Option A

4. Quick Check:

   Use write.format('delta').save() [OK]

Hint: Use write.format('delta').save() to create Delta tables [OK]

Common Mistakes:

• Using saveAsTable with wrong parameters
• Passing format as a keyword argument to save()
• Assuming createDeltaTable is a valid method

3. Given the following PySpark code, what will be the output count after the delete operation?

from delta.tables import DeltaTable

spark.sql("""CREATE TABLE IF NOT EXISTS events (id INT, event STRING) USING DELTA""")
spark.sql("INSERT INTO events VALUES (1, 'start'), (2, 'stop'), (3, 'pause')")
deltaTable = DeltaTable.forName(spark, 'events')
deltaTable.delete("id = 2")
count = spark.sql("SELECT COUNT(*) FROM events").collect()[0][0]
print(count)

medium

A. 3

B. 2

C. 1

D. 0

Solution

1. Step 1: Understand initial data

   The table 'events' has 3 rows with ids 1, 2, and 3.

2. Step 2: Analyze delete operation

   Deleting rows where id = 2 removes one row, leaving 2 rows.

3. Final Answer:

   2 -> Option B

4. Quick Check:

   3 rows - 1 deleted = 2 rows [OK]

Hint: Delete removes matching rows; count reflects remaining [OK]

Common Mistakes:

• Assuming delete removes all rows
• Forgetting to filter by condition in delete
• Confusing insert and delete effects

4. Identify the error in the following PySpark code snippet that tries to update a Delta table:

from delta.tables import DeltaTable

deltaTable = DeltaTable.forPath(spark, '/delta/events')
deltaTable.update(condition="id == 1", set={"event": 'resume'})

medium

A. The update method requires a SQL expression string, not a dictionary

B. The condition should be passed as a SQL expression, not a string

C. The set parameter must use column objects, not strings

D. The update method requires the condition parameter to be a string expression, and set to a dictionary with column names as keys and expressions as values

Solution

1. Step 1: Review update method signature

   DeltaTable.update expects condition as a string expression and set as a dictionary mapping column names to SQL expressions.

2. Step 2: Check given code

   Condition is correct string expression. set uses dict with column name key, but value 'resume' must be SQL expression like "'resume'".

3. Step 3: Identify missing quotes in code snippet

   The value in set lacks quotes for string literal, violating the requirement for expressions as values described in The update method requires the condition parameter to be a string expression, and set to a dictionary with column names as keys and expressions as values.

4. Final Answer:

   The update method requires the condition parameter to be a string expression, and set to a dictionary with column names as keys and expressions as values -> Option D

5. Quick Check:

   Update needs condition string and set dict [OK]

Hint: Update needs condition string and set dict with column keys [OK]

Common Mistakes:

• Passing condition as non-string
• Using incorrect types in set dictionary
• Not quoting string literals in set expressions

5. You have a Delta table with user transactions. You want to keep only the latest transaction per user and remove older duplicates. Which Delta Lake feature helps you do this efficiently?

hard

A. Delta Lake's VACUUM to clean old files

B. Delta Lake's Time Travel to query past versions

C. Delta Lake's MERGE INTO to upsert and deduplicate data

D. Delta Lake's OPTIMIZE to compact small files

Solution

1. Step 1: Understand the goal

   We want to keep only the latest transaction per user, removing duplicates.

2. Step 2: Identify Delta Lake features

   MERGE INTO allows conditional updates and inserts, perfect for upserting and deduplication.

3. Step 3: Exclude other options

   Time Travel queries past data, VACUUM cleans files, OPTIMIZE compacts files, none directly remove duplicates.

4. Final Answer:

   Delta Lake's MERGE INTO to upsert and deduplicate data -> Option C

5. Quick Check:

   Use MERGE INTO for upsert and deduplication [OK]

Hint: Use MERGE INTO to update and remove duplicates efficiently [OK]

Common Mistakes:

• Confusing Time Travel with deduplication
• Thinking VACUUM removes duplicates
• Assuming OPTIMIZE deletes old data