Delta Lake introduction in Apache Spark - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

Output of Delta Lake table creation and query

What is the output of the following Apache Spark code using Delta Lake?

Apache Spark

from pyspark.sql import SparkSession
spark = SparkSession.builder.appName("DeltaExample").getOrCreate()
data = [(1, "apple"), (2, "banana"), (3, "cherry")]
df = spark.createDataFrame(data, ["id", "fruit"])
df.write.format("delta").mode("overwrite").save("/tmp/delta-table")
df2 = spark.read.format("delta").load("/tmp/delta-table")
df2.filter(df2.id > 1).count()

A 2

B 0

C 1

D 3

💡 Hint

Count rows where id is greater than 1.

The original data has 3 rows with ids 1, 2, and 3. Filtering for id > 1 keeps rows with ids 2 and 3, so count is 2.

❓ data_output

intermediate

Result of Delta Lake update operation

Given a Delta Lake table with data [(1, 10), (2, 20), (3, 30)] stored at '/tmp/delta-update', what is the content of the table after running this update code?

Apache Spark

from delta.tables import DeltaTable
from pyspark.sql import SparkSession
spark = SparkSession.builder.appName("DeltaUpdate").getOrCreate()
data = [(1, 10), (2, 20), (3, 30)]
df = spark.createDataFrame(data, ["id", "value"])
df.write.format("delta").mode("overwrite").save("/tmp/delta-update")
deltaTable = DeltaTable.forPath(spark, "/tmp/delta-update")
deltaTable.update(condition = "id == 2", set = {"value": "value + 5"})
deltaTable.toDF().orderBy("id").collect()

A [(1, 10), (2, 20), (3, 35)]

B [(1, 10), (2, 25), (3, 30)]

C [(1, 15), (2, 20), (3, 30)]

D [(1, 10), (2, 5), (3, 30)]

💡 Hint

Only the row with id 2 is updated by adding 5 to its value.

The update changes the value for id 2 from 20 to 25. Other rows remain unchanged.

🔧 Debug

advanced

Identify the error in Delta Lake merge code

What error will this Delta Lake merge code produce?

Apache Spark

from delta.tables import DeltaTable
from pyspark.sql import SparkSession
spark = SparkSession.builder.appName("DeltaMerge").getOrCreate()
data_target = [(1, "a"), (2, "b")]
data_source = [(2, "bb"), (3, "c")]
df_target = spark.createDataFrame(data_target, ["id", "value"])
df_source = spark.createDataFrame(data_source, ["id", "value"])
df_target.write.format("delta").mode("overwrite").save("/tmp/delta-merge")
deltaTable = DeltaTable.forPath(spark, "/tmp/delta-merge")
deltaTable.alias("t").merge(
    df_source.alias("s"),
    "t.id = s.id"
).whenMatchedUpdate(set = {"value": "s.value"})
.whenNotMatchedInsert(values = {"id": "s.id", "value": "s.value"})
.execute()

A SyntaxError due to missing colon in merge condition

B No error, merge executes successfully

C AnalysisException due to missing Delta Lake package

D TypeError because set and values expect column expressions, not strings

💡 Hint

Check the types used in set and values arguments in merge.

The set and values parameters require column expressions, not string literals. Passing strings causes a TypeError.

❓ visualization

advanced

Visualize Delta Lake version history

Which option correctly describes how to visualize the version history of a Delta Lake table?

A Use 'DESCRIBE HISTORY delta.`/path/to/table`' SQL command to get version info and plot with matplotlib

B Run 'SHOW VERSIONS delta.`/path/to/table`' to get history and use seaborn to plot

C Use 'SELECT * FROM delta.`/path/to/table` VERSION AS OF 0' to get history and plot with pandas

D Run 'HISTORY delta.`/path/to/table`' command and plot with plotly

💡 Hint

Delta Lake supports DESCRIBE HISTORY command for version info.

DESCRIBE HISTORY returns the version history of a Delta table. This data can be collected and visualized using Python plotting libraries.

🧠 Conceptual

expert

Understanding Delta Lake ACID guarantees

Which statement best explains how Delta Lake ensures ACID transactions on big data?

A Delta Lake uses a single writer lock to prevent concurrent writes, ensuring durability.

B Delta Lake relies on HDFS replication to guarantee atomic writes and consistency.

C Delta Lake uses a transaction log that records all changes atomically and supports snapshot isolation for readers.

D Delta Lake stores data in a relational database to enforce ACID properties.

💡 Hint

Think about how Delta Lake tracks changes and manages concurrent access.

Delta Lake maintains a transaction log that records every change atomically. This log enables snapshot isolation, so readers see a consistent view even during concurrent writes.