ACID transactions on data lakes in Apache Spark - Interactive Code Practice

Practice - 5 Tasks

1fill in blank

easy

Complete the code to read a Delta table from a data lake.

Apache Spark

df = spark.read.format([1]).load("/data/delta/events")

Drag options to blanks, or click blank then click option'

Aparquet

Bcsv

Cdelta

Djson

Attempts:

3 left

💡 Hint

Common Mistakes

Using 'parquet' or 'csv' format which do not support ACID transactions directly.

The Delta format is used to enable ACID transactions on data lakes. Using format('delta') tells Spark to read the Delta table.

2fill in blank

medium

Complete the code to write a DataFrame as a Delta table with overwrite mode.

Apache Spark

df.write.format("delta").mode([1]).save("/data/delta/events")

Drag options to blanks, or click blank then click option'

Aappend

Boverwrite

Cignore

Derror

Attempts:

3 left

💡 Hint

Common Mistakes

Using 'append' mode when intending to replace data.

Using mode('overwrite') replaces the existing data in the Delta table, which is useful for updating data fully.

3fill in blank

hard

Fix the error in the code to update a Delta table using Spark SQL.

Apache Spark

spark.sql("UPDATE delta.`/data/delta/events` SET status = 'processed' WHERE [1] = 100")

Drag options to blanks, or click blank then click option'

Aid

Bdate

Cstatus

Dcount

Attempts:

3 left

💡 Hint

Common Mistakes

Using a non-unique column in the WHERE clause causing unintended updates.

The WHERE clause should filter by a column that identifies rows, such as id. Filtering by status or other columns may not target the intended rows.

4fill in blank

hard

Fill both blanks to create a Delta table with partitioning and enable schema evolution.

Apache Spark

df.write.format("delta").partitionBy([1]).option("mergeSchema", [2]).mode("append").save("/data/delta/events")

Drag options to blanks, or click blank then click option'

A"date"

Btrue

Cfalse

D"status"

Attempts:

3 left

💡 Hint

Common Mistakes

Using incorrect partition column names or disabling schema merge.

Partitioning by "date" helps organize data by date. Setting mergeSchema to true allows schema changes to be merged automatically.

5fill in blank

hard

Fill all three blanks to create a Delta table, update it, and read the latest version.

Apache Spark

df.write.format([1]).mode("overwrite").save("/data/delta/events")
spark.sql("UPDATE [2] SET status = 'done' WHERE id = 5")
df_latest = spark.read.format([3]).load("/data/delta/events")

Drag options to blanks, or click blank then click option'

Adelta

Bevents

Dparquet

Attempts:

3 left

💡 Hint

Common Mistakes

Mixing formats like parquet and delta causing errors.

Using wrong table names in SQL update.

The Delta format is used for writing and reading to support ACID transactions. The table name events is used in the SQL update. Reading with format delta ensures the latest version is read.

Practice

1. What does ACID stand for in the context of data lakes?

easy

A. Aggregation, Calculation, Insertion, Deletion

B. Accuracy, Clarity, Integrity, Data

C. Access, Control, Indexing, Distribution

D. Atomicity, Consistency, Isolation, Durability

Solution

1. Step 1: Understand ACID properties

   ACID is a set of properties that guarantee reliable processing of database transactions.

2. Step 2: Match each letter to its meaning

   Atomicity means all or nothing, Consistency means data stays correct, Isolation means transactions don't interfere, Durability means changes are permanent.

3. Final Answer:

   Atomicity, Consistency, Isolation, Durability -> Option D

4. Quick Check:

   ACID = Atomicity, Consistency, Isolation, Durability [OK]

Hint: Remember ACID as key database safety rules [OK]

Common Mistakes:

• Confusing ACID with unrelated terms
• Mixing up the letters' meanings
• Thinking ACID is only for SQL databases

2. Which syntax correctly creates a Delta table with ACID support in Apache Spark?

easy

A. spark.sql("CREATE TABLE delta_table TYPE delta LOCATION '/delta/path'")

B. spark.sql("CREATE DELTA TABLE delta_table AS '/delta/path'")

C. spark.sql("CREATE TABLE delta_table USING delta LOCATION '/delta/path'")

D. spark.sql("MAKE TABLE delta_table USING delta LOCATION '/delta/path'")

Solution

1. Step 1: Recall Delta Lake table creation syntax

   The correct syntax uses CREATE TABLE with USING delta and LOCATION to specify path.

2. Step 2: Identify correct syntax among options

   spark.sql("CREATE TABLE delta_table USING delta LOCATION '/delta/path'") matches the correct syntax exactly; others have wrong keywords or order.

3. Final Answer:

   spark.sql("CREATE TABLE delta_table USING delta LOCATION '/delta/path'") -> Option C

4. Quick Check:

   CREATE TABLE ... USING delta ... = spark.sql("CREATE TABLE delta_table USING delta LOCATION '/delta/path'") [OK]

Hint: Look for 'CREATE TABLE ... USING delta' syntax [OK]

Common Mistakes:

• Using 'MAKE TABLE' instead of 'CREATE TABLE'
• Wrong keyword order or missing USING
• Confusing AS with LOCATION

3. Given the following code snippet, what will be the output count after the update?

from delta.tables import DeltaTable

# Assume delta_table is a DeltaTable object
# Initial data has 3 rows with column 'value' = [10, 20, 30]
delta_table.update("value = 20", {"value": "value + 5"})
count = delta_table.toDF().count()
print(count)

medium

A. 4

B. 3

C. 2

D. 0

Solution

1. Step 1: Understand the update operation

   The update changes rows where value = 20 by adding 5, so that row's value becomes 25.

2. Step 2: Check if row count changes

   Update modifies data in place; it does not add or remove rows, so count remains 3.

3. Final Answer:

   3 -> Option B

4. Quick Check:

   Update changes values, not row count = 3 [OK]

Hint: Update changes data but not number of rows [OK]

Common Mistakes:

• Thinking update adds new rows
• Confusing update with insert
• Assuming count changes after update

4. Identify the error in this Delta Lake transaction code:

delta_table.update("value > 10", {"value": "value + 1"})
delta_table.delete("value < 5")
delta_table.commit()

medium

A. DeltaTable has no commit() method; transactions auto-commit

B. Update condition syntax is incorrect

C. Delete condition should be a function, not a string

D. Missing startTransaction() before update

Solution

1. Step 1: Check DeltaTable API methods

   DeltaTable supports update and delete, but does not have a commit() method; changes auto-commit.

2. Step 2: Verify other method usages

   Update and delete accept string conditions correctly; no need for explicit transaction start.

3. Final Answer:

   DeltaTable has no commit() method; transactions auto-commit -> Option A

4. Quick Check:

   No commit() in DeltaTable API = DeltaTable has no commit() method; transactions auto-commit [OK]

Hint: Delta Lake auto-commits; no commit() needed [OK]

Common Mistakes:

• Trying to call commit() on DeltaTable
• Misunderstanding condition parameter types
• Expecting manual transaction control

5. You want to ensure multiple users can safely update a Delta Lake table without conflicts. Which feature helps achieve this?

hard

A. Delta Lake's optimistic concurrency control with ACID transactions

B. Using Spark's cache() to lock the table

C. Manually locking files in the storage system

D. Running updates sequentially with sleep delays

Solution

1. Step 1: Understand concurrency in Delta Lake

   Delta Lake uses optimistic concurrency control to handle multiple users updating data safely.

2. Step 2: Evaluate other options

   Cache() only speeds reads, manual file locks are error-prone, and sleep delays are unreliable.

3. Final Answer:

   Delta Lake's optimistic concurrency control with ACID transactions -> Option A

4. Quick Check:

   Optimistic concurrency control = safe multi-user updates [OK]

Hint: Trust Delta Lake's built-in concurrency control [OK]

Common Mistakes:

• Thinking cache() locks data
• Trying manual file locks
• Using delays instead of transaction control