Time travel and versioning in Apache Spark - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

Output of reading a Delta table at a previous version

Given a Delta table with 3 versions, what will be the output count of rows when reading version 1?

Apache Spark

from delta.tables import DeltaTable
from pyspark.sql import SparkSession

spark = SparkSession.builder.appName('TimeTravelTest').getOrCreate()
delta_path = '/tmp/delta-table'

# Read version 1 of the table
version_1_df = spark.read.format('delta').option('versionAsOf', 1).load(delta_path)

print(version_1_df.count())

A The number of rows present in the table at the latest version

B The number of rows present in the table at version 1

C Always zero because version 1 is outdated

D Raises an error because version 1 is not accessible

💡 Hint

Think about what versionAsOf option does when reading a Delta table.

Using option('versionAsOf', 1) reads the table exactly as it was at version 1, so the row count matches that version's data.

❓ data_output

intermediate

Result of a Delta table history query

What does the following code output?

Apache Spark

from delta.tables import DeltaTable
from pyspark.sql import SparkSession

spark = SparkSession.builder.appName('HistoryTest').getOrCreate()
delta_path = '/tmp/delta-table'
delta_table = DeltaTable.forPath(spark, delta_path)
history_df = delta_table.history(2)
history_df.show()

A An empty DataFrame because history(2) is invalid

B A DataFrame with all rows of the Delta table

C An error because history() does not accept parameters

D A DataFrame showing the last 2 commits with metadata like version, timestamp, and operation

💡 Hint

The history() method shows commit history of the Delta table.

Calling history(2) returns a DataFrame with details of the last 2 commits, including version numbers and operation types.

🔧 Debug

advanced

Identify the error in time travel query

What error will this code raise?

Apache Spark

spark.read.format('delta').option('versionAsOf', 'two').load('/tmp/delta-table')

A TypeError because option expects an integer, not a string

B FileNotFoundError because the path does not exist

C AnalysisException because 'two' is not a valid version number

D No error, it reads the latest version

💡 Hint

Check the type and value expected for versionAsOf option.

The versionAsOf option expects an integer version number. Passing a string like 'two' causes an AnalysisException.

🚀 Application

advanced

Using timestamp for time travel in Delta Lake

Which code snippet correctly reads a Delta table as it was at '2024-01-01 12:00:00'?

A spark.read.format('delta').option('timestampAsOf', '2024-01-01 12:00:00').load('/tmp/delta-table')

B spark.read.format('delta').option('asOfTimestamp', '2024-01-01 12:00:00').load('/tmp/delta-table')

C spark.read.format('delta').option('timeTravel', '2024-01-01 12:00:00').load('/tmp/delta-table')

D spark.read.format('delta').option('versionAsOf', '2024-01-01 12:00:00').load('/tmp/delta-table')

💡 Hint

Look for the correct option name for timestamp-based time travel.

The correct option to specify a timestamp for time travel is 'timestampAsOf'. Other options are invalid.

🧠 Conceptual

expert

Understanding Delta Lake version retention and vacuum

What happens if you run vacuum on a Delta table and then try to time travel to a version older than the retention period?

A Time travel to that old version will fail because the data files are deleted

B Time travel will succeed because Delta Lake keeps all versions indefinitely

C Vacuum only deletes metadata, so time travel is unaffected

D Time travel will return empty DataFrame but no error

💡 Hint

Consider what vacuum does to old data files and how time travel depends on them.

Vacuum deletes data files older than the retention period. Time travel to versions relying on those files will fail.

Practice

1. What is the main purpose of time travel in Apache Spark's Delta Lake?

easy

A. To access previous versions of data easily

B. To speed up data processing

C. To create new tables automatically

D. To delete old data permanently

Solution

1. Step 1: Understand time travel concept

   Time travel allows users to look at past versions of data stored in Delta Lake.

2. Step 2: Identify main use

   This feature helps in recovering old data, auditing changes, and comparing versions.

3. Final Answer:

   To access previous versions of data easily -> Option A

4. Quick Check:

   Time travel = Access past data versions [OK]

Hint: Time travel means looking back at old data versions [OK]

Common Mistakes:

• Thinking time travel speeds up processing
• Confusing time travel with data deletion
• Assuming it creates new tables automatically

2. Which option is the correct way to load a Delta table at a specific version using Spark?

easy

A. spark.read.format('delta').option('version', 5).load('/data/table')

B. spark.read.format('delta').option('versionAsOf', 5).load('/data/table')

C. spark.read.format('delta').option('timestampAsOf', '5').load('/data/table')

D. spark.read.format('delta').option('versionAt', 5).load('/data/table')

Solution

1. Step 1: Recall correct option name for version

   The correct option to specify version is 'versionAsOf'.

2. Step 2: Check syntax correctness

   spark.read.format('delta').option('versionAsOf', 5).load('/data/table') uses 'versionAsOf' with a number, which is correct syntax.

3. Final Answer:

   spark.read.format('delta').option('versionAsOf', 5).load('/data/table') -> Option B

4. Quick Check:

   Use 'versionAsOf' to load specific version [OK]

Hint: Use 'versionAsOf' option to load specific version [OK]

Common Mistakes:

• Using 'version' instead of 'versionAsOf'
• Passing version as string in 'timestampAsOf'
• Using wrong option names like 'versionAt'

3. Given the code below, what will be the output count?

df = spark.read.format('delta').option('versionAsOf', 2).load('/data/events')
print(df.count())

Assuming version 2 of the table has 150 rows.

medium

A. Error due to wrong option

B. 0

C. Latest row count

D. 150

Solution

1. Step 1: Understand versionAsOf usage

   The option 'versionAsOf' loads the table exactly at version 2.

2. Step 2: Check row count at version 2

   Since version 2 has 150 rows, the count will be 150.

3. Final Answer:

   150 -> Option D

4. Quick Check:

   Count at version 2 = 150 [OK]

Hint: Count matches rows in specified version [OK]

Common Mistakes:

• Assuming count is zero for old versions
• Confusing latest count with versioned count
• Expecting error from correct syntax

4. Identify the error in the following code snippet for loading a Delta table at a timestamp:

df = spark.read.format('delta').option('timestampAsOf', 20230101).load('/data/sales')

medium

A. Load path is incorrect

B. timestampAsOf option does not exist

C. timestampAsOf value should be a string with date-time format

D. Missing .option('versionAsOf', )

Solution

1. Step 1: Check timestampAsOf option usage

   The 'timestampAsOf' option expects a string representing date and time, e.g., '2023-01-01T00:00:00'.

2. Step 2: Identify error in value type

   Here, 20230101 is an integer, not a string with proper format, causing error.

3. Final Answer:

   timestampAsOf value should be a string with date-time format -> Option C

4. Quick Check:

   timestampAsOf needs string date-time [OK]

Hint: Use string date-time format for timestampAsOf [OK]

Common Mistakes:

• Passing integer instead of string for timestamp
• Confusing versionAsOf with timestampAsOf
• Assuming option does not exist

5. You want to compare the number of rows between version 3 and version 5 of a Delta table located at '/data/logs'. Which code snippet correctly loads both versions for comparison?

hard

A. df_v3 = spark.read.format('delta').option('versionAsOf', 3).load('/data/logs')
df_v5 = spark.read.format('delta').option('versionAsOf', 5).load('/data/logs')

B. df_v3 = spark.read.format('delta').option('timestampAsOf', 3).load('/data/logs')
df_v5 = spark.read.format('delta').option('timestampAsOf', 5).load('/data/logs')

C. df_v3 = spark.read.format('delta').option('version', 3).load('/data/logs')
df_v5 = spark.read.format('delta').option('version', 5).load('/data/logs')

D. df_v3 = spark.read.format('delta').option('versionAsOf', '3').load('/data/logs')
df_v5 = spark.read.format('delta').option('versionAsOf', '5').load('/data/logs')

Solution

1. Step 1: Identify correct option for version number

   The option 'versionAsOf' expects an integer version number, not a string.

2. Step 2: Check code correctness for both versions

   df_v3 = spark.read.format('delta').option('versionAsOf', 3).load('/data/logs') df_v5 = spark.read.format('delta').option('versionAsOf', 5).load('/data/logs') uses 'versionAsOf' with integers 3 and 5, correctly loading both versions.

3. Final Answer:

   df_v3 = spark.read.format('delta').option('versionAsOf', 3).load('/data/logs') df_v5 = spark.read.format('delta').option('versionAsOf', 5).load('/data/logs') -> Option A

4. Quick Check:

   Use 'versionAsOf' with integers for version loading [OK]

Hint: Use versionAsOf with integers to load specific versions [OK]

Common Mistakes:

• Using timestampAsOf with integers
• Passing version numbers as strings
• Using non-existent 'version' option