Time travel and versioning in Apache Spark - Step-by-Step Execution

Concept Flow - Time travel and versioning

Start with Delta Table

↓

Write Data Version 1

↓

Update or Append Data

↓

Create Data Version 2

↓

Query Specific Version

↓

Get Data as it was at Version X

↓

Use for Auditing or Rollback

This flow shows how data is saved in versions and how you can query past versions using time travel.

Execution Sample

Apache Spark

from delta.tables import DeltaTable

# Write initial data
spark.range(5).write.format('delta').save('/tmp/delta-table')

# Update data
DeltaTable.forPath(spark, '/tmp/delta-table').update("id == 3", {"id": 30})

# Read version 0
spark.read.format('delta').option('versionAsOf', 0).load('/tmp/delta-table').show()

This code writes data, updates it, then reads the original version using time travel.

Execution Table

Step	Action	Data Version	Data Content	Result/Output
1	Write initial data with ids 0 to 4	0	[0,1,2,3,4]	Data saved as version 0
2	Update id 3 to 30	1	[0,1,2,30,4]	Data saved as version 1
3	Read data with versionAsOf=0	0	[0,1,2,3,4]	Shows original data before update
4	Read data with versionAsOf=1	1	[0,1,2,30,4]	Shows updated data
5	Read latest data without version option	1	[0,1,2,30,4]	Shows latest data
6	Exit	-	-	No more steps

💡 All steps executed, demonstrating writing, updating, and reading specific versions.

Variable Tracker

Variable	Start	After Step 1	After Step 2	After Step 3	After Step 4	Final
data_version	None	0	1	1	1	1
data_content	Empty	[0,1,2,3,4]	[0,1,2,30,4]	[0,1,2,3,4]	[0,1,2,30,4]	[0,1,2,30,4]

Key Moments - 2 Insights

Why does reading with versionAsOf=0 show old data even after update?

Because versionAsOf=0 tells Spark to read the data as it was at version 0, before the update at version 1 (see execution_table row 3).

What happens if you read data without specifying versionAsOf after updates?

Spark reads the latest version of the data, which includes all updates (see execution_table row 5).

Visual Quiz - 3 Questions

Look at the execution table, what is the data content after step 2?

A [30,1,2,3,4]

B [0,1,2,3,4]

C [0,1,2,30,4]

D [0,1,2,3,30]

💡 Hint

Check the 'Data Content' column at step 2 in the execution_table.

At which step does the data version become 1?

A Step 1

B Step 2

C Step 3

D Step 4

💡 Hint

Look at the 'Data Version' column in the execution_table and variable_tracker after each step.

If you want to see the data before any updates, which version should you query?

A versionAsOf=0

B versionAsOf=1

C versionAsOf=2

D No version option

💡 Hint

Refer to execution_table rows 3 and 4 to see which version shows original data.

Concept Snapshot

Time travel in Delta Lake lets you save data in versions.
You can query any past version using option('versionAsOf', version_number).
Updates create new versions without deleting old data.
This helps audit changes or rollback data.
Use DeltaTable API to update or read versions.
Always specify version to see historical data.

Full Transcript

This lesson shows how Delta Lake saves data in versions and allows querying past versions using time travel. We start by writing initial data as version 0. Then we update the data, creating version 1. Using the option 'versionAsOf', we can read data as it was at any version. Reading with versionAsOf=0 shows the original data before updates. Reading without specifying version shows the latest data. This helps in auditing and recovering data. The execution table traces each step, showing data content and versions. The variable tracker shows how data_version and data_content change after each step. Key moments clarify why querying old versions shows old data and what happens when no version is specified. The quiz tests understanding of data content and versions at each step. The snapshot summarizes the main points about time travel and versioning in Delta Lake.

Practice

1. What is the main purpose of time travel in Apache Spark's Delta Lake?

easy

A. To access previous versions of data easily

B. To speed up data processing

C. To create new tables automatically

D. To delete old data permanently

Solution

1. Step 1: Understand time travel concept

   Time travel allows users to look at past versions of data stored in Delta Lake.

2. Step 2: Identify main use

   This feature helps in recovering old data, auditing changes, and comparing versions.

3. Final Answer:

   To access previous versions of data easily -> Option A

4. Quick Check:

   Time travel = Access past data versions [OK]

Hint: Time travel means looking back at old data versions [OK]

Common Mistakes:

• Thinking time travel speeds up processing
• Confusing time travel with data deletion
• Assuming it creates new tables automatically

2. Which option is the correct way to load a Delta table at a specific version using Spark?

easy

A. spark.read.format('delta').option('version', 5).load('/data/table')

B. spark.read.format('delta').option('versionAsOf', 5).load('/data/table')

C. spark.read.format('delta').option('timestampAsOf', '5').load('/data/table')

D. spark.read.format('delta').option('versionAt', 5).load('/data/table')

Solution

1. Step 1: Recall correct option name for version

   The correct option to specify version is 'versionAsOf'.

2. Step 2: Check syntax correctness

   spark.read.format('delta').option('versionAsOf', 5).load('/data/table') uses 'versionAsOf' with a number, which is correct syntax.

3. Final Answer:

   spark.read.format('delta').option('versionAsOf', 5).load('/data/table') -> Option B

4. Quick Check:

   Use 'versionAsOf' to load specific version [OK]

Hint: Use 'versionAsOf' option to load specific version [OK]

Common Mistakes:

• Using 'version' instead of 'versionAsOf'
• Passing version as string in 'timestampAsOf'
• Using wrong option names like 'versionAt'

3. Given the code below, what will be the output count?

df = spark.read.format('delta').option('versionAsOf', 2).load('/data/events')
print(df.count())

Assuming version 2 of the table has 150 rows.

medium

A. Error due to wrong option

B. 0

C. Latest row count

D. 150

Solution

1. Step 1: Understand versionAsOf usage

   The option 'versionAsOf' loads the table exactly at version 2.

2. Step 2: Check row count at version 2

   Since version 2 has 150 rows, the count will be 150.

3. Final Answer:

   150 -> Option D

4. Quick Check:

   Count at version 2 = 150 [OK]

Hint: Count matches rows in specified version [OK]

Common Mistakes:

• Assuming count is zero for old versions
• Confusing latest count with versioned count
• Expecting error from correct syntax

4. Identify the error in the following code snippet for loading a Delta table at a timestamp:

df = spark.read.format('delta').option('timestampAsOf', 20230101).load('/data/sales')

medium

A. Load path is incorrect

B. timestampAsOf option does not exist

C. timestampAsOf value should be a string with date-time format

D. Missing .option('versionAsOf', )

Solution

1. Step 1: Check timestampAsOf option usage

   The 'timestampAsOf' option expects a string representing date and time, e.g., '2023-01-01T00:00:00'.

2. Step 2: Identify error in value type

   Here, 20230101 is an integer, not a string with proper format, causing error.

3. Final Answer:

   timestampAsOf value should be a string with date-time format -> Option C

4. Quick Check:

   timestampAsOf needs string date-time [OK]

Hint: Use string date-time format for timestampAsOf [OK]

Common Mistakes:

• Passing integer instead of string for timestamp
• Confusing versionAsOf with timestampAsOf
• Assuming option does not exist

5. You want to compare the number of rows between version 3 and version 5 of a Delta table located at '/data/logs'. Which code snippet correctly loads both versions for comparison?

hard

A. df_v3 = spark.read.format('delta').option('versionAsOf', 3).load('/data/logs')
df_v5 = spark.read.format('delta').option('versionAsOf', 5).load('/data/logs')

B. df_v3 = spark.read.format('delta').option('timestampAsOf', 3).load('/data/logs')
df_v5 = spark.read.format('delta').option('timestampAsOf', 5).load('/data/logs')

C. df_v3 = spark.read.format('delta').option('version', 3).load('/data/logs')
df_v5 = spark.read.format('delta').option('version', 5).load('/data/logs')

D. df_v3 = spark.read.format('delta').option('versionAsOf', '3').load('/data/logs')
df_v5 = spark.read.format('delta').option('versionAsOf', '5').load('/data/logs')

Solution

1. Step 1: Identify correct option for version number

   The option 'versionAsOf' expects an integer version number, not a string.

2. Step 2: Check code correctness for both versions

   df_v3 = spark.read.format('delta').option('versionAsOf', 3).load('/data/logs') df_v5 = spark.read.format('delta').option('versionAsOf', 5).load('/data/logs') uses 'versionAsOf' with integers 3 and 5, correctly loading both versions.

3. Final Answer:

   df_v3 = spark.read.format('delta').option('versionAsOf', 3).load('/data/logs') df_v5 = spark.read.format('delta').option('versionAsOf', 5).load('/data/logs') -> Option A

4. Quick Check:

   Use 'versionAsOf' with integers for version loading [OK]

Hint: Use versionAsOf with integers to load specific versions [OK]

Common Mistakes:

• Using timestampAsOf with integers
• Passing version numbers as strings
• Using non-existent 'version' option