Delta Lake introduction in Apache Spark - Step-by-Step Execution

Concept Flow - Delta Lake introduction

Start with Raw Data

↓

Write Data as Delta Table

↓

Perform Transactions: Insert, Update, Delete

↓

Delta Lake Manages ACID & Versioning

↓

Read Data with Time Travel or Latest Snapshot

↓

Use Data for Analytics or ML

Delta Lake stores data with ACID transactions and version control, allowing safe updates and easy data reads.

Execution Sample

Apache Spark

from delta.tables import DeltaTable

# Write data as Delta
spark.range(5).write.format("delta").save("/tmp/delta-table")

# Read Delta Table
deltaTable = DeltaTable.forPath(spark, "/tmp/delta-table")

This code writes a simple range of numbers as a Delta table and then reads it back.

Execution Table

Step	Action	Input/Condition	Result/Output
1	Create DataFrame	spark.range(5)	DataFrame with numbers 0 to 4
2	Write DataFrame as Delta	format='delta', path='/tmp/delta-table'	Delta table created at path
3	Read Delta Table	DeltaTable.forPath(spark, '/tmp/delta-table')	DeltaTable object referencing saved data
4	Perform Update	deltaTable.update(condition="id == 3", set={"id": 30})	Row with id=3 updated to id=30
5	Read Latest Data	deltaTable.toDF().show()	DataFrame shows updated data with id=30
6	Time Travel Read	spark.read.format('delta').option('versionAsOf', 0).load('/tmp/delta-table')	DataFrame shows original data before update
7	Exit	No more actions	End of demonstration

💡 All steps executed to show Delta Lake write, update, read, and time travel

Variable Tracker

Variable	Start	After Step 1	After Step 2	After Step 3	After Step 4	After Step 5	After Step 6	Final
DataFrame	None	Range(0-4)	Range(0-4)	Range(0-4)	Range with id=30 updated	Range with id=30 updated	Range(0-4)	Range(0-4) or updated depending on read
deltaTable	None	None	None	DeltaTable object created	DeltaTable object	DeltaTable object	DeltaTable object	DeltaTable object

Key Moments - 2 Insights

Why does the data show old values when using time travel after an update?

Because Delta Lake keeps versions of data, time travel reads a previous version (see execution_table step 6) showing data before the update.

What happens if you try to update data without Delta Lake?

Without Delta Lake, updates can cause data corruption or partial writes. Delta Lake ensures ACID transactions (see concept_flow step 'Delta Lake Manages ACID & Versioning').

Visual Quiz - 3 Questions

Look at the execution table, what is the result after step 4 (Perform Update)?

A All rows are deleted

B New row with id=30 is added

C Row with id=3 is changed to id=30

D No change to data

💡 Hint

Check the 'Result/Output' column in execution_table row for step 4

At which step does the DeltaTable object get created?

A Step 2

B Step 3

C Step 1

D Step 5

💡 Hint

Look at the 'Action' and 'Result/Output' columns in execution_table for when DeltaTable.forPath is called

If you skip step 2 (writing data as Delta), what happens when you try step 3?

A Error because no Delta table exists at path

B DeltaTable object is created with empty data

C Data is read from a CSV file instead

D Data is automatically created

💡 Hint

DeltaTable.forPath requires existing Delta data, see concept_flow step 'Write Data as Delta Table'

Concept Snapshot

Delta Lake stores data as tables with ACID transactions.
You write data using format 'delta' and a path.
You can update, delete, and read data safely.
Delta Lake keeps versions for time travel reads.
Use DeltaTable API to manage data programmatically.

Full Transcript

Delta Lake is a storage layer that adds reliability to data lakes. It allows you to write data as Delta tables, which support safe updates and deletes with ACID transactions. You start by writing data in Delta format to a path. Then you can read it back using the DeltaTable API. Delta Lake keeps versions of the data, so you can read previous snapshots using time travel. This example showed writing a range of numbers, updating one row, reading the latest data, and reading an older version. Delta Lake helps keep data consistent and easy to manage for analytics and machine learning.