Delta Lake introduction in Apache Spark - Mini Project: Build & Apply

Delta Lake Introduction with Apache Spark

📖 Scenario: You work as a data analyst at a retail company. You receive daily sales data as CSV files. You want to store this data efficiently and safely so you can update it easily and query it fast. Your team decides to use Delta Lake on Apache Spark to manage this data.

🎯 Goal: Build a simple Delta Lake table from a small dataset, configure a write mode, and read the data back to see the results.

📋 What You'll Learn

Create a Spark DataFrame with sample sales data

Write the DataFrame to a Delta Lake table

Set the write mode to 'overwrite' to replace existing data

Read the Delta Lake table back into a DataFrame

Show the contents of the DataFrame

💡 Why This Matters

🌍 Real World

Delta Lake helps companies store and manage large amounts of data reliably. It supports updates, deletes, and fast queries, which are important for daily business reports and analytics.

💼 Career

Data engineers and data analysts use Delta Lake to build robust data pipelines and ensure data quality in big data environments.

1

Create a Spark DataFrame with sales data

Create a Spark DataFrame called sales_df with these exact rows: ("2024-06-01", "StoreA", 100), ("2024-06-01", "StoreB", 150), and ("2024-06-02", "StoreA", 200). Use columns named date, store, and sales.

Apache Spark

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("DeltaLakeIntro").getOrCreate()
# Create the sales_df DataFrame with the specified data and columns
# Your code here

Need a hint?

Use spark.createDataFrame(data, schema=columns) where data is a list of tuples and columns is a list of column names.

2

Set the Delta Lake write mode

Create a variable called write_mode and set it to the string "overwrite" to replace existing data when writing.

Apache Spark

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("DeltaLakeIntro").getOrCreate()
data = [("2024-06-01", "StoreA", 100), ("2024-06-01", "StoreB", 150), ("2024-06-02", "StoreA", 200)]
columns = ["date", "store", "sales"]
sales_df = spark.createDataFrame(data, schema=columns)
# Create a variable write_mode and set it to "overwrite"
# Your code here

Need a hint?

Just assign the string "overwrite" to the variable write_mode.

3

Write the DataFrame to a Delta Lake table

Use sales_df.write.format("delta") with mode(write_mode) to write the data to a Delta Lake table at path "/tmp/delta/sales".

Apache Spark

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("DeltaLakeIntro").getOrCreate()
data = [("2024-06-01", "StoreA", 100), ("2024-06-01", "StoreB", 150), ("2024-06-02", "StoreA", 200)]
columns = ["date", "store", "sales"]
sales_df = spark.createDataFrame(data, schema=columns)
write_mode = "overwrite"
# Write sales_df to Delta Lake at "/tmp/delta/sales" using write_mode
# Your code here

Need a hint?

Use sales_df.write.format("delta").mode(write_mode).save(path) to write the DataFrame.

4

Read and display the Delta Lake table

Read the Delta Lake table from "/tmp/delta/sales" into a DataFrame called delta_df using spark.read.format("delta"). Then print the contents of delta_df using show().

Apache Spark

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("DeltaLakeIntro").getOrCreate()
data = [("2024-06-01", "StoreA", 100), ("2024-06-01", "StoreB", 150), ("2024-06-02", "StoreA", 200)]
columns = ["date", "store", "sales"]
sales_df = spark.createDataFrame(data, schema=columns)
write_mode = "overwrite"
sales_df.write.format("delta").mode(write_mode).save("/tmp/delta/sales")
# Read the Delta Lake table into delta_df and show its contents
# Your code here

Need a hint?

Use spark.read.format("delta").load(path) to read the table, then show() to display.