Schema evolution and enforcement in Apache Spark - Step-by-Step Execution

Concept Flow - Schema evolution and enforcement

Start with initial schema

↓

Read data with schema enforcement

↓

Check if new data matches schema?

No→Schema evolution triggered

↓

Update schema with new fields

↓

Process data with enforced schema

↓

Save or use data

This flow shows how Spark reads data with a schema, enforces it, and updates the schema if new data has changes.

Execution Sample

Apache Spark

from pyspark.sql import SparkSession

spark = SparkSession.builder.getOrCreate()

# Initial schema read
df1 = spark.read.schema('id INT, name STRING').json('data1.json')

# New data with extra column
df2 = spark.read.schema('id INT, name STRING').json('data2.json')

# Read with schema evolution
df2_evolved = spark.read.option('mergeSchema', 'true').schema('id INT, name STRING, age INT').json('data2.json')

This code reads JSON data with a fixed schema, then reads new data with an extra column using schema evolution.

Execution Table

Step	Action	Schema Used	Data Columns	Result
1	Read data1.json with schema enforcement	id INT, name STRING	id, name	Data loaded with 2 columns
2	Read data2.json with same schema enforcement	id INT, name STRING	id, name, age (extra in data)	Error or missing 'age' column
3	Read data2.json with mergeSchema option	id INT, name STRING + mergeSchema	id, name, age	Schema evolved, data loaded with 3 columns
4	Process data with evolved schema	id INT, name STRING, age INT	id, name, age	Data processed successfully

💡 Execution stops after data is processed with the evolved schema including new columns.

Variable Tracker

Variable	Start	After Step 1	After Step 2	After Step 3	Final
df1.schema	None	id INT, name STRING	id INT, name STRING	id INT, name STRING	id INT, name STRING
df2.schema	None	None	id INT, name STRING	id INT, name STRING, age INT	id INT, name STRING, age INT
df2_evolved.schema	None	None	None	id INT, name STRING, age INT	id INT, name STRING, age INT

Key Moments - 2 Insights

Why does reading data2.json with the original schema cause an error or missing column?

Because the original schema does not include the new 'age' column present in data2.json, Spark enforces the schema strictly and ignores or errors on unexpected columns (see execution_table step 2).

How does the mergeSchema option help with schema evolution?

The mergeSchema option tells Spark to combine the original schema with new columns found in the data, allowing the schema to evolve and include new fields (see execution_table step 3).

Visual Quiz - 3 Questions

Look at the execution table, what columns does df2 have when read without mergeSchema at step 2?

A id, name, age

B id, name

C age only

D No columns

💡 Hint

Check the 'Data Columns' column in execution_table row for step 2.

At which step does Spark successfully load data with the new 'age' column?

A Step 3

B Step 2

C Step 1

D Step 4

💡 Hint

Look for when 'age' appears in the 'Data Columns' and 'Result' shows successful loading.

If we remove the mergeSchema option, what happens when new columns appear in data?

A Spark automatically adds new columns

B Spark deletes old columns

C Spark throws an error or ignores new columns

D Spark merges schemas anyway

💡 Hint

Refer to execution_table step 2 and key_moments about schema enforcement.

Concept Snapshot

Schema evolution in Spark means updating the schema when new data has extra columns.
Schema enforcement means Spark reads data strictly matching the schema.
Use option('mergeSchema', 'true') to allow schema evolution.
Without mergeSchema, new columns cause errors or are ignored.
Schema evolution helps keep data processing flexible and consistent.

Full Transcript

Schema evolution and enforcement in Apache Spark control how Spark reads and processes data with schemas. Initially, Spark reads data using a fixed schema, enforcing that the data matches exactly. If new data files have extra columns, reading them with the original schema causes errors or missing data. To handle this, Spark offers schema evolution using the mergeSchema option, which merges the original schema with new columns found in the data. This allows Spark to load and process data with updated schemas without errors. The execution trace shows reading data with and without schema evolution, highlighting how the schema changes and how data columns are handled. Beginners often get confused why new columns cause errors without mergeSchema and how schema evolution fixes this. The visual quiz tests understanding of these steps and outcomes. Overall, schema evolution and enforcement help maintain data quality and flexibility in Spark data pipelines.

Practice

1. What does schema evolution in Apache Spark allow you to do?

easy

A. Add or change columns without breaking old data

B. Automatically delete old data files

C. Enforce strict data types on all columns

D. Compress data files for faster reading

Solution

1. Step 1: Understand schema evolution concept

   Schema evolution means you can update the schema by adding or changing columns without losing compatibility with existing data.

2. Step 2: Compare options with definition

   Only Add or change columns without breaking old data correctly describes this ability. Other options describe unrelated features.

3. Final Answer:

   Add or change columns without breaking old data -> Option A

4. Quick Check:

   Schema evolution = Add/change columns safely [OK]

Hint: Schema evolution means changing schema without breaking data [OK]

Common Mistakes:

• Confusing schema evolution with schema enforcement
• Thinking schema evolution deletes old data
• Assuming schema evolution compresses data

2. Which option correctly enables schema merging when reading Parquet files in Spark?

easy

A. spark.read.option("schemaEvolution", "enabled").parquet(path)

B. spark.read.option("enforceSchema", "true").parquet(path)

C. spark.read.option("merge", "false").parquet(path)

D. spark.read.option("mergeSchema", "true").parquet(path)

Solution

1. Step 1: Recall correct option name for schema merging

   The correct option to merge schemas is "mergeSchema" set to "true".

2. Step 2: Verify syntax correctness

   spark.read.option("mergeSchema", "true").parquet(path) uses the exact syntax spark.read.option("mergeSchema", "true").parquet(path), which is valid.

3. Final Answer:

   spark.read.option("mergeSchema", "true").parquet(path) -> Option D

4. Quick Check:

   Use mergeSchema=true to merge schemas [OK]

Hint: Use mergeSchema option set to true for schema merging [OK]

Common Mistakes:

• Using wrong option names like enforceSchema or schemaEvolution
• Setting mergeSchema to false disables merging
• Confusing option names with other Spark settings

3. Given two Parquet files with schemas:
file1: (id INT, name STRING)
file2: (id INT, name STRING, age INT)
What will be the schema of the DataFrame when reading both files with mergeSchema=true?

medium

A. (id INT, name STRING, age INT)

B. (id INT, name STRING)

C. (id INT, age INT)

D. (name STRING, age INT)

Solution

1. Step 1: Understand schema merging behavior

   When mergeSchema=true, Spark combines all columns from all files, including new columns.

2. Step 2: Combine columns from both schemas

   file1 has (id, name), file2 adds (age). Merged schema includes all three columns.

3. Final Answer:

   (id INT, name STRING, age INT) -> Option A

4. Quick Check:

   Merged schema includes all columns from files [OK]

Hint: Merged schema includes all columns from all files [OK]

Common Mistakes:

• Ignoring new columns in second file
• Assuming schema stays as first file only
• Dropping columns not in all files

4. You try to read Parquet files with different schemas but get an error about schema mismatch. What is the likely cause?

medium

A. The files are corrupted

B. You did not set mergeSchema option to true

C. You used the wrong file path

D. Spark does not support schema evolution

Solution

1. Step 1: Identify cause of schema mismatch error

   Schema mismatch errors happen when Spark tries to read files with different schemas without merging enabled.

2. Step 2: Check mergeSchema option usage

   Not setting mergeSchema=true causes Spark to expect identical schemas, leading to errors.

3. Final Answer:

   You did not set mergeSchema option to true -> Option B

4. Quick Check:

   Enable mergeSchema=true to avoid schema mismatch errors [OK]

Hint: Set mergeSchema=true to fix schema mismatch errors [OK]

Common Mistakes:

• Blaming file corruption without checking options
• Assuming Spark cannot evolve schemas
• Ignoring mergeSchema option

5. You have a Delta Lake table with schema (id INT, name STRING). You want to add a new column age INT and keep old data accessible. Which approach correctly applies schema evolution and enforcement?

hard

A. Overwrite the table schema manually without options

B. Delete old data and write new data with new schema

C. Use ALTER TABLE ADD COLUMN age INT and write new data with mergeSchema=true

D. Read old data without options and append new data with new schema

Solution

1. Step 1: Add new column using ALTER TABLE

   ALTER TABLE ADD COLUMN adds the new column to the Delta table schema safely.

2. Step 2: Write new data with mergeSchema=true

   Writing with mergeSchema=true allows Spark to merge new schema with old data without errors.

3. Final Answer:

   Use ALTER TABLE ADD COLUMN age INT and write new data with mergeSchema=true -> Option C

4. Quick Check:

   ALTER TABLE + mergeSchema=true enables safe schema evolution [OK]

Hint: Use ALTER TABLE and mergeSchema=true to evolve Delta schema safely [OK]

Common Mistakes:

• Overwriting schema without mergeSchema causes data loss
• Deleting old data loses history
• Appending without mergeSchema causes errors