Schema validation in Apache Spark

Introduction

Schema validation helps check if data matches the expected format before using it. This avoids errors and keeps data clean.

When loading data from files like CSV or JSON to ensure columns have correct types.

When receiving data from external sources to confirm it fits the expected structure.

Before running analysis or machine learning to avoid crashes from bad data.

When transforming data to make sure changes keep the right format.

When combining data from different places to ensure consistency.

Syntax

Apache Spark

from pyspark.sql.types import StructType, StructField, StringType, IntegerType

schema = StructType([
    StructField("name", StringType(), True),
    StructField("age", IntegerType(), True)
])

df = spark.read.schema(schema).json("path/to/file.json")

Define a schema using StructType and StructField to specify column names and types.

Use spark.read.schema(schema) to apply the schema when reading data.

Examples

This creates a schema with two columns: name as text and age as number.

Apache Spark

from pyspark.sql.types import StructType, StructField, StringType, IntegerType

schema = StructType([
    StructField("name", StringType(), True),
    StructField("age", IntegerType(), True)
])

Reads a JSON file using the defined schema to check data format.

Apache Spark

df = spark.read.schema(schema).json("people.json")

Shows the schema of the loaded DataFrame to verify column types.

Apache Spark

df.printSchema()

Sample Program

This program creates a Spark session, defines a schema, loads JSON data with that schema, and prints the data and schema.

Apache Spark

from pyspark.sql import SparkSession
from pyspark.sql.types import StructType, StructField, StringType, IntegerType

spark = SparkSession.builder.appName("SchemaValidationExample").getOrCreate()

# Define schema
schema = StructType([
    StructField("name", StringType(), True),
    StructField("age", IntegerType(), True)
])

# Sample data as JSON string
json_data = '[{"name": "Alice", "age": 30}, {"name": "Bob", "age": 25}]'

# Create RDD from JSON string
rdd = spark.sparkContext.parallelize([json_data])

# Read JSON with schema
df = spark.read.option("multiLine", "true").schema(schema).json(rdd)

# Show data
print("DataFrame content:")
df.show()

# Print schema
print("DataFrame schema:")
df.printSchema()

spark.stop()

Important Notes

If data does not match the schema, Spark may set wrong types to null or throw errors depending on settings.

Setting nullable=True means the column can have missing values.

Always check schema after loading data to confirm it matches expectations.

Summary

Schema validation checks data format before analysis.

Define schemas with StructType and StructField.

Apply schema when reading data to catch errors early.