Schema definition and inference in Apache Spark

Introduction

We use schemas to tell Spark what kind of data to expect. This helps Spark understand and organize data better.

When you want to make sure your data has the right types before analysis.

When loading data from files like CSV or JSON and you want Spark to guess the data types automatically.

When you want to speed up data loading by giving Spark the exact structure.

When you want to avoid errors caused by wrong data types in your data.

When you want to control how Spark reads complex data formats.

Syntax

Apache Spark

from pyspark.sql.types import StructType, StructField, StringType, IntegerType

schema = StructType([
    StructField('name', StringType(), True),
    StructField('age', IntegerType(), True)
])

df = spark.read.schema(schema).csv('path/to/file.csv', header=True, inferSchema=False)

# Or to let Spark guess schema automatically:
df_auto = spark.read.csv('path/to/file.csv', header=True, inferSchema=True)

Use StructType and StructField to define schema manually.

Set inferSchema=True to let Spark guess data types automatically.

Examples

This example defines a schema with two columns: 'name' as text and 'age' as number. Then it reads a CSV file using this schema.

Apache Spark

from pyspark.sql.types import StructType, StructField, StringType, IntegerType

schema = StructType([
    StructField('name', StringType(), True),
    StructField('age', IntegerType(), True)
])

df = spark.read.schema(schema).csv('people.csv', header=True, inferSchema=False)

This example lets Spark guess the data types by setting inferSchema=True. It is easier but less controlled.

Apache Spark

df_auto = spark.read.csv('people.csv', header=True, inferSchema=True)

Sample Program

This program creates a small CSV file, then reads it twice: once with a manual schema and once with schema inference. It prints the schema and data each time.

Apache Spark

from pyspark.sql import SparkSession
from pyspark.sql.types import StructType, StructField, StringType, IntegerType

spark = SparkSession.builder.appName('SchemaExample').getOrCreate()

# Define schema manually
schema = StructType([
    StructField('name', StringType(), True),
    StructField('age', IntegerType(), True)
])

# Create sample data file
sample_data = 'name,age\nAlice,30\nBob,25\nCharlie,35'
with open('people.csv', 'w') as f:
    f.write(sample_data)

# Read with manual schema
df_manual = spark.read.schema(schema).csv('people.csv', header=True, inferSchema=False)
print('Manual Schema:')
df_manual.printSchema()
df_manual.show()

# Read with inferred schema
df_infer = spark.read.csv('people.csv', header=True, inferSchema=True)
print('Inferred Schema:')
df_infer.printSchema()
df_infer.show()

spark.stop()

Important Notes

Manual schema helps catch errors early by enforcing data types.

Schema inference is convenient but can be slower on big data.

Always check the schema after loading data to avoid surprises.

Summary

Schemas tell Spark the structure and types of your data.

You can define schemas manually or let Spark guess them.

Using schemas helps make data processing safer and clearer.