Schema validation in Apache Spark - Mini Project: Build & Apply

Schema validation

📖 Scenario: You work in a company that collects customer data. You want to make sure the data is clean and follows the correct format before analysis.

🎯 Goal: You will create a Spark DataFrame with customer data, define a schema to validate the data types, and check if the data matches the schema.

📋 What You'll Learn

Create a Spark DataFrame with given customer data

Define a schema with correct data types for each column

Apply the schema to the DataFrame to validate data

Show the validated DataFrame

💡 Why This Matters

🌍 Real World

Data scientists often receive raw data that may have inconsistent types. Schema validation helps ensure data quality before analysis.

💼 Career

Knowing how to define and apply schemas in Spark is essential for data engineers and data scientists working with big data pipelines.

1

Create customer data as a list of tuples

Create a variable called customer_data as a list of tuples with these exact entries: (1, 'Alice', 29), (2, 'Bob', 35), (3, 'Charlie', 40).

Apache Spark

# Create a variable called customer_data with the given tuples
# Your code here

Hint

Use square brackets to create a list and parentheses for each tuple.

2

Define the schema for the customer data

Create a variable called schema using StructType with three fields: id as IntegerType(), name as StringType(), and age as IntegerType(). Import the necessary types from pyspark.sql.types.

Apache Spark

customer_data = [(1, 'Alice', 29), (2, 'Bob', 35), (3, 'Charlie', 40)]
# Import StructType, StructField, IntegerType, StringType
# Define schema with fields id, name, age
# Your code here

Hint

Use StructType and StructField to define the schema fields.

3

Create a Spark DataFrame with the schema

Create a Spark DataFrame called df from customer_data using spark.createDataFrame() and apply the schema to it.

Apache Spark

from pyspark.sql.types import StructType, StructField, IntegerType, StringType

customer_data = [(1, 'Alice', 29), (2, 'Bob', 35), (3, 'Charlie', 40)]

schema = StructType([
    StructField('id', IntegerType(), True),
    StructField('name', StringType(), True),
    StructField('age', IntegerType(), True)
])

# Create a DataFrame called df using spark.createDataFrame with customer_data and schema
# Your code here

Hint

Use spark.createDataFrame() with the schema argument.

4

Show the validated DataFrame

Use printSchema() on df to display the schema, then use df.show() to display the data.

Apache Spark

from pyspark.sql.types import StructType, StructField, IntegerType, StringType

customer_data = [(1, 'Alice', 29), (2, 'Bob', 35), (3, 'Charlie', 40)]

schema = StructType([
    StructField('id', IntegerType(), True),
    StructField('name', StringType(), True),
    StructField('age', IntegerType(), True)
])

df = spark.createDataFrame(customer_data, schema=schema)

# Print the schema of df
# Show the data in df
# Your code here

Hint

Use df.printSchema() to see the schema and df.show() to see the data.

Practice

1. What is the main purpose of schema validation in Apache Spark?

easy

A. To check if the data matches the expected format before processing

B. To speed up the Spark job execution

C. To automatically clean the data

D. To visualize data distributions

Solution

1. Step 1: Understand schema validation role

   Schema validation ensures data matches the expected structure and types before analysis.

2. Step 2: Identify the main goal

   This helps catch errors early and prevents processing wrong data formats.

3. Final Answer:

   To check if the data matches the expected format before processing -> Option A

4. Quick Check:

   Schema validation = format check [OK]

Hint: Schema validation means checking data format first [OK]

Common Mistakes:

• Thinking schema speeds up execution directly
• Assuming schema cleans data automatically
• Confusing schema validation with visualization

2. Which of the following is the correct way to define a schema with two fields: 'name' as string and 'age' as integer in Spark?

easy

A. Schema(['name': 'string', 'age': 'int'])

B. StructType('name' StringType, 'age' IntegerType)

C. StructType([StructField('name', StringType()), StructField('age', IntegerType())])

D. StructField('name', 'string'), StructField('age', 'int')

Solution

1. Step 1: Recall correct schema syntax

   In Spark, schema is defined using StructType with a list of StructField objects specifying name and type.

2. Step 2: Match syntax to options

   StructType([StructField('name', StringType()), StructField('age', IntegerType())]) correctly uses StructType with StructField and proper types.

3. Final Answer:

   StructType([StructField('name', StringType()), StructField('age', IntegerType())]) -> Option C

4. Quick Check:

   Schema = StructType + StructField [OK]

Hint: Use StructType with list of StructField for schema [OK]

Common Mistakes:

• Omitting StructType wrapper
• Using wrong syntax without brackets
• Confusing schema with dictionary or list

3. Given this code snippet:

from pyspark.sql.types import StructType, StructField, StringType, IntegerType
schema = StructType([
  StructField('name', StringType(), True),
  StructField('age', IntegerType(), True)
])
data = [('Alice', 30), ('Bob', 'thirty')]
df = spark.createDataFrame(data, schema)
df.show()

What will happen when this code runs?

medium

A. It will show both rows with 'age' as string for Bob

B. It will raise an error due to type mismatch in 'age' for Bob

C. It will convert 'thirty' to null silently

D. It will ignore the second row and show only the first

Solution

1. Step 1: Analyze schema and data types

   The schema expects 'age' as IntegerType, but Bob's age is a string 'thirty'.

2. Step 2: Understand Spark behavior on type mismatch

   Spark will raise a runtime error when data does not match schema types during DataFrame creation.

3. Final Answer:

   It will raise an error due to type mismatch in 'age' for Bob -> Option B

4. Quick Check:

   Type mismatch causes error [OK]

Hint: Mismatch in data type causes error, not silent conversion [OK]

Common Mistakes:

• Assuming Spark converts wrong types silently
• Thinking Spark ignores bad rows by default
• Believing all rows display regardless of errors

4. You wrote this schema and code:

schema = StructType([
  StructField('id', IntegerType()),
  StructField('value', StringType())
])
data = [(1, 'a'), (2, 'b')]
df = spark.createDataFrame(data, schema)
df.printSchema()

But you get an error: TypeError: StructField() missing 1 required positional argument: 'nullable'. What is the fix?

medium

A. Add the 'nullable' argument to each StructField, e.g. StructField('id', IntegerType(), True)

B. Remove the schema and let Spark infer it

C. Change StructType to SchemaType

D. Use a list of tuples instead of StructField

Solution

1. Step 1: Understand StructField parameters

   StructField requires three arguments: name, data type, and nullable (True/False).

2. Step 2: Fix the code by adding nullable

   Add the third argument 'nullable=True' to each StructField to fix the error.

3. Final Answer:

   Add the 'nullable' argument to each StructField, e.g. StructField('id', IntegerType(), True) -> Option A

4. Quick Check:

   StructField needs nullable argument [OK]

Hint: Always include nullable in StructField [OK]

Common Mistakes:

• Forgetting nullable argument
• Trying to rename StructType
• Ignoring error and running code

5. You have a CSV file with columns: 'user', 'score', and 'date'. You want to read it with Spark and ensure 'score' is an integer and 'date' is a date type. Which schema definition and read method is correct to validate this?

hard

A. schema = StructType([StructField('user', StringType(), False), StructField('score', IntegerType(), False), StructField('date', DateType(), False)])
df = spark.read.option('header', 'true').csv('file.csv', schema=schema)

B. schema = StructType([StructField('user', StringType()), StructField('score', StringType()), StructField('date', StringType())])
df = spark.read.csv('file.csv', schema=schema)

C. df = spark.read.csv('file.csv', inferSchema=True, header=True)

D. schema = StructType([StructField('user', StringType(), True), StructField('score', IntegerType(), True), StructField('date', DateType(), True)])
df = spark.read.csv('file.csv', schema=schema, header=True)

Solution

1. Step 1: Define schema with correct types and nullable

   Use StructType with StructField for 'user' as string, 'score' as integer, and 'date' as DateType, all nullable True.

2. Step 2: Read CSV with schema and header option

   Use spark.read.csv with schema argument and header=True to apply schema validation on read.

3. Final Answer:

   schema = StructType([StructField('user', StringType(), True), StructField('score', IntegerType(), True), StructField('date', DateType(), True)]) df = spark.read.csv('file.csv', schema=schema, header=True) -> Option D

4. Quick Check:

   Schema + header=True validates CSV read [OK]

Hint: Define schema with types, use header=True when reading CSV [OK]

Common Mistakes:

• Using StringType for all columns ignoring types
• Not setting header=True for CSV with headers
• Setting nullable=False unnecessarily causing errors