Schema validation in Apache Spark - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

Output of schema validation with missing required field

Given the following Spark DataFrame schema and data, what will be the output when validating the data against the schema?

Apache Spark

from pyspark.sql import SparkSession
from pyspark.sql.types import StructType, StructField, StringType, IntegerType

spark = SparkSession.builder.master('local').appName('SchemaValidation').getOrCreate()

schema = StructType([
    StructField('name', StringType(), nullable=False),
    StructField('age', IntegerType(), nullable=False)
])

data = [('Alice', 30), (None, 25)]

df = spark.createDataFrame(data, schema=schema)
df.show()

A Throws an error during DataFrame creation due to null in non-nullable field

B Shows both rows; second row has null for 'name' without error

C Shows only the first row; second row is dropped silently

D Shows both rows; null in 'name' replaced by empty string

💡 Hint

Think about how Spark handles nullable fields when creating DataFrames.

Spark allows creating DataFrames with nulls in non-nullable fields but may cause errors later during operations like writing to certain formats. The show() method will display both rows including the null value.

❓ data_output

intermediate

Result of schema enforcement with cast

What is the output DataFrame when applying the schema with enforced types to the following data?

Apache Spark

from pyspark.sql import SparkSession
from pyspark.sql.types import StructType, StructField, IntegerType

spark = SparkSession.builder.master('local').appName('SchemaValidation').getOrCreate()

schema = StructType([
    StructField('id', IntegerType(), nullable=False)
])

data = [('1',), ('2',), ('three',)]

df = spark.createDataFrame(data, ['id'])
df_cast = df.selectExpr('cast(id as int) as id')
df_cast.show()

A

+----+
|  id|
+----+
|null|
|null|
|null|
+----+

B Throws a runtime error due to 'three' not castable to int

C

+----+
|  id|
+----+
|   1|
|   2|
|   3|
+----+

D

+----+
|  id|
+----+
|   1|
|   2|
|null|
+----+

💡 Hint

Casting invalid strings to int results in null values in Spark.

When casting 'three' to int, Spark returns null instead of throwing an error. The other values are cast correctly.

🔧 Debug

advanced

Identify the error in schema validation code

What error will this Spark code raise when trying to create a DataFrame with the given schema and data?

Apache Spark

from pyspark.sql import SparkSession
from pyspark.sql.types import StructType, StructField, StringType, IntegerType

spark = SparkSession.builder.master('local').appName('SchemaValidation').getOrCreate()

schema = StructType([
    StructField('name', StringType(), nullable=False),
    StructField('age', IntegerType(), nullable=False)
])

data = [('Bob', 'thirty')]

df = spark.createDataFrame(data, schema=schema)
df.show()

A TypeError: Data type mismatch for field 'age'

B RuntimeException: Failed to convert value 'thirty' to int

C No error; shows 'Bob' and null for age

D ValueError: Cannot cast string 'thirty' to IntegerType

💡 Hint

Consider how Spark handles invalid type values during DataFrame creation.

Spark does not throw an error at DataFrame creation for invalid type values but sets them to null. The show() method will display 'Bob' and null for age.

🧠 Conceptual

advanced

Understanding nullable field behavior in Spark schema

Which statement best describes how Spark handles nullable fields in a schema during DataFrame operations?

A Spark allows nulls in non-nullable fields during DataFrame creation but may fail during write operations

B Spark enforces non-nullable fields strictly and throws errors if nulls appear at any time

C Spark automatically replaces nulls in non-nullable fields with default values

D Spark ignores nullable settings and treats all fields as nullable

💡 Hint

Think about when Spark enforces schema constraints strictly.

Spark does not enforce non-nullable constraints strictly during DataFrame creation or transformations but may throw errors when writing data to external storage that requires strict schema adherence.

🚀 Application

expert

Detecting schema mismatch in streaming data ingestion

You have a Spark Structured Streaming job reading JSON data with a predefined schema. If incoming JSON messages have missing required fields, what will happen during streaming processing?

A Rows with missing required fields will have nulls in those fields without failing the job

B The streaming job will fail immediately with a schema mismatch error

C Spark will skip rows missing required fields silently without warning

D Spark will automatically infer missing fields and fill them with default values

💡 Hint

Consider how Spark handles schema enforcement in streaming ingestion.

In Spark Structured Streaming, missing fields in JSON data are set to null if the schema expects them. The job continues running unless explicit validation or checks are added.

Practice

1. What is the main purpose of schema validation in Apache Spark?

easy

A. To check if the data matches the expected format before processing

B. To speed up the Spark job execution

C. To automatically clean the data

D. To visualize data distributions

Solution

1. Step 1: Understand schema validation role

   Schema validation ensures data matches the expected structure and types before analysis.

2. Step 2: Identify the main goal

   This helps catch errors early and prevents processing wrong data formats.

3. Final Answer:

   To check if the data matches the expected format before processing -> Option A

4. Quick Check:

   Schema validation = format check [OK]

Hint: Schema validation means checking data format first [OK]

Common Mistakes:

• Thinking schema speeds up execution directly
• Assuming schema cleans data automatically
• Confusing schema validation with visualization

2. Which of the following is the correct way to define a schema with two fields: 'name' as string and 'age' as integer in Spark?

easy

A. Schema(['name': 'string', 'age': 'int'])

B. StructType('name' StringType, 'age' IntegerType)

C. StructType([StructField('name', StringType()), StructField('age', IntegerType())])

D. StructField('name', 'string'), StructField('age', 'int')

Solution

1. Step 1: Recall correct schema syntax

   In Spark, schema is defined using StructType with a list of StructField objects specifying name and type.

2. Step 2: Match syntax to options

   StructType([StructField('name', StringType()), StructField('age', IntegerType())]) correctly uses StructType with StructField and proper types.

3. Final Answer:

   StructType([StructField('name', StringType()), StructField('age', IntegerType())]) -> Option C

4. Quick Check:

   Schema = StructType + StructField [OK]

Hint: Use StructType with list of StructField for schema [OK]

Common Mistakes:

• Omitting StructType wrapper
• Using wrong syntax without brackets
• Confusing schema with dictionary or list

3. Given this code snippet:

from pyspark.sql.types import StructType, StructField, StringType, IntegerType
schema = StructType([
  StructField('name', StringType(), True),
  StructField('age', IntegerType(), True)
])
data = [('Alice', 30), ('Bob', 'thirty')]
df = spark.createDataFrame(data, schema)
df.show()

What will happen when this code runs?

medium

A. It will show both rows with 'age' as string for Bob

B. It will raise an error due to type mismatch in 'age' for Bob

C. It will convert 'thirty' to null silently

D. It will ignore the second row and show only the first

Solution

1. Step 1: Analyze schema and data types

   The schema expects 'age' as IntegerType, but Bob's age is a string 'thirty'.

2. Step 2: Understand Spark behavior on type mismatch

   Spark will raise a runtime error when data does not match schema types during DataFrame creation.

3. Final Answer:

   It will raise an error due to type mismatch in 'age' for Bob -> Option B

4. Quick Check:

   Type mismatch causes error [OK]

Hint: Mismatch in data type causes error, not silent conversion [OK]

Common Mistakes:

• Assuming Spark converts wrong types silently
• Thinking Spark ignores bad rows by default
• Believing all rows display regardless of errors

4. You wrote this schema and code:

schema = StructType([
  StructField('id', IntegerType()),
  StructField('value', StringType())
])
data = [(1, 'a'), (2, 'b')]
df = spark.createDataFrame(data, schema)
df.printSchema()

But you get an error: TypeError: StructField() missing 1 required positional argument: 'nullable'. What is the fix?

medium

A. Add the 'nullable' argument to each StructField, e.g. StructField('id', IntegerType(), True)

B. Remove the schema and let Spark infer it

C. Change StructType to SchemaType

D. Use a list of tuples instead of StructField

Solution

1. Step 1: Understand StructField parameters

   StructField requires three arguments: name, data type, and nullable (True/False).

2. Step 2: Fix the code by adding nullable

   Add the third argument 'nullable=True' to each StructField to fix the error.

3. Final Answer:

   Add the 'nullable' argument to each StructField, e.g. StructField('id', IntegerType(), True) -> Option A

4. Quick Check:

   StructField needs nullable argument [OK]

Hint: Always include nullable in StructField [OK]

Common Mistakes:

• Forgetting nullable argument
• Trying to rename StructType
• Ignoring error and running code

5. You have a CSV file with columns: 'user', 'score', and 'date'. You want to read it with Spark and ensure 'score' is an integer and 'date' is a date type. Which schema definition and read method is correct to validate this?

hard

A. schema = StructType([StructField('user', StringType(), False), StructField('score', IntegerType(), False), StructField('date', DateType(), False)])
df = spark.read.option('header', 'true').csv('file.csv', schema=schema)

B. schema = StructType([StructField('user', StringType()), StructField('score', StringType()), StructField('date', StringType())])
df = spark.read.csv('file.csv', schema=schema)

C. df = spark.read.csv('file.csv', inferSchema=True, header=True)

D. schema = StructType([StructField('user', StringType(), True), StructField('score', IntegerType(), True), StructField('date', DateType(), True)])
df = spark.read.csv('file.csv', schema=schema, header=True)

Solution

1. Step 1: Define schema with correct types and nullable

   Use StructType with StructField for 'user' as string, 'score' as integer, and 'date' as DateType, all nullable True.

2. Step 2: Read CSV with schema and header option

   Use spark.read.csv with schema argument and header=True to apply schema validation on read.

3. Final Answer:

   schema = StructType([StructField('user', StringType(), True), StructField('score', IntegerType(), True), StructField('date', DateType(), True)]) df = spark.read.csv('file.csv', schema=schema, header=True) -> Option D

4. Quick Check:

   Schema + header=True validates CSV read [OK]

Hint: Define schema with types, use header=True when reading CSV [OK]

Common Mistakes:

• Using StringType for all columns ignoring types
• Not setting header=True for CSV with headers
• Setting nullable=False unnecessarily causing errors