Schema definition and inference in Apache Spark - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

Output of schema inference with mixed data types

What is the output schema of the following Spark DataFrame code snippet?

Apache Spark

from pyspark.sql import SparkSession
from pyspark.sql.types import *
spark = SparkSession.builder.getOrCreate()
data = [(1, 'Alice', 29), (2, 'Bob', 'thirty'), (3, 'Cathy', 25)]
df = spark.createDataFrame(data, ['id', 'name', 'age'])
df.printSchema()

A

root
 |-- id: integer (nullable = true)
 |-- name: string (nullable = true)
 |-- age: integer (nullable = true)

B

root
 |-- id: long (nullable = true)
 |-- name: string (nullable = true)
 |-- age: string (nullable = true)

C

root
 |-- id: long (nullable = true)
 |-- name: string (nullable = true)
 |-- age: long (nullable = true)

D

root
 |-- id: string (nullable = true)
 |-- name: string (nullable = true)
 |-- age: string (nullable = true)

💡 Hint

Spark infers the column type based on all values. Mixed types in a column cause Spark to choose a common type that can hold all values.

Since the 'age' column has both integers and a string ('thirty'), Spark infers the entire column as string type to accommodate all values.

❓ data_output

intermediate

Number of columns after explicit schema definition

Given the explicit schema below, how many columns will the resulting DataFrame have?

Apache Spark

from pyspark.sql import SparkSession
from pyspark.sql.types import StructType, StructField, StringType, IntegerType
spark = SparkSession.builder.getOrCreate()
schema = StructType([
    StructField('name', StringType(), True),
    StructField('age', IntegerType(), True),
    StructField('city', StringType(), True)
])
data = [('Alice', 29, 'NY'), ('Bob', 35, 'LA')]
df = spark.createDataFrame(data, schema=schema)
df.columns

A 3

B 2

C 4

D 1

💡 Hint

Count the fields defined in the StructType schema.

The schema defines three fields: name, age, and city, so the DataFrame has 3 columns.

🔧 Debug

advanced

Identify the error in schema definition code

What error does the following code raise when creating a DataFrame with this schema?

Apache Spark

from pyspark.sql import SparkSession
from pyspark.sql.types import StructType, StructField, IntegerType
spark = SparkSession.builder.getOrCreate()
schema = StructType([
    StructField('id', IntegerType(), True),
    StructField('value', IntegerType())
])
data = [(1, 100), (2, 200)]
df = spark.createDataFrame(data, schema=schema)

A TypeError: StructField() missing 1 required positional argument: 'nullable'

B ValueError: Data does not match schema

C No error, DataFrame created successfully

D SyntaxError: invalid syntax

💡 Hint

StructField constructor has nullable=True as the default value.

StructField(name, dataType, nullable=True, metadata=None). The nullable argument defaults to True, so the code runs successfully without error.

🧠 Conceptual

advanced

Understanding schema inference behavior with null values

When Spark infers schema from JSON data containing null values in some fields, what is the expected behavior?

A Spark infers the field as non-nullable regardless of null values

B Spark treats null values as strings and infers the field as string type

C Spark raises an error if null values are present during inference

D Spark infers the data type ignoring nulls and marks the field as nullable

💡 Hint

Think about how Spark handles missing or null data during schema inference.

Spark infers the data type from non-null values and marks fields with nulls as nullable to allow missing data.

🚀 Application

expert

Choosing schema definition for performance optimization

You have a large CSV dataset with millions of rows. You want to optimize Spark job performance by defining a schema instead of relying on inference. Which schema definition approach is best?

A Define a StructType schema with exact data types matching the CSV columns and pass it to createDataFrame

B Use schema inference by reading a small sample of the CSV file

C Skip schema definition and let Spark infer schema automatically

D Define all columns as StringType to avoid type mismatch errors

💡 Hint

Explicit schema helps Spark avoid scanning data multiple times.

Defining an exact schema reduces overhead and speeds up reading large datasets by avoiding costly inference.