Schema definition and inference in Apache Spark - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

Output of schema inference with mixed data types

What is the output schema of the following Spark DataFrame code snippet?

Apache Spark

from pyspark.sql import SparkSession
from pyspark.sql.types import *
spark = SparkSession.builder.getOrCreate()
data = [(1, 'Alice', 29), (2, 'Bob', 'thirty'), (3, 'Cathy', 25)]
df = spark.createDataFrame(data, ['id', 'name', 'age'])
df.printSchema()

A

root
 |-- id: integer (nullable = true)
 |-- name: string (nullable = true)
 |-- age: integer (nullable = true)

B

root
 |-- id: long (nullable = true)
 |-- name: string (nullable = true)
 |-- age: string (nullable = true)

C

root
 |-- id: long (nullable = true)
 |-- name: string (nullable = true)
 |-- age: long (nullable = true)

D

root
 |-- id: string (nullable = true)
 |-- name: string (nullable = true)
 |-- age: string (nullable = true)

💡 Hint

Spark infers the column type based on all values. Mixed types in a column cause Spark to choose a common type that can hold all values.

Since the 'age' column has both integers and a string ('thirty'), Spark infers the entire column as string type to accommodate all values.

❓ data_output

intermediate

Number of columns after explicit schema definition

Given the explicit schema below, how many columns will the resulting DataFrame have?

Apache Spark

from pyspark.sql import SparkSession
from pyspark.sql.types import StructType, StructField, StringType, IntegerType
spark = SparkSession.builder.getOrCreate()
schema = StructType([
    StructField('name', StringType(), True),
    StructField('age', IntegerType(), True),
    StructField('city', StringType(), True)
])
data = [('Alice', 29, 'NY'), ('Bob', 35, 'LA')]
df = spark.createDataFrame(data, schema=schema)
df.columns

A 3

B 2

C 4

D 1

💡 Hint

Count the fields defined in the StructType schema.

The schema defines three fields: name, age, and city, so the DataFrame has 3 columns.

🔧 Debug

advanced

Identify the error in schema definition code

What error does the following code raise when creating a DataFrame with this schema?

Apache Spark

from pyspark.sql import SparkSession
from pyspark.sql.types import StructType, StructField, IntegerType
spark = SparkSession.builder.getOrCreate()
schema = StructType([
    StructField('id', IntegerType(), True),
    StructField('value', IntegerType())
])
data = [(1, 100), (2, 200)]
df = spark.createDataFrame(data, schema=schema)

A TypeError: StructField() missing 1 required positional argument: 'nullable'

B ValueError: Data does not match schema

C No error, DataFrame created successfully

D SyntaxError: invalid syntax

💡 Hint

StructField constructor has nullable=True as the default value.

StructField(name, dataType, nullable=True, metadata=None). The nullable argument defaults to True, so the code runs successfully without error.

🧠 Conceptual

advanced

Understanding schema inference behavior with null values

When Spark infers schema from JSON data containing null values in some fields, what is the expected behavior?

A Spark infers the field as non-nullable regardless of null values

B Spark treats null values as strings and infers the field as string type

C Spark raises an error if null values are present during inference

D Spark infers the data type ignoring nulls and marks the field as nullable

💡 Hint

Think about how Spark handles missing or null data during schema inference.

Spark infers the data type from non-null values and marks fields with nulls as nullable to allow missing data.

🚀 Application

expert

Choosing schema definition for performance optimization

You have a large CSV dataset with millions of rows. You want to optimize Spark job performance by defining a schema instead of relying on inference. Which schema definition approach is best?

A Define a StructType schema with exact data types matching the CSV columns and pass it to createDataFrame

B Use schema inference by reading a small sample of the CSV file

C Skip schema definition and let Spark infer schema automatically

D Define all columns as StringType to avoid type mismatch errors

💡 Hint

Explicit schema helps Spark avoid scanning data multiple times.

Defining an exact schema reduces overhead and speeds up reading large datasets by avoiding costly inference.

Practice

1. What is the main purpose of defining a schema in Apache Spark?

easy

A. To speed up the Spark installation process

B. To store data permanently on disk

C. To create visualizations automatically

D. To tell Spark the structure and data types of the data

Solution

1. Step 1: Understand what a schema represents

   A schema defines the structure and data types of columns in a dataset.

2. Step 2: Identify the role of schema in Spark

   It helps Spark know how to read and process the data safely and clearly.

3. Final Answer:

   To tell Spark the structure and data types of the data -> Option D

4. Quick Check:

   Schemas = Data structure info [OK]

Hint: Schemas describe data layout and types [OK]

Common Mistakes:

• Thinking schema speeds up installation
• Confusing schema with visualization
• Believing schema stores data permanently

2. Which of the following is the correct way to define a schema manually in Spark using StructType?

easy

A. schema = StructType([StructField('name', StringType(), True), StructField('age', IntegerType(), True)])

B. schema = Struct('name' StringType, 'age' IntegerType)

C. schema = Schema('name': 'string', 'age': 'int')

D. schema = StructType('name', 'age')

Solution

1. Step 1: Recall the syntax for manual schema definition

   In Spark, schemas are defined using StructType with a list of StructField objects.

2. Step 2: Match the correct syntax

   schema = StructType([StructField('name', StringType(), True), StructField('age', IntegerType(), True)]) correctly uses StructType and StructField with data types and nullable flags.

3. Final Answer:

   schema = StructType([StructField('name', StringType(), True), StructField('age', IntegerType(), True)]) -> Option A

4. Quick Check:

   StructType + StructField list = correct schema [OK]

Hint: Use StructType with list of StructField for schemas [OK]

Common Mistakes:

• Omitting parentheses in StructField
• Using wrong class names like Schema or Struct
• Not passing nullable argument

3. Given the following code, what will be the schema of the DataFrame?

from pyspark.sql import SparkSession
from pyspark.sql.types import *
spark = SparkSession.builder.getOrCreate()
data = [('Alice', 30), ('Bob', 25)]
schema = StructType([StructField('name', StringType(), True), StructField('age', IntegerType(), True)])
df = spark.createDataFrame(data, schema)
df.printSchema()

medium

A. root
|-- name: string (nullable = false)
|-- age: integer (nullable = false)

B. root
|-- name: integer (nullable = true)
|-- age: string (nullable = true)

C. root
|-- name: string (nullable = true)
|-- age: integer (nullable = true)

D. root
|-- name: string (nullable = true)
|-- age: string (nullable = true)

Solution

1. Step 1: Analyze the schema definition

   The schema defines 'name' as StringType and 'age' as IntegerType, both nullable True.

2. Step 2: Understand printSchema output

   printSchema shows the root with fields and their types and nullable status.

3. Final Answer:

   root |-- name: string (nullable = true) |-- age: integer (nullable = true) -> Option C

4. Quick Check:

   Schema matches defined types and nullable flags [OK]

Hint: Check schema field types and nullable flags carefully [OK]

Common Mistakes:

• Mixing up data types for fields
• Assuming nullable is False by default
• Confusing printSchema format

4. What is wrong with this schema definition code?

from pyspark.sql.types import StructType, StructField, StringType, IntegerType
schema = StructType([
  StructField('name', StringType),
  StructField('age', IntegerType, True)
])

medium

A. Nullable argument must be False

B. Missing parentheses after StringType and IntegerType

C. StructType should not use a list

D. StructField names should be integers

Solution

1. Step 1: Check usage of data types in StructField

   Data types like StringType and IntegerType must be called with parentheses to create instances.

2. Step 2: Identify the error in the code

   Here, StringType and IntegerType are used without parentheses, causing an error.

3. Final Answer:

   Missing parentheses after StringType and IntegerType -> Option B

4. Quick Check:

   Data types need () to instantiate [OK]

Hint: Always add () after data type classes [OK]

Common Mistakes:

• Forgetting parentheses on data types
• Using wrong types for field names
• Misunderstanding nullable argument

5. You have a JSON file with mixed data types and some missing fields. You want to load it into Spark with a schema that allows missing values and infers types automatically. Which approach is best?

hard

A. Load the JSON without schema and let Spark infer schema automatically

B. Define a manual schema with all fields nullable, then load with that schema

C. Define a schema with all fields non-nullable and load the JSON

D. Convert JSON to CSV first, then load without schema

Solution

1. Step 1: Understand schema inference in Spark

   Spark can automatically detect data types and handle missing fields when loading JSON without a manual schema.

2. Step 2: Compare options for handling missing and mixed types

   Letting Spark infer schema is simpler and handles missing values well; manual schema requires careful nullable settings.

3. Final Answer:

   Load the JSON without schema and let Spark infer schema automatically -> Option A

4. Quick Check:

   Schema inference handles mixed and missing data best [OK]

Hint: Use schema inference for mixed and missing JSON data [OK]

Common Mistakes:

• Making all fields non-nullable causing errors
• Manually defining schema without nullable fields
• Converting JSON unnecessarily before loading