Reading JSON and nested data in Apache Spark - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

Output of reading nested JSON with Spark

What is the output of the following Spark code when reading a nested JSON file and selecting a nested field?

Apache Spark

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName('Test').getOrCreate()

json_data = '''[
  {"id": 1, "info": {"name": "Alice", "age": 30}},
  {"id": 2, "info": {"name": "Bob", "age": 25}}
]'''

rdd = spark.sparkContext.parallelize([json_data])
df = spark.read.json(rdd)
df.select('info.name').show()

A AnalysisException: cannot resolve 'info.name' given input columns: [id, info]

B

+----------+
|info.name |
+----------+
|Alice     |
|Bob       |
+----------+

C

+----------+
|info.name |
+----------+
|null      |
|null      |
+----------+

D

+----------+
|name      |
+----------+
|Alice     |
|Bob       |
+----------+

💡 Hint

Remember how Spark handles nested fields in select statements.

In Spark, to select a nested field, you must use the dot notation without quotes or use col() function. Using 'info.name' as a string in select causes an AnalysisException.

❓ data_output

intermediate

Count of nested JSON records after flattening

Given a DataFrame loaded from nested JSON, what is the count of rows after flattening the nested array field?

Apache Spark

from pyspark.sql import SparkSession
from pyspark.sql.functions import explode

spark = SparkSession.builder.appName('Test').getOrCreate()

json_data = '''[
  {"id": 1, "items": [{"name": "apple"}, {"name": "banana"}]},
  {"id": 2, "items": [{"name": "orange"}]}
]'''
rdd = spark.sparkContext.parallelize([json_data])
df = spark.read.json(rdd)

flat_df = df.select('id', explode('items').alias('item'))
count = flat_df.count()

A 3

B 1

C 2

D 0

💡 Hint

Count how many total items are in all nested arrays combined.

The first record has 2 items, the second has 1 item, so total rows after explode are 3.

🔧 Debug

advanced

Identify the error when reading malformed nested JSON

What error will this Spark code produce when reading a malformed nested JSON string?

Apache Spark

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName('Test').getOrCreate()

json_data = '''[
  {"id": 1, "info": {"name": "Alice", "age": 30},
  {"id": 2, "info": {"name": "Bob", "age": 25}}
]'''

rdd = spark.sparkContext.parallelize([json_data])
df = spark.read.json(rdd)
df.show()

A ValueError: Expecting ',' delimiter

B AnalysisException: Malformed records are detected in record parsing

C No error, DataFrame shows both records correctly

D TypeError: 'str' object is not iterable

💡 Hint

Check the JSON syntax carefully for missing commas or brackets.

The JSON string is missing a comma between the two objects, causing Spark to detect malformed records and raise AnalysisException.

🚀 Application

advanced

Extract nested fields and create new columns

Which code snippet correctly extracts the nested 'age' field from 'info' and creates a new column 'age' in the DataFrame?

Apache Spark

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName('Test').getOrCreate()

json_data = '''[
  {"id": 1, "info": {"name": "Alice", "age": 30}},
  {"id": 2, "info": {"name": "Bob", "age": 25}}
]'''
rdd = spark.sparkContext.parallelize([json_data])
df = spark.read.json(rdd)

A df.withColumn('age', df['info.age']).show()

B df.withColumn('age', df.info['age']).show()

C df.withColumn('age', df['info']['age']).show()

D df.withColumn('age', df.info.age).show()

💡 Hint

Use dot notation to access nested fields in Spark DataFrame columns.

In Spark, nested fields can be accessed using dot notation like df.info.age. Other options cause errors or do not work.

🧠 Conceptual

expert

Understanding schema inference for nested JSON in Spark

When Spark reads a nested JSON file without a predefined schema, what is the behavior of schema inference for nested fields?

A Spark requires a user-defined schema to read nested JSON; otherwise, it raises an error.

B Spark infers only the top-level fields and treats nested fields as strings.

C Spark infers the schema recursively, including all nested fields and their types automatically.

D Spark infers the schema but ignores nested arrays and maps, flattening them into strings.

💡 Hint

Think about how Spark handles complex JSON structures by default.

Spark's JSON reader automatically infers nested schemas recursively, detecting nested structs, arrays, and their types.