Reading JSON and nested data in Apache Spark - Step-by-Step Execution

Concept Flow - Reading JSON and nested data

Start: JSON file path

↓

Read JSON with Spark

↓

Create DataFrame

↓

Inspect schema

↓

Access nested fields

↓

Show data or use nested fields

The flow starts by reading a JSON file into a Spark DataFrame, then inspecting and accessing nested fields step-by-step.

Execution Sample

Apache Spark

from pyspark.sql import SparkSession
spark = SparkSession.builder.getOrCreate()
df = spark.read.json('data.json')
df.printSchema()
df = df.select('name', 'address.city')
df.show()

This code reads a JSON file with nested data, prints its schema, and shows selected nested fields.

Execution Table

Step	Action	Input/Condition	Result/Output
1	Create SparkSession	No input	SparkSession object created
2	Read JSON file	File path 'data.json'	DataFrame with nested schema created
3	Print schema	DataFrame	Schema shows fields: name (string), address (struct) with city, state
4	Select fields	Select 'name' and 'address.city'	DataFrame with columns: name, city
5	Show data	Selected columns	Table output with names and city values
6	End	All steps done	Data displayed, execution complete

💡 All JSON data read and nested fields accessed successfully.

Variable Tracker

Variable	Start	After Step 2	After Step 4	Final
spark	None	SparkSession object	SparkSession object	SparkSession object
df	None	DataFrame with nested JSON data	DataFrame with selected columns	DataFrame with selected columns

Key Moments - 3 Insights

How do we know the structure of nested JSON fields?

By using df.printSchema() (see execution_table step 3), we see the nested fields and their types clearly.

How to access a nested field like city inside address?

Use dot notation like 'address.city' in df.select() as shown in step 4 of the execution_table.

What if the nested field does not exist in some rows?

Spark will show null for those rows in the selected nested column, allowing safe access without errors.

Visual Quiz - 3 Questions

Look at the execution_table, what does step 3 output?

A A DataFrame with selected columns

B The schema of the DataFrame showing nested fields

C The JSON file content printed as text

D An error message about missing fields

💡 Hint

Refer to execution_table row with Step 3 describing printSchema output.

At which step is the nested field 'address.city' accessed?

A Step 4

B Step 3

C Step 2

D Step 5

💡 Hint

Check execution_table row Step 4 where select('name', 'address.city') is done.

If the JSON file had no nested fields, how would the schema print differ?

A It would show nested fields as arrays

B It would show an error in printSchema

C It would show only top-level fields without structs

D It would not create a DataFrame

💡 Hint

Think about how printSchema shows nested structs vs flat fields (see step 3).

Concept Snapshot

Read JSON with spark.read.json('file')
Use df.printSchema() to see nested structure
Access nested fields with dot notation: df.select('parent.child')
Show or use nested data safely
Spark handles missing nested fields as null

Full Transcript

This visual trace shows how to read JSON files with nested data in Apache Spark. First, a SparkSession is created. Then, spark.read.json reads the JSON file into a DataFrame. Using df.printSchema(), we inspect the nested structure, seeing fields like 'address' with subfields like 'city'. We select nested fields using dot notation, for example 'address.city', and display the data. The variable tracker shows how the DataFrame changes from raw JSON to selected columns. Key moments clarify how to understand schema and access nested data. The quizzes test understanding of schema output, nested field access, and schema differences for flat JSON. This step-by-step trace helps beginners see exactly how Spark handles nested JSON data.

Practice

1. What does Apache Spark do when reading a JSON file with nested data?

easy

A. It automatically parses nested fields into a structured DataFrame.

B. It ignores nested fields and reads only top-level data.

C. It converts JSON into plain text without structure.

D. It requires manual flattening before reading nested data.

Solution

1. Step 1: Understand Spark JSON reading

   Spark reads JSON files and automatically detects nested structures.

2. Step 2: Effect on DataFrame

   Nested JSON fields become nested columns accessible via dot notation.

3. Final Answer:

   It automatically parses nested fields into a structured DataFrame. -> Option A

4. Quick Check:

   Spark JSON reading = automatic nested parsing [OK]

Hint: Spark reads nested JSON as nested columns automatically [OK]

Common Mistakes:

• Thinking Spark ignores nested fields
• Assuming JSON is read as plain text
• Believing manual flattening is always needed

2. Which of the following is the correct syntax to read a JSON file named data.json into a Spark DataFrame?

easy

A. spark.read.text("data.json")

B. spark.read.csv("data.json")

C. spark.load.json("data.json")

D. spark.read.json("data.json")

Solution

1. Step 1: Identify correct Spark method for JSON

   The method to read JSON files is spark.read.json().

2. Step 2: Check other options

   csv reads CSV files, load is generic but needs format specified, text reads plain text lines.

3. Final Answer:

   spark.read.json("data.json") -> Option D

4. Quick Check:

   Read JSON file = spark.read.json() [OK]

Hint: Use spark.read.json() to load JSON files [OK]

Common Mistakes:

• Using spark.read.csv() for JSON files
• Using spark.load.json() without format
• Using spark.read.text() for structured JSON

3. Given the following JSON data loaded into a DataFrame df:

[{"name": "Alice", "info": {"age": 30, "city": "NY"}}, {"name": "Bob", "info": {"age": 25, "city": "LA"}}]

What will be the output of df.select("info.age").collect()?

medium

A. [Row(age="NY"), Row(age="LA")]

B. [Row(info=30), Row(info=25)]

C. [Row(age=30), Row(age=25)]

D. [30, 25]

Solution

1. Step 1: Understand nested field selection

   Using select("info.age") selects the nested field age inside info.

2. Step 2: Result of collect()

   collect() returns a list of Row objects with the selected field values.

3. Final Answer:

   [Row(age=30), Row(age=25)] -> Option C

4. Quick Check:

   Select nested field = list of Rows with that field [OK]

Hint: Selecting nested fields returns Rows with field names [OK]

Common Mistakes:

• Expecting plain values instead of Row objects
• Confusing nested field name with top-level
• Selecting wrong nested key

4. You try to select a nested field using df.select("info-age") but get an error. What is the likely cause?

medium

A. Using a dash (-) instead of dot (.) to access nested fields.

B. The JSON file is empty.

C. The DataFrame was not loaded with spark.read.json().

D. Nested fields cannot be selected in Spark.

Solution

1. Step 1: Check syntax for nested field access

   Spark uses dot notation like info.age, not dash.

2. Step 2: Understand error cause

   Using info-age is invalid syntax causing an error.

3. Final Answer:

   Using a dash (-) instead of dot (.) to access nested fields. -> Option A

4. Quick Check:

   Nested field access = dot notation [OK]

Hint: Use dot, not dash, to access nested JSON fields [OK]

Common Mistakes:

• Using dash instead of dot for nested fields
• Assuming nested fields can't be selected
• Ignoring error message details

5. You have a JSON file with nested arrays like this:

[{"id": 1, "scores": [10, 20, 30]}, {"id": 2, "scores": [15, 25]}]

How can you transform this DataFrame to have one row per score with columns id and score?

hard

A. Use df.select("id", "scores") without explode.

B. Use df.select("id", explode("scores").alias("score")) to flatten the array.

C. Use df.groupBy("id").count() to count scores.

D. Use df.withColumn("score", array("scores")) to copy scores.

Solution

1. Step 1: Understand array flattening

   The explode() function creates a new row for each element in the array.

2. Step 2: Apply explode to nested array

   Using df.select("id", explode("scores").alias("score")) flattens scores into separate rows.

3. Final Answer:

   Use df.select("id", explode("scores").alias("score")) to flatten the array. -> Option B

4. Quick Check:

   Flatten nested arrays = explode() [OK]

Hint: Use explode() to flatten nested arrays into rows [OK]

Common Mistakes:

• Not using explode and keeping arrays nested
• Using groupBy() instead of explode()
• Trying to copy arrays without flattening