Reading JSON and nested data in Apache Spark - Mini Project: Build & Apply

Reading JSON and Nested Data

📖 Scenario: You work as a data analyst for a company that collects customer information in JSON format. The data includes nested details like addresses and orders. You need to read this JSON data using Apache Spark and extract useful information.

🎯 Goal: Learn how to read JSON data with nested structures in Apache Spark and extract specific fields into a DataFrame.

📋 What You'll Learn

Read JSON data from a string using Spark

Access nested fields inside the JSON

Create a DataFrame with selected columns

Display the extracted data

💡 Why This Matters

🌍 Real World

Companies often receive data in JSON format with nested details. Being able to read and extract this data using Spark helps analyze customer information, logs, or events efficiently.

💼 Career

Data engineers and data scientists frequently work with JSON data in big data platforms like Spark. This skill is essential for preparing data for analysis or machine learning.

1

Create JSON data string

Create a variable called json_data that contains this exact JSON string: [{"name": "Alice", "age": 30, "address": {"city": "New York", "zip": "10001"}}, {"name": "Bob", "age": 25, "address": {"city": "Los Angeles", "zip": "90001"}}]

Apache Spark

# Create a variable called json_data with the given JSON string
# Your code here

Hint

Use single quotes around the whole string and double quotes inside for JSON format.

2

Create Spark session

Create a Spark session variable called spark using SparkSession.builder.appName("JSONReader").getOrCreate()

Apache Spark

json_data = '[{"name": "Alice", "age": 30, "address": {"city": "New York", "zip": "10001"}}, {"name": "Bob", "age": 25, "address": {"city": "Los Angeles", "zip": "90001"}}]'
# Create a Spark session called spark
# Your code here

Hint

Import SparkSession from pyspark.sql before creating the session.

3

Read JSON string into DataFrame

Use spark.read.json() with spark.sparkContext.parallelize([json_data]) to create a DataFrame called df from the JSON string

Apache Spark

from pyspark.sql import SparkSession

json_data = '[{"name": "Alice", "age": 30, "address": {"city": "New York", "zip": "10001"}}, {"name": "Bob", "age": 25, "address": {"city": "Los Angeles", "zip": "90001"}}]'
spark = SparkSession.builder.appName("JSONReader").getOrCreate()
# Create a DataFrame called df by reading the JSON string
# Your code here

Hint

Use parallelize to convert the JSON string into an RDD before reading.

4

Select and show nested data

Select the columns name, age, and the nested field address.city from df into a new DataFrame called result. Then print the contents of result using result.show()

Apache Spark

from pyspark.sql import SparkSession

json_data = '[{"name": "Alice", "age": 30, "address": {"city": "New York", "zip": "10001"}}, {"name": "Bob", "age": 25, "address": {"city": "Los Angeles", "zip": "90001"}}]'
spark = SparkSession.builder.appName("JSONReader").getOrCreate()
df = spark.read.json(spark.sparkContext.parallelize([json_data]))
# Select name, age, and address.city into result and show it
# Your code here

Hint

Use df.select("name", "age", "address.city") to access nested fields.

Practice

1. What does Apache Spark do when reading a JSON file with nested data?

easy

A. It automatically parses nested fields into a structured DataFrame.

B. It ignores nested fields and reads only top-level data.

C. It converts JSON into plain text without structure.

D. It requires manual flattening before reading nested data.

Solution

1. Step 1: Understand Spark JSON reading

   Spark reads JSON files and automatically detects nested structures.

2. Step 2: Effect on DataFrame

   Nested JSON fields become nested columns accessible via dot notation.

3. Final Answer:

   It automatically parses nested fields into a structured DataFrame. -> Option A

4. Quick Check:

   Spark JSON reading = automatic nested parsing [OK]

Hint: Spark reads nested JSON as nested columns automatically [OK]

Common Mistakes:

• Thinking Spark ignores nested fields
• Assuming JSON is read as plain text
• Believing manual flattening is always needed

2. Which of the following is the correct syntax to read a JSON file named data.json into a Spark DataFrame?

easy

A. spark.read.text("data.json")

B. spark.read.csv("data.json")

C. spark.load.json("data.json")

D. spark.read.json("data.json")

Solution

1. Step 1: Identify correct Spark method for JSON

   The method to read JSON files is spark.read.json().

2. Step 2: Check other options

   csv reads CSV files, load is generic but needs format specified, text reads plain text lines.

3. Final Answer:

   spark.read.json("data.json") -> Option D

4. Quick Check:

   Read JSON file = spark.read.json() [OK]

Hint: Use spark.read.json() to load JSON files [OK]

Common Mistakes:

• Using spark.read.csv() for JSON files
• Using spark.load.json() without format
• Using spark.read.text() for structured JSON

3. Given the following JSON data loaded into a DataFrame df:

[{"name": "Alice", "info": {"age": 30, "city": "NY"}}, {"name": "Bob", "info": {"age": 25, "city": "LA"}}]

What will be the output of df.select("info.age").collect()?

medium

A. [Row(age="NY"), Row(age="LA")]

B. [Row(info=30), Row(info=25)]

C. [Row(age=30), Row(age=25)]

D. [30, 25]

Solution

1. Step 1: Understand nested field selection

   Using select("info.age") selects the nested field age inside info.

2. Step 2: Result of collect()

   collect() returns a list of Row objects with the selected field values.

3. Final Answer:

   [Row(age=30), Row(age=25)] -> Option C

4. Quick Check:

   Select nested field = list of Rows with that field [OK]

Hint: Selecting nested fields returns Rows with field names [OK]

Common Mistakes:

• Expecting plain values instead of Row objects
• Confusing nested field name with top-level
• Selecting wrong nested key

4. You try to select a nested field using df.select("info-age") but get an error. What is the likely cause?

medium

A. Using a dash (-) instead of dot (.) to access nested fields.

B. The JSON file is empty.

C. The DataFrame was not loaded with spark.read.json().

D. Nested fields cannot be selected in Spark.

Solution

1. Step 1: Check syntax for nested field access

   Spark uses dot notation like info.age, not dash.

2. Step 2: Understand error cause

   Using info-age is invalid syntax causing an error.

3. Final Answer:

   Using a dash (-) instead of dot (.) to access nested fields. -> Option A

4. Quick Check:

   Nested field access = dot notation [OK]

Hint: Use dot, not dash, to access nested JSON fields [OK]

Common Mistakes:

• Using dash instead of dot for nested fields
• Assuming nested fields can't be selected
• Ignoring error message details

5. You have a JSON file with nested arrays like this:

[{"id": 1, "scores": [10, 20, 30]}, {"id": 2, "scores": [15, 25]}]

How can you transform this DataFrame to have one row per score with columns id and score?

hard

A. Use df.select("id", "scores") without explode.

B. Use df.select("id", explode("scores").alias("score")) to flatten the array.

C. Use df.groupBy("id").count() to count scores.

D. Use df.withColumn("score", array("scores")) to copy scores.

Solution

1. Step 1: Understand array flattening

   The explode() function creates a new row for each element in the array.

2. Step 2: Apply explode to nested array

   Using df.select("id", explode("scores").alias("score")) flattens scores into separate rows.

3. Final Answer:

   Use df.select("id", explode("scores").alias("score")) to flatten the array. -> Option B

4. Quick Check:

   Flatten nested arrays = explode() [OK]

Hint: Use explode() to flatten nested arrays into rows [OK]

Common Mistakes:

• Not using explode and keeping arrays nested
• Using groupBy() instead of explode()
• Trying to copy arrays without flattening