Reading JSON and nested data in Apache Spark - Cheat Sheet & Quick Revision

Recall & Review

beginner

What is the main function to read a JSON file in Apache Spark?

You use spark.read.json(path) to load a JSON file into a DataFrame.

beginner

How do you access nested fields in a JSON column in Spark?

You use dot notation like df.select("nestedField.subField") to access nested data.

intermediate

What Spark function helps to flatten nested JSON structures?

You can use explode() to turn nested arrays into separate rows, helping flatten the data.

beginner

How can you infer the schema automatically when reading JSON in Spark?

By default, spark.read.json() infers the schema from the JSON data automatically.

intermediate

Why is it useful to understand the schema of nested JSON data before processing?

Knowing the schema helps you select, filter, and transform nested fields correctly without errors.

Which Spark method reads a JSON file into a DataFrame?

A spark.read.json()

B spark.load.csv()

C spark.read.text()

D spark.load.json()

The correct method to read JSON files is spark.read.json().

How do you select a nested field named 'address.city' from a DataFrame?

A df.select('address_city')

B df.select('address-city')

C df.select('city')

D df.select('address.city')

Use dot notation like df.select('address.city') to access nested fields.

What does the explode() function do in Spark?

A Joins two DataFrames

B Flattens nested arrays into multiple rows

C Filters rows based on condition

D Aggregates data by key

explode() turns each element in an array into a separate row.

If a JSON file has nested objects, how does Spark handle the schema by default?

A Infers nested schema automatically

B Reads all data as strings

C Fails to read nested data

D Requires manual schema definition

Spark automatically infers nested schemas when reading JSON files.

Why is schema knowledge important when working with nested JSON?

A To convert JSON to CSV

B To speed up file reading

C To correctly select and transform nested fields

D To delete nested data

Understanding schema helps avoid errors when accessing nested fields.

Explain how to read a nested JSON file in Apache Spark and access a nested field.

Think about how JSON structure maps to DataFrame columns.

Describe the purpose and use of the explode() function when working with nested JSON data in Spark.

Imagine turning a list inside a cell into multiple rows.