Creating DataFrames from files (CSV, JSON, Parquet) in Apache Spark - Practice Exercises

Challenge - 5 Problems

❓ Predict Output

intermediate

Output of reading CSV with header option

What will be the output schema of the DataFrame after running this code snippet in Apache Spark?

Apache Spark

from pyspark.sql import SparkSession
spark = SparkSession.builder.appName('Test').getOrCreate()
df = spark.read.csv('data.csv', header=True, inferSchema=True)
df.printSchema()

A

root
 |-- column1: string (nullable = true)
 |-- column2: integer (nullable = true)

B

root
 |-- _c0: integer (nullable = true)
 |-- _c1: integer (nullable = true)

C

root
 |-- _c0: string (nullable = true)
 |-- _c1: string (nullable = true)

D

root
 |-- column1: integer (nullable = true)
 |-- column2: string (nullable = true)

💡 Hint

Check the effect of header=True and inferSchema=True when reading CSV files.

When header=True, Spark uses the first row as column names. inferSchema=True lets Spark detect data types automatically. So columns get proper names and types.

❓ data_output

intermediate

DataFrame content after reading JSON file

Given a JSON file with records [{"name":"Alice","age":30},{"name":"Bob","age":25}], what will be the output of df.show() after reading it with spark.read.json('people.json')?

Apache Spark

from pyspark.sql import SparkSession
spark = SparkSession.builder.appName('Test').getOrCreate()
df = spark.read.json('people.json')
df.show()

A

+-----+---+
| name|age|
+-----+---+
|  30 |Alice|
|  25 | Bob|
+-----+---+

B

+-----+---+
| name|age|
+-----+---+
|Alice| 30|
|  Bob| 25|
+-----+---+

C

+-----+---+
| name|age|
+-----+---+
|Alice|"30"|
|  Bob|"25"|
+-----+---+

D

+-----+---+
| name|age|
+-----+---+
|Alice|null|
|  Bob|null|
+-----+---+

💡 Hint

Spark reads JSON and infers schema and types automatically.

Spark reads JSON records and infers the schema with correct types. Numbers remain integers, strings remain strings.

❓ visualization

advanced

Visualizing Parquet file schema

You load a Parquet file with spark.read.parquet('data.parquet'). Which Spark method will show a tree-like schema visualization of the DataFrame?

Apache Spark

from pyspark.sql import SparkSession
spark = SparkSession.builder.appName('Test').getOrCreate()
df = spark.read.parquet('data.parquet')
# Which method to call next?

A df.schemaTree()

B df.showSchema()

C df.printSchema()

D df.displaySchema()

💡 Hint

The method name is printSchema, not showSchema or others.

printSchema() prints the schema in a tree format. Other options do not exist or cause errors.

🔧 Debug

advanced

Error when reading CSV without header option

What error or issue will occur if you run this code and the CSV file has a header row but you do NOT specify header=True? from pyspark.sql import SparkSession spark = SparkSession.builder.appName('Test').getOrCreate() df = spark.read.csv('data.csv') df.show()

Apache Spark

from pyspark.sql import SparkSession
spark = SparkSession.builder.appName('Test').getOrCreate()
df = spark.read.csv('data.csv')
df.show()

A The DataFrame will be empty because header is not read.

B Spark will raise a FileNotFoundError because header is missing.

C Spark will raise a ValueError about missing header option.

D The first row will be treated as data, not header, so column names will be _c0, _c1, etc.

💡 Hint

By default, header=False, so first row is data.

Without header=True, Spark treats the first row as data and assigns default column names like _c0, _c1.

🚀 Application

expert

Choosing file format for large dataset with nested data

You have a large dataset with nested JSON structures and want to store it efficiently for fast queries in Spark. Which file format should you choose?

A Parquet, because it supports nested data and is columnar for fast queries

B CSV, because it is simple and widely supported

C JSON, because it preserves nested structures naturally

D TXT, because it is easy to read and write

💡 Hint

Think about file format efficiency and support for nested data.

Parquet is a columnar storage format that supports nested data and compression, making it efficient for large datasets and fast queries.

Practice

1. What is the main purpose of creating a DataFrame from a file in Apache Spark?

easy

A. To write data back to a file

B. To organize data in rows and columns for easy analysis

C. To delete data from the file system

D. To convert data into plain text format

Solution

1. Step 1: Understand DataFrame concept

   A DataFrame is a table-like structure with rows and columns used for data analysis.

2. Step 2: Purpose of loading files

   Loading files into DataFrames allows Spark to process and analyze data efficiently.

3. Final Answer:

   To organize data in rows and columns for easy analysis -> Option B

4. Quick Check:

   DataFrames = organized data [OK]

Hint: DataFrames = tables for easy data work [OK]

Common Mistakes:

• Confusing DataFrame creation with file writing
• Thinking DataFrames delete files
• Assuming DataFrames convert data to plain text

2. Which of the following is the correct syntax to load a CSV file into a Spark DataFrame?

easy

A. spark.read.format('csv').load('file.csv')

B. spark.read.load('file.csv', format='csv')

C. spark.load('csv', 'file.csv')

D. spark.read.csv('file.csv')

Solution

1. Step 1: Recall Spark CSV loading syntax

   The standard way is to use spark.read.format('csv').load('filename').

2. Step 2: Check each option

   spark.read.format('csv').load('file.csv') matches the correct syntax; others are incorrect or invalid methods.

3. Final Answer:

   spark.read.format('csv').load('file.csv') -> Option A

4. Quick Check:

   Use format('csv').load() for CSV [OK]

Hint: Use spark.read.format('csv').load() for CSV files [OK]

Common Mistakes:

• Using spark.read.csv() without options
• Trying spark.load() which doesn't exist
• Passing format as a keyword argument in load()

3. Given the following code, what will be the schema of the DataFrame?

df = spark.read.format('json').load('people.json')
df.printSchema()

Assuming 'people.json' contains records like {"name": "Alice", "age": 30}.

medium

A. root
|-- name: string (nullable = true)
|-- age: long (nullable = true)

B. root
|-- name: integer (nullable = true)
|-- age: string (nullable = true)

C. root
|-- name: string (nullable = false)
|-- age: integer (nullable = false)

D. root
|-- name: string (nullable = true)
|-- age: string (nullable = true)

Solution

1. Step 1: Understand JSON schema inference

   Spark infers types from JSON: "name" is string, "age" is number (long).

2. Step 2: Check nullability and types

   By default, fields are nullable. Age inferred as long (integer type in Spark).

3. Final Answer:

   root |-- name: string (nullable = true) |-- age: long (nullable = true) -> Option A

4. Quick Check:

   JSON fields inferred correctly [OK]

Hint: JSON fields infer types automatically [OK]

Common Mistakes:

• Assuming age is string instead of long
• Thinking fields are non-nullable by default
• Confusing integer with string types

4. What is wrong with this code snippet to load a Parquet file?

df = spark.read.format('parquet').load('data.parquet')
df.show()

Assuming 'data.parquet' exists and is valid.

medium

A. You must specify schema before loading Parquet files

B. The format should be 'parq' instead of 'parquet'

C. The load method requires a file extension in the path

D. Nothing is wrong; this code will load and show the DataFrame correctly

Solution

1. Step 1: Check Parquet loading syntax

   Using spark.read.format('parquet').load('file') is correct and standard.

2. Step 2: Validate assumptions

   If the file exists and is valid, df.show() will display the data without error.

3. Final Answer:

   Nothing is wrong; this code will load and show the DataFrame correctly -> Option D

4. Quick Check:

   Correct Parquet load syntax [OK]

Hint: Parquet loads with format('parquet').load() [OK]

Common Mistakes:

• Using wrong format string like 'parq'
• Thinking schema is mandatory for Parquet
• Believing file extension is required in load()

5. You want to load a CSV file with a header and infer the schema automatically. Which code snippet correctly achieves this?

hard

A. spark.read.format('csv').load('data.csv').option('header', 'true').option('inferSchema', 'true')

B. spark.read.csv('data.csv', header=True, inferSchema=True)

C. spark.read.format('csv').option('header', 'true').option('inferSchema', 'true').load('data.csv')

D. spark.read.load('data.csv').option('header', 'true').option('inferSchema', 'true')

Solution

1. Step 1: Understand CSV options usage

   Options like header and inferSchema must be set before load() when using format().

2. Step 2: Analyze each option

   spark.read.format('csv').option('header', 'true').option('inferSchema', 'true').load('data.csv') sets options correctly before load(). spark.read.csv('data.csv', header=True, inferSchema=True) uses invalid syntax for Spark. spark.read.format('csv').load('data.csv').option('header', 'true').option('inferSchema', 'true') sets options after load(), which is wrong. spark.read.load('data.csv').option('header', 'true').option('inferSchema', 'true') misses format('csv').

3. Final Answer:

   spark.read.format('csv').option('header', 'true').option('inferSchema', 'true').load('data.csv') -> Option C

4. Quick Check:

   Set options before load() [OK]

Hint: Set CSV options before load() with format('csv') [OK]

Common Mistakes:

• Setting options after load() call
• Using Python-style keyword arguments in Spark
• Not specifying format('csv') before load()