Creating DataFrames from files (CSV, JSON, Parquet) in Apache Spark - Try It Yourself

Creating DataFrames from files (CSV, JSON, Parquet)

📖 Scenario: You work as a data analyst at a retail company. You receive sales data in different file formats: CSV, JSON, and Parquet. Your task is to load these files into Spark DataFrames to analyze the sales.

🎯 Goal: Learn how to create Spark DataFrames by reading data from CSV, JSON, and Parquet files.

📋 What You'll Learn

Use SparkSession to read files

Read a CSV file into a DataFrame

Read a JSON file into a DataFrame

Read a Parquet file into a DataFrame

Print the schema of each DataFrame

💡 Why This Matters

🌍 Real World

Data scientists and analysts often receive data in different file formats. Knowing how to load these files into Spark DataFrames is essential for data processing and analysis.

💼 Career

This skill is important for roles like Data Engineer, Data Scientist, and Big Data Analyst who work with large datasets stored in various formats.

1

Create a SparkSession

Create a SparkSession called spark with the app name "FileReaderApp".

Apache Spark

# Create a SparkSession called spark with app name "FileReaderApp"
# Your code here

Hint

Use SparkSession.builder.appName(...).getOrCreate() to create the SparkSession.

2

Read the CSV file into a DataFrame

Use the spark session to read the CSV file named "sales.csv" into a DataFrame called csv_df. Assume the CSV file has a header row.

Apache Spark

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("FileReaderApp").getOrCreate()

# Read the CSV file "sales.csv" with header into csv_df
# Your code here

Hint

Use spark.read.option("header", True).csv("sales.csv") to read the CSV file with headers.

3

Read the JSON and Parquet files into DataFrames

Use the spark session to read the JSON file named "sales.json" into a DataFrame called json_df, and the Parquet file named "sales.parquet" into a DataFrame called parquet_df.

Apache Spark

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("FileReaderApp").getOrCreate()
csv_df = spark.read.option("header", True).csv("sales.csv")

# Read the JSON file "sales.json" into json_df
# Read the Parquet file "sales.parquet" into parquet_df
# Your code here

Hint

Use spark.read.json(...) and spark.read.parquet(...) to read JSON and Parquet files respectively.

4

Print the schema of each DataFrame

Print the schema of csv_df, json_df, and parquet_df using the .printSchema() method.

Apache Spark

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("FileReaderApp").getOrCreate()
csv_df = spark.read.option("header", True).csv("sales.csv")
json_df = spark.read.json("sales.json")
parquet_df = spark.read.parquet("sales.parquet")

# Print the schema of csv_df
# Print the schema of json_df
# Print the schema of parquet_df
# Your code here

Hint

Use printSchema() on each DataFrame to see its structure.

Practice

1. What is the main purpose of creating a DataFrame from a file in Apache Spark?

easy

A. To write data back to a file

B. To organize data in rows and columns for easy analysis

C. To delete data from the file system

D. To convert data into plain text format

Solution

1. Step 1: Understand DataFrame concept

   A DataFrame is a table-like structure with rows and columns used for data analysis.

2. Step 2: Purpose of loading files

   Loading files into DataFrames allows Spark to process and analyze data efficiently.

3. Final Answer:

   To organize data in rows and columns for easy analysis -> Option B

4. Quick Check:

   DataFrames = organized data [OK]

Hint: DataFrames = tables for easy data work [OK]

Common Mistakes:

• Confusing DataFrame creation with file writing
• Thinking DataFrames delete files
• Assuming DataFrames convert data to plain text

2. Which of the following is the correct syntax to load a CSV file into a Spark DataFrame?

easy

A. spark.read.format('csv').load('file.csv')

B. spark.read.load('file.csv', format='csv')

C. spark.load('csv', 'file.csv')

D. spark.read.csv('file.csv')

Solution

1. Step 1: Recall Spark CSV loading syntax

   The standard way is to use spark.read.format('csv').load('filename').

2. Step 2: Check each option

   spark.read.format('csv').load('file.csv') matches the correct syntax; others are incorrect or invalid methods.

3. Final Answer:

   spark.read.format('csv').load('file.csv') -> Option A

4. Quick Check:

   Use format('csv').load() for CSV [OK]

Hint: Use spark.read.format('csv').load() for CSV files [OK]

Common Mistakes:

• Using spark.read.csv() without options
• Trying spark.load() which doesn't exist
• Passing format as a keyword argument in load()

3. Given the following code, what will be the schema of the DataFrame?

df = spark.read.format('json').load('people.json')
df.printSchema()

Assuming 'people.json' contains records like {"name": "Alice", "age": 30}.

medium

A. root
|-- name: string (nullable = true)
|-- age: long (nullable = true)

B. root
|-- name: integer (nullable = true)
|-- age: string (nullable = true)

C. root
|-- name: string (nullable = false)
|-- age: integer (nullable = false)

D. root
|-- name: string (nullable = true)
|-- age: string (nullable = true)

Solution

1. Step 1: Understand JSON schema inference

   Spark infers types from JSON: "name" is string, "age" is number (long).

2. Step 2: Check nullability and types

   By default, fields are nullable. Age inferred as long (integer type in Spark).

3. Final Answer:

   root |-- name: string (nullable = true) |-- age: long (nullable = true) -> Option A

4. Quick Check:

   JSON fields inferred correctly [OK]

Hint: JSON fields infer types automatically [OK]

Common Mistakes:

• Assuming age is string instead of long
• Thinking fields are non-nullable by default
• Confusing integer with string types

4. What is wrong with this code snippet to load a Parquet file?

df = spark.read.format('parquet').load('data.parquet')
df.show()

Assuming 'data.parquet' exists and is valid.

medium

A. You must specify schema before loading Parquet files

B. The format should be 'parq' instead of 'parquet'

C. The load method requires a file extension in the path

D. Nothing is wrong; this code will load and show the DataFrame correctly

Solution

1. Step 1: Check Parquet loading syntax

   Using spark.read.format('parquet').load('file') is correct and standard.

2. Step 2: Validate assumptions

   If the file exists and is valid, df.show() will display the data without error.

3. Final Answer:

   Nothing is wrong; this code will load and show the DataFrame correctly -> Option D

4. Quick Check:

   Correct Parquet load syntax [OK]

Hint: Parquet loads with format('parquet').load() [OK]

Common Mistakes:

• Using wrong format string like 'parq'
• Thinking schema is mandatory for Parquet
• Believing file extension is required in load()

5. You want to load a CSV file with a header and infer the schema automatically. Which code snippet correctly achieves this?

hard

A. spark.read.format('csv').load('data.csv').option('header', 'true').option('inferSchema', 'true')

B. spark.read.csv('data.csv', header=True, inferSchema=True)

C. spark.read.format('csv').option('header', 'true').option('inferSchema', 'true').load('data.csv')

D. spark.read.load('data.csv').option('header', 'true').option('inferSchema', 'true')

Solution

1. Step 1: Understand CSV options usage

   Options like header and inferSchema must be set before load() when using format().

2. Step 2: Analyze each option

   spark.read.format('csv').option('header', 'true').option('inferSchema', 'true').load('data.csv') sets options correctly before load(). spark.read.csv('data.csv', header=True, inferSchema=True) uses invalid syntax for Spark. spark.read.format('csv').load('data.csv').option('header', 'true').option('inferSchema', 'true') sets options after load(), which is wrong. spark.read.load('data.csv').option('header', 'true').option('inferSchema', 'true') misses format('csv').

3. Final Answer:

   spark.read.format('csv').option('header', 'true').option('inferSchema', 'true').load('data.csv') -> Option C

4. Quick Check:

   Set options before load() [OK]

Hint: Set CSV options before load() with format('csv') [OK]

Common Mistakes:

• Setting options after load() call
• Using Python-style keyword arguments in Spark
• Not specifying format('csv') before load()