Reading CSV files with options in Apache Spark - Mini Project: Build & Apply

Reading CSV files with options

📖 Scenario: You have a CSV file containing sales data from a store. The file has a header row, and some fields might be missing. You want to read this file into a Spark DataFrame correctly by specifying options.

🎯 Goal: Learn how to read a CSV file in Apache Spark using options like header and inferSchema to get a DataFrame with correct column names and data types.

📋 What You'll Learn

Create a Spark session

Read a CSV file with header and schema inference options

Show the DataFrame content

💡 Why This Matters

🌍 Real World

Reading CSV files with options is common when working with data exported from spreadsheets or databases. It helps ensure data is read correctly with proper column names and types.

💼 Career

Data scientists and data engineers often need to read CSV files with different formats and options to prepare data for analysis or machine learning.

1

Create Spark session and specify CSV file path

Create a Spark session called spark and create a variable csv_file_path with the value "sales_data.csv".

Apache Spark

# Create Spark session and set CSV file path
# Your code here

Need a hint?

Use SparkSession.builder.appName(...).getOrCreate() to create the Spark session.

2

Set options for reading CSV file

Create a variable csv_options as a dictionary with keys header set to "true" and inferSchema set to "true".

Apache Spark

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("CSVReader").getOrCreate()
csv_file_path = "sales_data.csv"

# Create csv_options dictionary with header and inferSchema set to "true"
# Your code here

Need a hint?

Use a Python dictionary with keys header and inferSchema both set to string "true".

3

Read the CSV file into a DataFrame using options

Use spark.read.options(**csv_options).csv(csv_file_path) to read the CSV file into a DataFrame called df.

Apache Spark

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("CSVReader").getOrCreate()
csv_file_path = "sales_data.csv"
csv_options = {"header": "true", "inferSchema": "true"}

# Read CSV file into DataFrame df using csv_options
# Your code here

Need a hint?

Use spark.read.options(**csv_options).csv(csv_file_path) to read the CSV file.

4

Show the DataFrame content

Call df.show() to display the DataFrame content.

Apache Spark

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("CSVReader").getOrCreate()
csv_file_path = "sales_data.csv"
csv_options = {"header": "true", "inferSchema": "true"}
df = spark.read.options(**csv_options).csv(csv_file_path)

# Show the DataFrame content
# Your code here

Need a hint?

Use df.show() to display the DataFrame rows in the console.