Creating DataFrames from files (CSV, JSON, Parquet) in Apache Spark - Visual Walkthrough

Concept Flow - Creating DataFrames from files (CSV, JSON, Parquet)

Start Spark Session

↓

Choose file type: CSV/JSON/Parquet

↓

Call spark.read.format(file_type)

↓

Set options (header, inferSchema, etc.)

↓

Load file path

↓

Create DataFrame

↓

Use DataFrame for analysis or show()

This flow shows how Spark reads different file types step-by-step to create a DataFrame for analysis.

Execution Sample

Apache Spark

from pyspark.sql import SparkSession
spark = SparkSession.builder.getOrCreate()
df = spark.read.csv('data.csv', header=True, inferSchema=True)
df.show()

This code starts Spark, reads a CSV file with headers and schema inference, then shows the DataFrame content.

Execution Table

Step	Action	Input/Parameters	Result/Output
1	Start Spark Session	None	SparkSession object created
2	Choose file type	csv	Set format to 'csv'
3	Set options	header=True, inferSchema=True	Options set for reading CSV
4	Load file path	'data.csv'	File path set to 'data.csv'
5	Read file	spark.read.csv(...)	DataFrame created with data from CSV
6	Show DataFrame	df.show()	Prints first 20 rows of DataFrame
7	End	No more actions	Process complete

💡 All steps completed, DataFrame created and displayed

Variable Tracker

Variable	Start	After Step 1	After Step 5	Final
spark	None	SparkSession object	SparkSession object	SparkSession object
df	None	None	DataFrame with CSV data	DataFrame with CSV data

Key Moments - 3 Insights

Why do we need to set header=True when reading a CSV?

Setting header=True tells Spark the first row has column names, so it uses them instead of default column names like _c0, _c1. See execution_table step 3.

What happens if inferSchema=False?

If inferSchema=False, Spark treats all columns as strings by default, which may cause wrong data types. This is controlled in execution_table step 3.

How does Spark know which file format to read?

Spark uses the format method or the specific read method like read.csv, read.json, or read.parquet to know the file type. See execution_table step 2.

Visual Quiz - 3 Questions

Look at the execution table, what is the result after step 5?

A File path set to 'data.csv'

B SparkSession object created

C A DataFrame created with data from the CSV file

D Options set for reading CSV

💡 Hint

Check the 'Result/Output' column in row with Step 5

At which step do we specify that the CSV file has a header row?

A Step 3

B Step 2

C Step 4

D Step 5

💡 Hint

Look at the 'Action' and 'Input/Parameters' columns for header=True

If we change the file type to JSON, which step changes in the execution table?

A Step 3 only

B Steps 2 and 5

C Step 2 only

D Step 4 only

💡 Hint

File type affects format setting and reading method, see steps 2 and 5

Concept Snapshot

Creating DataFrames from files in Spark:
- Start SparkSession
- Use spark.read.format('csv'/'json'/'parquet')
- Set options like header=True, inferSchema=True
- Load file path with .load() or specific method
- Result is a DataFrame ready for analysis

Full Transcript

This lesson shows how to create DataFrames in Apache Spark by reading files like CSV, JSON, or Parquet. First, we start a SparkSession. Then, we choose the file type by setting the format or using specific read methods. We set options such as header=True to tell Spark if the file has column names, and inferSchema=True to detect data types automatically. Next, we load the file path. Spark reads the file and creates a DataFrame. Finally, we can use the DataFrame to analyze or display data. The execution table traces each step from starting Spark to showing the DataFrame. Key moments clarify why options like header and inferSchema matter. The visual quiz tests understanding of each step's role. This process helps beginners see how Spark reads files and creates DataFrames for data science tasks.

Practice

1. What is the main purpose of creating a DataFrame from a file in Apache Spark?

easy

A. To write data back to a file

B. To organize data in rows and columns for easy analysis

C. To delete data from the file system

D. To convert data into plain text format

Solution

1. Step 1: Understand DataFrame concept

   A DataFrame is a table-like structure with rows and columns used for data analysis.

2. Step 2: Purpose of loading files

   Loading files into DataFrames allows Spark to process and analyze data efficiently.

3. Final Answer:

   To organize data in rows and columns for easy analysis -> Option B

4. Quick Check:

   DataFrames = organized data [OK]

Hint: DataFrames = tables for easy data work [OK]

Common Mistakes:

• Confusing DataFrame creation with file writing
• Thinking DataFrames delete files
• Assuming DataFrames convert data to plain text

2. Which of the following is the correct syntax to load a CSV file into a Spark DataFrame?

easy

A. spark.read.format('csv').load('file.csv')

B. spark.read.load('file.csv', format='csv')

C. spark.load('csv', 'file.csv')

D. spark.read.csv('file.csv')

Solution

1. Step 1: Recall Spark CSV loading syntax

   The standard way is to use spark.read.format('csv').load('filename').

2. Step 2: Check each option

   spark.read.format('csv').load('file.csv') matches the correct syntax; others are incorrect or invalid methods.

3. Final Answer:

   spark.read.format('csv').load('file.csv') -> Option A

4. Quick Check:

   Use format('csv').load() for CSV [OK]

Hint: Use spark.read.format('csv').load() for CSV files [OK]

Common Mistakes:

• Using spark.read.csv() without options
• Trying spark.load() which doesn't exist
• Passing format as a keyword argument in load()

3. Given the following code, what will be the schema of the DataFrame?

df = spark.read.format('json').load('people.json')
df.printSchema()

Assuming 'people.json' contains records like {"name": "Alice", "age": 30}.

medium

A. root
|-- name: string (nullable = true)
|-- age: long (nullable = true)

B. root
|-- name: integer (nullable = true)
|-- age: string (nullable = true)

C. root
|-- name: string (nullable = false)
|-- age: integer (nullable = false)

D. root
|-- name: string (nullable = true)
|-- age: string (nullable = true)

Solution

1. Step 1: Understand JSON schema inference

   Spark infers types from JSON: "name" is string, "age" is number (long).

2. Step 2: Check nullability and types

   By default, fields are nullable. Age inferred as long (integer type in Spark).

3. Final Answer:

   root |-- name: string (nullable = true) |-- age: long (nullable = true) -> Option A

4. Quick Check:

   JSON fields inferred correctly [OK]

Hint: JSON fields infer types automatically [OK]

Common Mistakes:

• Assuming age is string instead of long
• Thinking fields are non-nullable by default
• Confusing integer with string types

4. What is wrong with this code snippet to load a Parquet file?

df = spark.read.format('parquet').load('data.parquet')
df.show()

Assuming 'data.parquet' exists and is valid.

medium

A. You must specify schema before loading Parquet files

B. The format should be 'parq' instead of 'parquet'

C. The load method requires a file extension in the path

D. Nothing is wrong; this code will load and show the DataFrame correctly

Solution

1. Step 1: Check Parquet loading syntax

   Using spark.read.format('parquet').load('file') is correct and standard.

2. Step 2: Validate assumptions

   If the file exists and is valid, df.show() will display the data without error.

3. Final Answer:

   Nothing is wrong; this code will load and show the DataFrame correctly -> Option D

4. Quick Check:

   Correct Parquet load syntax [OK]

Hint: Parquet loads with format('parquet').load() [OK]

Common Mistakes:

• Using wrong format string like 'parq'
• Thinking schema is mandatory for Parquet
• Believing file extension is required in load()

5. You want to load a CSV file with a header and infer the schema automatically. Which code snippet correctly achieves this?

hard

A. spark.read.format('csv').load('data.csv').option('header', 'true').option('inferSchema', 'true')

B. spark.read.csv('data.csv', header=True, inferSchema=True)

C. spark.read.format('csv').option('header', 'true').option('inferSchema', 'true').load('data.csv')

D. spark.read.load('data.csv').option('header', 'true').option('inferSchema', 'true')

Solution

1. Step 1: Understand CSV options usage

   Options like header and inferSchema must be set before load() when using format().

2. Step 2: Analyze each option

   spark.read.format('csv').option('header', 'true').option('inferSchema', 'true').load('data.csv') sets options correctly before load(). spark.read.csv('data.csv', header=True, inferSchema=True) uses invalid syntax for Spark. spark.read.format('csv').load('data.csv').option('header', 'true').option('inferSchema', 'true') sets options after load(), which is wrong. spark.read.load('data.csv').option('header', 'true').option('inferSchema', 'true') misses format('csv').

3. Final Answer:

   spark.read.format('csv').option('header', 'true').option('inferSchema', 'true').load('data.csv') -> Option C

4. Quick Check:

   Set options before load() [OK]

Hint: Set CSV options before load() with format('csv') [OK]

Common Mistakes:

• Setting options after load() call
• Using Python-style keyword arguments in Spark
• Not specifying format('csv') before load()