Creating DataFrames from files (CSV, JSON, Parquet) in Apache Spark - Visual Walkthrough

Concept Flow - Creating DataFrames from files (CSV, JSON, Parquet)

Start Spark Session

↓

Choose file type: CSV/JSON/Parquet

↓

Call spark.read.format(file_type)

↓

Set options (header, inferSchema, etc.)

↓

Load file path

↓

Create DataFrame

↓

Use DataFrame for analysis or show()

This flow shows how Spark reads different file types step-by-step to create a DataFrame for analysis.

Execution Sample

Apache Spark

from pyspark.sql import SparkSession
spark = SparkSession.builder.getOrCreate()
df = spark.read.csv('data.csv', header=True, inferSchema=True)
df.show()

This code starts Spark, reads a CSV file with headers and schema inference, then shows the DataFrame content.

Execution Table

Step	Action	Input/Parameters	Result/Output
1	Start Spark Session	None	SparkSession object created
2	Choose file type	csv	Set format to 'csv'
3	Set options	header=True, inferSchema=True	Options set for reading CSV
4	Load file path	'data.csv'	File path set to 'data.csv'
5	Read file	spark.read.csv(...)	DataFrame created with data from CSV
6	Show DataFrame	df.show()	Prints first 20 rows of DataFrame
7	End	No more actions	Process complete

💡 All steps completed, DataFrame created and displayed

Variable Tracker

Variable	Start	After Step 1	After Step 5	Final
spark	None	SparkSession object	SparkSession object	SparkSession object
df	None	None	DataFrame with CSV data	DataFrame with CSV data

Key Moments - 3 Insights

Why do we need to set header=True when reading a CSV?

Setting header=True tells Spark the first row has column names, so it uses them instead of default column names like _c0, _c1. See execution_table step 3.

What happens if inferSchema=False?

If inferSchema=False, Spark treats all columns as strings by default, which may cause wrong data types. This is controlled in execution_table step 3.

How does Spark know which file format to read?

Spark uses the format method or the specific read method like read.csv, read.json, or read.parquet to know the file type. See execution_table step 2.

Visual Quiz - 3 Questions

Look at the execution table, what is the result after step 5?

A File path set to 'data.csv'

B SparkSession object created

C A DataFrame created with data from the CSV file

D Options set for reading CSV

💡 Hint

Check the 'Result/Output' column in row with Step 5

At which step do we specify that the CSV file has a header row?

A Step 3

B Step 2

C Step 4

D Step 5

💡 Hint

Look at the 'Action' and 'Input/Parameters' columns for header=True

If we change the file type to JSON, which step changes in the execution table?

A Step 3 only

B Steps 2 and 5

C Step 2 only

D Step 4 only

💡 Hint

File type affects format setting and reading method, see steps 2 and 5

Concept Snapshot

Creating DataFrames from files in Spark:
- Start SparkSession
- Use spark.read.format('csv'/'json'/'parquet')
- Set options like header=True, inferSchema=True
- Load file path with .load() or specific method
- Result is a DataFrame ready for analysis

Full Transcript

This lesson shows how to create DataFrames in Apache Spark by reading files like CSV, JSON, or Parquet. First, we start a SparkSession. Then, we choose the file type by setting the format or using specific read methods. We set options such as header=True to tell Spark if the file has column names, and inferSchema=True to detect data types automatically. Next, we load the file path. Spark reads the file and creates a DataFrame. Finally, we can use the DataFrame to analyze or display data. The execution table traces each step from starting Spark to showing the DataFrame. Key moments clarify why options like header and inferSchema matter. The visual quiz tests understanding of each step's role. This process helps beginners see how Spark reads files and creates DataFrames for data science tasks.