Writing output with partitioning in Apache Spark - Step-by-Step Execution

Concept Flow - Writing output with partitioning

Start with DataFrame

↓

Choose partition column(s)

↓

Write DataFrame with partitionBy()

↓

Spark creates folders for each partition

↓

Data saved in partition folders

↓

Output ready with organized partitions

This flow shows how Spark writes data by splitting it into folders based on chosen columns, making data organized and easy to query.

Execution Sample

Apache Spark

df.write.partitionBy('country').mode('overwrite').parquet('output_path')

This code writes the DataFrame to disk, splitting data into folders by 'country' values.

Execution Table

Step	Action	Partition Column Value	Folder Created	Data Written
1	Start writing DataFrame	-	-	-
2	Identify unique 'country' values	USA, Canada, Mexico	-	-
3	Create folder for 'country=USA'	USA	output_path/country=USA	-
4	Write rows with country=USA	USA	output_path/country=USA	Rows with USA
5	Create folder for 'country=Canada'	Canada	output_path/country=Canada	-
6	Write rows with country=Canada	Canada	output_path/country=Canada	Rows with Canada
7	Create folder for 'country=Mexico'	Mexico	output_path/country=Mexico	-
8	Write rows with country=Mexico	Mexico	output_path/country=Mexico	Rows with Mexico
9	Finish writing all partitions	-	-	All data saved in partition folders

💡 All data is saved in separate folders by 'country', completing the partitioned write.

Variable Tracker

Variable	Start	After Step 2	After Step 4	After Step 6	After Step 8	Final
df	Full DataFrame	Full DataFrame	Filtered USA rows	Filtered Canada rows	Filtered Mexico rows	All partitions written
partition_column_values	Not identified	['USA', 'Canada', 'Mexico']	['USA', 'Canada', 'Mexico']	['USA', 'Canada', 'Mexico']	['USA', 'Canada', 'Mexico']	['USA', 'Canada', 'Mexico']
folders_created	None	None	output_path/country=USA	output_path/country=USA, output_path/country=Canada	output_path/country=USA, output_path/country=Canada, output_path/country=Mexico	All partition folders created

Key Moments - 3 Insights

Why does Spark create separate folders for each partition value?

Spark creates folders to organize data by partition column values, making it faster to read only needed partitions later. See execution_table steps 3, 5, and 7.

What happens if the partition column has many unique values?

Spark will create many folders, one per unique value, which can slow down writing and reading. This is shown in execution_table step 2 where unique values are identified.

Does partitioning change the original DataFrame data?

No, partitioning only changes how data is saved on disk, not the DataFrame itself. The variable_tracker shows df filtered per partition only during writing, but original df stays full.

Visual Quiz - 3 Questions

Look at the execution_table, at which step does Spark write data for 'country=Canada'?

A Step 7

B Step 4

C Step 6

D Step 8

💡 Hint

Check the 'Partition Column Value' and 'Data Written' columns in execution_table rows.

According to variable_tracker, what is the state of 'folders_created' after Step 6?

A USA and Canada folders created

B Only USA folder created

C No folders created yet

D All partition folders created

💡 Hint

Look at the 'folders_created' row and the 'After Step 6' column in variable_tracker.

If the DataFrame had no 'country' column, what would happen when running the code?

A Spark writes data without partitioning

B Spark throws an error about missing partition column

C Spark creates folders with empty names

D Spark writes data to a single folder named 'country'

💡 Hint

Think about how Spark uses the partition column to create folders, see concept_flow.

Concept Snapshot

Writing output with partitioning in Spark:
- Use df.write.partitionBy('col') to split data by column values
- Spark creates folders named col=value
- Data is saved inside these folders
- Helps faster queries by reading only needed partitions
- Partition column must exist in DataFrame

Full Transcript

This lesson shows how Apache Spark writes data with partitioning. Starting with a DataFrame, Spark identifies unique values in the chosen partition column. For each unique value, it creates a folder named with that value. Then Spark writes rows matching that value into the folder. This organizes data on disk by partition, making later queries faster. The variable tracker shows how the DataFrame is filtered per partition during writing, but the original DataFrame remains unchanged. Key points include why folders are created, what happens with many unique values, and that partitioning only affects storage, not the DataFrame itself. The quiz tests understanding of steps where data is written, folder creation state, and error if partition column is missing.