Writing output with partitioning in Apache Spark - Mini Project: Build & Apply

Writing output with partitioning

📖 Scenario: You work at a retail company. You have sales data for different stores and dates. You want to save this data so it is easy to find sales by store.

🎯 Goal: Create a Spark DataFrame with sales data, set a partition column, write the data partitioned by store, and show the output path structure.

📋 What You'll Learn

Create a Spark DataFrame with columns: store, date, sales

Create a variable called partition_column with value 'store'

Write the DataFrame to disk partitioned by the partition_column

Print the list of partition folders created

💡 Why This Matters

🌍 Real World

Partitioning data by a column like store helps organize large datasets so queries can run faster by reading only needed partitions.

💼 Career

Data engineers and data scientists often write partitioned data to improve performance and manageability in big data systems.

1

Create the sales DataFrame

Create a Spark DataFrame called sales_df with these rows exactly: ('StoreA', '2024-01-01', 100), ('StoreB', '2024-01-01', 150), ('StoreA', '2024-01-02', 200). The columns must be store, date, and sales.

Apache Spark

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName('PartitionExample').getOrCreate()
# Create the sales_df DataFrame with the exact rows and columns
# Your code here

Hint

Use spark.createDataFrame() with a list of tuples and a list of column names.

2

Set the partition column

Create a variable called partition_column and set it to the string 'store'.

Apache Spark

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName('PartitionExample').getOrCreate()
data = [('StoreA', '2024-01-01', 100), ('StoreB', '2024-01-01', 150), ('StoreA', '2024-01-02', 200)]
columns = ['store', 'date', 'sales']
sales_df = spark.createDataFrame(data, schema=columns)
# Create partition_column variable with value 'store'
# Your code here

Hint

Just assign the string 'store' to the variable partition_column.

3

Write the DataFrame partitioned by store

Write the sales_df DataFrame to disk in Parquet format, partitioned by the column stored in partition_column. Use the path 'output/sales_data'. Use mode('overwrite') to replace existing data.

Apache Spark

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName('PartitionExample').getOrCreate()
data = [('StoreA', '2024-01-01', 100), ('StoreB', '2024-01-01', 150), ('StoreA', '2024-01-02', 200)]
columns = ['store', 'date', 'sales']
sales_df = spark.createDataFrame(data, schema=columns)
partition_column = 'store'
# Write sales_df partitioned by partition_column to 'output/sales_data' with overwrite mode
# Your code here

Hint

Use sales_df.write.mode('overwrite').partitionBy(partition_column).parquet('output/sales_data').

4

List the partition folders

Import os. Use os.listdir to list the folders inside 'output/sales_data'. Print the list of folder names.

Apache Spark

from pyspark.sql import SparkSession
import os

spark = SparkSession.builder.appName('PartitionExample').getOrCreate()
data = [('StoreA', '2024-01-01', 100), ('StoreB', '2024-01-01', 150), ('StoreA', '2024-01-02', 200)]
columns = ['store', 'date', 'sales']
sales_df = spark.createDataFrame(data, schema=columns)
partition_column = 'store'
sales_df.write.mode('overwrite').partitionBy(partition_column).parquet('output/sales_data')
# List and print the folders inside 'output/sales_data'
# Your code here

Hint

Use os.listdir('output/sales_data') and print the result.

Practice

1. What does the partitionBy method do when writing data in Apache Spark?

easy

A. It sorts the data before saving to disk.

B. It compresses the output data to save disk space.

C. It splits the output data into folders based on column values.

D. It encrypts the output data for security.

Solution

1. Step 1: Understand the purpose of partitionBy

   The partitionBy method organizes output files into folders named by unique values of specified columns.

2. Step 2: Recognize its effect on data storage

   This splitting helps in faster data reading and better organization by grouping similar data together.

3. Final Answer:

   It splits the output data into folders based on column values. -> Option C

4. Quick Check:

   partitionBy splits data folders by column values [OK]

Hint: partitionBy creates folders by column values when saving [OK]

Common Mistakes:

• Thinking partitionBy compresses data
• Confusing partitionBy with sorting
• Assuming partitionBy encrypts data

2. Which of the following is the correct syntax to write a DataFrame df partitioned by column country in Spark?

easy

A. df.write.save.partitionBy("country")("path/to/output")

B. df.partitionBy.write("country").save("path/to/output")

C. df.write.save("path/to/output").partitionBy("country")

D. df.write.partitionBy("country").save("path/to/output")

Solution

1. Step 1: Recall the method chaining order

   In Spark, partitionBy is called on the write object before save.

2. Step 2: Verify correct syntax

   The correct syntax is df.write.partitionBy("country").save("path"). Other options misuse method order or call.

3. Final Answer:

   df.write.partitionBy("country").save("path/to/output") -> Option D

4. Quick Check:

   partitionBy called after write, before save [OK]

Hint: Use df.write.partitionBy(...).save(...) syntax [OK]

Common Mistakes:

• Calling partitionBy after save
• Using partitionBy on df instead of df.write
• Incorrect method chaining order

3. Given the following code, what folder structure will be created when saving the DataFrame?

df.write.partitionBy("year", "month").mode("overwrite").parquet("/data/events")

Assuming year has values 2023 and 2024, and month has values 1 and 2.

medium

A. /data/events/year=2023/month=1/, /data/events/year=2023/month=2/, /data/events/year=2024/month=1/, /data/events/year=2024/month=2/

B. /data/events/2023/1/, /data/events/2023/2/, /data/events/2024/1/, /data/events/2024/2/

C. /data/events/year/month/2023/1/, /data/events/year/month/2023/2/, /data/events/year/month/2024/1/, /data/events/year/month/2024/2/

D. /data/events/year=month=2023_1/, /data/events/year=month=2023_2/, /data/events/year=month=2024_1/, /data/events/year=month=2024_2/

Solution

1. Step 1: Understand partition folder naming

   Spark creates folders named by each partition column and its value, nested in order of columns.

2. Step 2: Apply to given columns and values

   Folders will be nested as /year=2023/month=1/ etc., not just values or combined keys.

3. Final Answer:

   /data/events/year=2023/month=1/, /data/events/year=2023/month=2/, /data/events/year=2024/month=1/, /data/events/year=2024/month=2/ -> Option A

4. Quick Check:

   Partition folders named as column=value pairs [OK]

Hint: Partition folders use column=value format nested by columns [OK]

Common Mistakes:

• Using only values without column names in folders
• Combining columns into one folder name
• Adding extra folder levels incorrectly

4. Identify the error in this Spark code that tries to write a DataFrame partitioned by category:

df.write.save("/output/path").partitionBy("category")

medium

A. The path should be a directory, not a file.

B. partitionBy should be called before save, not after.

C. partitionBy requires a list of columns, not a single string.

D. The DataFrame must be cached before writing.

Solution

1. Step 1: Check method chaining order

   In Spark, partitionBy must be called on the write object before save.

2. Step 2: Identify the error in the code

   The code calls save first, then partitionBy, which is invalid and causes an error.

3. Final Answer:

   partitionBy should be called before save, not after. -> Option B

4. Quick Check:

   partitionBy before save is required [OK]

Hint: Call partitionBy before save to avoid errors [OK]

Common Mistakes:

• Calling partitionBy after save
• Passing a single string instead of list (both allowed but string is valid)
• Confusing caching with writing order

5. You have a large DataFrame df with columns country, year, and sales. You want to save it partitioned by country and year to speed up queries filtering by these columns. Which approach is best to write the data efficiently?

hard

A. Use df.write.partitionBy("country", "year").mode("overwrite").parquet("/data/sales") to save partitioned parquet files.

B. Save the DataFrame without partitioning and filter later during reading.

C. Use df.write.mode("overwrite").csv("/data/sales") without partitioning for simplicity.

D. Write the DataFrame multiple times, once per country and year combination.

Solution

1. Step 1: Understand partitioning benefits

   Partitioning by country and year organizes data into folders, speeding up queries filtering on these columns.

2. Step 2: Evaluate options for efficient writing

   Use df.write.partitionBy("country", "year").mode("overwrite").parquet("/data/sales") to save partitioned parquet files. This uses partitionBy with parquet format and overwrite mode, which is efficient and standard practice.

3. Step 3: Reject less efficient options

   Options B and C do not partition, causing slower queries. Writing the DataFrame multiple times, once per country and year combination, is inefficient and error-prone.

4. Final Answer:

   Use df.write.partitionBy("country", "year").mode("overwrite").parquet("/data/sales") to save partitioned parquet files. -> Option A

5. Quick Check:

   partitionBy with parquet speeds queries [OK]

Hint: Partition by query filter columns using parquet format [OK]

Common Mistakes:

• Not partitioning data before saving
• Writing multiple times instead of once
• Using CSV without partitioning for large data