Parquet format and columnar storage in Apache Spark - Mini Project: Build & Apply

Explore Parquet Format and Columnar Storage with Apache Spark

📖 Scenario: You work as a data analyst at a retail company. You have sales data in a simple list format. Your manager wants you to save this data efficiently using Parquet format, which stores data in columns. This helps save space and speeds up queries.

🎯 Goal: Learn how to create a Spark DataFrame from a list, save it as a Parquet file, read it back, and understand the benefits of columnar storage.

📋 What You'll Learn

Create a Spark DataFrame from a list of sales records

Define a configuration variable for the Parquet file path

Save the DataFrame as a Parquet file

Read the Parquet file back into a DataFrame

Print the loaded DataFrame to see the data

💡 Why This Matters

🌍 Real World

Parquet is widely used in big data systems to store large datasets efficiently. It helps companies save storage and speed up data analysis.

💼 Career

Data engineers and data scientists use Parquet format to handle large datasets in distributed systems like Apache Spark.

1

Create a Spark DataFrame with sales data

Create a list called sales_data with these exact entries: ("2024-01-01", "StoreA", 100), ("2024-01-02", "StoreB", 150), ("2024-01-03", "StoreA", 200). Then create a Spark DataFrame called df from sales_data with columns "date", "store", and "sales".

Apache Spark

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("ParquetExample").getOrCreate()
# Create the list sales_data with the given tuples
# Create the DataFrame df from sales_data with columns 'date', 'store', 'sales'

Hint

Use spark.createDataFrame() with your list and specify the column names as a list.

2

Set the Parquet file path

Create a string variable called parquet_path and set it to "sales_data.parquet".

Apache Spark

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("ParquetExample").getOrCreate()
sales_data = [("2024-01-01", "StoreA", 100), ("2024-01-02", "StoreB", 150), ("2024-01-03", "StoreA", 200)]
df = spark.createDataFrame(sales_data, ["date", "store", "sales"])
# Create the variable parquet_path and set it to "sales_data.parquet"

Hint

Just assign the string "sales_data.parquet" to the variable parquet_path.

3

Save the DataFrame as a Parquet file

Use the DataFrame df to save the data as a Parquet file at the path stored in parquet_path. Use the .write.parquet() method.

Apache Spark

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("ParquetExample").getOrCreate()
sales_data = [("2024-01-01", "StoreA", 100), ("2024-01-02", "StoreB", 150), ("2024-01-03", "StoreA", 200)]
df = spark.createDataFrame(sales_data, ["date", "store", "sales"])
parquet_path = "sales_data.parquet"
# Save df as a Parquet file at parquet_path

Hint

Use df.write.parquet(parquet_path) to save the DataFrame in Parquet format.

4

Read the Parquet file and display the data

Read the Parquet file from parquet_path into a new DataFrame called df_loaded. Then use print() with df_loaded.show() to display the data.

Apache Spark

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("ParquetExample").getOrCreate()
sales_data = [("2024-01-01", "StoreA", 100), ("2024-01-02", "StoreB", 150), ("2024-01-03", "StoreA", 200)]
df = spark.createDataFrame(sales_data, ["date", "store", "sales"])
parquet_path = "sales_data.parquet"
df.write.parquet(parquet_path)
# Read the Parquet file into df_loaded
# Print the contents of df_loaded using df_loaded.show()

Hint

Use spark.read.parquet(parquet_path) to read the file. Then call df_loaded.show() inside a print statement.

Practice

1. What is the main advantage of storing data in Parquet format in Apache Spark?

easy

A. It compresses data by converting it to images.

B. It stores data as plain text for easy reading.

C. It stores data only in memory without saving to disk.

D. It stores data column-wise, making queries faster and storage smaller.

Solution

1. Step 1: Understand Parquet storage method

   Parquet stores data in a columnar format, meaning data is saved column by column instead of row by row.

2. Step 2: Benefits of columnar storage

   This column-wise storage allows faster queries on specific columns and reduces storage size due to better compression.

3. Final Answer:

   It stores data column-wise, making queries faster and storage smaller. -> Option D

4. Quick Check:

   Columnar storage = faster queries + smaller files [OK]

Hint: Remember: Parquet = column-wise storage for speed and size [OK]

Common Mistakes:

• Thinking Parquet stores data as plain text
• Confusing row-wise with column-wise storage
• Assuming Parquet only stores in memory

2. Which of the following is the correct way to save a Spark DataFrame named df as a Parquet file?

easy

A. df.saveAsParquet('path/to/file')

B. df.write.parquet('path/to/file')

C. df.write.save('path/to/file', format='csv')

D. df.save('path/to/file', format='parquet')

Solution

1. Step 1: Recall Spark DataFrame write syntax

   In Spark, to save a DataFrame as Parquet, use the write.parquet() method.

2. Step 2: Check each option

   df.write.parquet('path/to/file') uses df.write.parquet('path'), which is correct. Others use incorrect methods or formats.

3. Final Answer:

   df.write.parquet('path/to/file') -> Option B

4. Quick Check:

   Use write.parquet() to save DataFrame [OK]

Hint: Use df.write.parquet(path) to save DataFrame as Parquet [OK]

Common Mistakes:

• Using saveAsParquet which does not exist
• Saving with wrong format like CSV
• Using df.save() without write

3. Given the following Spark code, what will be the output schema of the loaded DataFrame?

df.write.parquet('data.parquet')
df2 = spark.read.parquet('data.parquet')
df2.printSchema()

medium

A. The schema will be the same as the original DataFrame df.

B. The schema will be empty because Parquet does not store schema.

C. The schema will only contain column names but no data types.

D. The schema will be reversed compared to df.

Solution

1. Step 1: Understand Parquet schema storage

   Parquet files store both data and schema (column names and types).

2. Step 2: Loading Parquet preserves schema

   When reading with spark.read.parquet(), the schema is restored exactly as saved.

3. Final Answer:

   The schema will be the same as the original DataFrame df. -> Option A

4. Quick Check:

   Parquet stores schema = schema preserved [OK]

Hint: Parquet files keep schema; reading restores it fully [OK]

Common Mistakes:

• Thinking Parquet files do not store schema
• Assuming schema is lost or incomplete
• Believing schema order changes on read

4. You try to read a Parquet file using spark.read.parquet('file_path') but get an error. Which of the following is the most likely cause?

medium

A. The file path is incorrect or the file does not exist.

B. Parquet files cannot be read by Spark.

C. You must use read.csv() to read Parquet files.

D. The DataFrame was not saved before reading.

Solution

1. Step 1: Check common read errors

   Errors reading Parquet usually happen if the file path is wrong or file is missing.

2. Step 2: Verify other options

   Parquet files are supported by Spark, so B and C are wrong. D is unrelated because reading does not depend on saving in the same session.

3. Final Answer:

   The file path is incorrect or the file does not exist. -> Option A

4. Quick Check:

   File not found = read error [OK]

Hint: Check file path and existence before reading Parquet [OK]

Common Mistakes:

• Assuming Spark cannot read Parquet
• Using wrong read method like read.csv
• Thinking saving is required in same session

5. You have a large Spark DataFrame with many columns but only need to query a few columns frequently. How does using Parquet format help improve performance?

hard

A. Parquet stores data row-wise, so all columns are always read together.

B. Parquet converts all columns to JSON strings for faster parsing.

C. Parquet stores data column-wise, so reading only needed columns is faster and uses less disk I/O.

D. Parquet compresses data by removing all null values automatically.

Solution

1. Step 1: Recall Parquet columnar storage benefits

   Parquet stores data column-wise, allowing queries to read only selected columns.

2. Step 2: Understand impact on performance

   Reading fewer columns reduces disk I/O and speeds up queries, especially on large datasets.

3. Final Answer:

   Parquet stores data column-wise, so reading only needed columns is faster and uses less disk I/O. -> Option C

4. Quick Check:

   Columnar storage = selective fast reads [OK]

Hint: Parquet reads only needed columns, saving time and I/O [OK]

Common Mistakes:

• Thinking Parquet stores data row-wise
• Believing Parquet converts data to JSON
• Assuming Parquet removes nulls automatically