Why data format affects performance in Apache Spark - See It in Action

Why Data Format Affects Performance in Apache Spark

📖 Scenario: You work as a data analyst at a company that collects sales data daily. The data is saved in different file formats. Your manager wants to understand how the file format affects the speed of reading and processing data in Apache Spark.

🎯 Goal: You will create a Spark DataFrame from a small sales dataset saved in CSV format, then configure a variable to select a file format, read the data in that format, and finally measure and print the time taken to read the data. This will help you see how data format affects performance.

📋 What You'll Learn

Create a Spark session

Create a small sales dataset with exact values

Configure a variable to select the data format

Read the data using the selected format

Measure and print the time taken to read the data

💡 Why This Matters

🌍 Real World

Data engineers and analysts often choose data formats to optimize speed and storage when working with big data tools like Apache Spark.

💼 Career

Understanding how data format affects performance helps in designing efficient data pipelines and improves job performance in data engineering roles.

1

Create the sales dataset as a list of dictionaries

Create a variable called sales_data as a list of dictionaries with these exact entries: {'date': '2024-01-01', 'product': 'A', 'amount': 100}, {'date': '2024-01-02', 'product': 'B', 'amount': 150}, and {'date': '2024-01-03', 'product': 'C', 'amount': 200}.

Apache Spark

# Your code here

Hint

Use a list with three dictionaries exactly as shown.

2

Create a variable to select the data format

Create a variable called data_format and set it to the string 'csv'.

Apache Spark

sales_data = [
    {'date': '2024-01-01', 'product': 'A', 'amount': 100},
    {'date': '2024-01-02', 'product': 'B', 'amount': 150},
    {'date': '2024-01-03', 'product': 'C', 'amount': 200}
]
# Your code here

Hint

Set the variable data_format exactly to the string 'csv'.

3

Read the data using the selected format and measure time

Create a Spark session called spark. Then create a DataFrame called df by reading the sales_data in the format given by data_format. Use spark.createDataFrame(sales_data) to create the DataFrame first, then write it to a temporary file in the selected format. Finally, read the file back using spark.read.format(data_format).load(). Use the time module to measure the time taken to read the data and store it in a variable called read_time.

Apache Spark

sales_data = [
    {'date': '2024-01-01', 'product': 'A', 'amount': 100},
    {'date': '2024-01-02', 'product': 'B', 'amount': 150},
    {'date': '2024-01-03', 'product': 'C', 'amount': 200}
]
data_format = 'csv'
# Your code here

Hint

Use spark.createDataFrame to create the DataFrame, write it to a file in the chosen format, then read it back with spark.read.format(data_format).load(path). Use time.time() to measure reading time.

4

Print the time taken to read the data

Write a print statement to display the text "Time taken to read data in format: {data_format} is {read_time:.4f} seconds" using an f-string.

Apache Spark

import time
from pyspark.sql import SparkSession

sales_data = [
    {'date': '2024-01-01', 'product': 'A', 'amount': 100},
    {'date': '2024-01-02', 'product': 'B', 'amount': 150},
    {'date': '2024-01-03', 'product': 'C', 'amount': 200}
]
data_format = 'csv'

spark = SparkSession.builder.appName('DataFormatPerformance').getOrCreate()
df = spark.createDataFrame(sales_data)

# Write DataFrame to temporary path in selected format
path = f'/tmp/sales_data.{data_format}'
if data_format == 'csv':
    df.write.mode('overwrite').option('header', 'true').csv(path)
elif data_format == 'parquet':
    df.write.mode('overwrite').parquet(path)
elif data_format == 'json':
    df.write.mode('overwrite').json(path)

start_time = time.time()
df_read = spark.read.format(data_format).option('header', 'true').load(path)
read_time = time.time() - start_time
# Your code here

Hint

Use print(f"Time taken to read data in format: {data_format} is {read_time:.4f} seconds") to show the result.

Practice

1. Why does using a columnar data format like Parquet improve performance in Apache Spark?

easy

A. Because it converts all data into JSON format for easy parsing.

B. Because it compresses data into a single large file without structure.

C. Because it stores data by columns, allowing faster reads of specific columns.

D. Because it stores data as plain text for better compatibility.

Solution

1. Step 1: Understand columnar storage

   Columnar formats store data by columns, not rows, so Spark reads only needed columns.

2. Step 2: Impact on performance

   Reading fewer columns reduces disk I/O and speeds up queries.

3. Final Answer:

   Because it stores data by columns, allowing faster reads of specific columns. -> Option C

4. Quick Check:

   Columnar storage = faster column reads [OK]

Hint: Columnar = faster column access, less data read [OK]

Common Mistakes:

• Thinking all formats compress data the same way
• Confusing row-based with column-based storage
• Assuming JSON is always faster due to readability

2. Which of the following is the correct way to read a Parquet file in Apache Spark using PySpark?

easy

A. spark.read.parquet('data.parquet')

B. spark.read.csv('data.parquet')

C. spark.load('data.parquet')

D. spark.read.json('data.parquet')

Solution

1. Step 1: Recall Spark read syntax

   Spark uses spark.read.format() or shortcut methods like parquet() to read files.

2. Step 2: Identify correct method for Parquet

   spark.read.parquet() is the correct method to read Parquet files.

3. Final Answer:

   spark.read.parquet('data.parquet') -> Option A

4. Quick Check:

   Use spark.read.parquet() for Parquet files [OK]

Hint: Use spark.read.parquet() to read Parquet files [OK]

Common Mistakes:

• Using csv() or json() methods for Parquet files
• Trying to use spark.load() which is invalid
• Confusing file format with method name

3. Given the following PySpark code, what will be the output type of df.count() when reading a Parquet file?

df = spark.read.parquet('data.parquet')
count = df.count()

medium

A. A DataFrame containing the count

B. An integer representing the number of rows

C. A string describing the count

D. A list of counts per column

Solution

1. Step 1: Understand df.count() method

   The count() method returns the total number of rows as an integer.

2. Step 2: Confirm output type

   Since count() returns a number, the output is an integer, not a DataFrame or list.

3. Final Answer:

   An integer representing the number of rows -> Option B

4. Quick Check:

   df.count() returns integer row count [OK]

Hint: count() returns integer row count [OK]

Common Mistakes:

• Expecting count() to return a DataFrame
• Thinking count() returns a string or list
• Confusing count() with collect()

4. You try to read a JSON file with spark.read.parquet('data.json') and get an error. What is the most likely cause?

medium

A. Spark cannot read any file formats other than Parquet

B. The file path is incorrect

C. The JSON file is empty

D. You are using the wrong read method for the file format

Solution

1. Step 1: Identify file format mismatch

   The code uses read.parquet() on a JSON file, which causes a format error.

2. Step 2: Correct method for JSON files

   Use spark.read.json() to read JSON files correctly.

3. Final Answer:

   You are using the wrong read method for the file format -> Option D

4. Quick Check:

   Use matching read method for file format [OK]

Hint: Match read method to file format (json, parquet, csv) [OK]

Common Mistakes:

• Assuming file path is always the problem
• Thinking Spark only reads Parquet files
• Ignoring file format when reading data

5. You have a large dataset stored as CSV and want to improve query speed in Spark. Which approach best improves performance while minimizing storage size?

hard

A. Convert CSV to Parquet format and use Spark SQL queries

B. Keep CSV but compress files with gzip

C. Convert CSV to JSON format for better readability

D. Split CSV into many small files without changing format

Solution

1. Step 1: Understand format impact on performance

   Parquet is columnar and compressed, improving query speed and reducing storage.

2. Step 2: Compare alternatives

   Compressing CSV helps storage but not query speed; JSON is slower; many small files hurt performance.

3. Final Answer:

   Convert CSV to Parquet format and use Spark SQL queries -> Option A

4. Quick Check:

   Parquet = faster queries + smaller size [OK]

Hint: Convert to Parquet for speed and compression [OK]

Common Mistakes:

• Thinking compression alone speeds queries
• Choosing JSON for performance
• Splitting files without format change improves speed