Why data format affects performance in Apache Spark - Challenge Your Understanding

Challenge - 5 Problems

🧠 Conceptual

intermediate

Why does Parquet format improve query speed compared to CSV?

In Apache Spark, you have two datasets: one stored as CSV files and another as Parquet files. Both contain the same data. Why does reading Parquet files usually result in faster query performance?

A Parquet files are compressed and store data in a columnar format, allowing Spark to read only needed columns, reducing disk I/O.

B CSV files are binary and harder to parse, so Spark spends more time decoding them.

C Parquet files are stored in memory by default, so reading them is instant.

D CSV files contain metadata that slows down Spark's query optimizer.

💡 Hint

Think about how data is stored and how Spark reads only parts of the data it needs.

Parquet is a columnar storage format that compresses data and stores metadata, enabling Spark to skip reading unnecessary columns. CSV is row-based and uncompressed, so Spark reads all data even if only some columns are needed.

❓ data_output

intermediate

Comparing file sizes of CSV and Parquet

You save the same DataFrame in Spark as CSV and Parquet formats. Which option correctly shows the typical size difference between the two files?

Apache Spark

df.write.csv('data_csv')
df.write.parquet('data_parquet')

# After saving, check file sizes of 'data_csv' and 'data_parquet' folders

A The Parquet folder size is larger because it stores extra metadata.

B The CSV folder size is smaller because CSV is a plain text format.

C Both folders have the same size because they store the same data.

D The Parquet folder size is usually smaller than the CSV folder size due to compression.

💡 Hint

Consider how compression affects file size.

Parquet files use compression and efficient encoding, so they usually take less disk space than CSV files, which are plain text and uncompressed.

❓ Predict Output

advanced

What is the output of this Spark code reading JSON vs Parquet?

Consider this Spark code snippet:

df_json = spark.read.json('data.json')
df_parquet = spark.read.parquet('data.parquet')

print(df_json.count())
print(df_parquet.count())

Assuming both files contain the same data, what will be the output?

Apache Spark

df_json = spark.read.json('data.json')
df_parquet = spark.read.parquet('data.parquet')

print(df_json.count())
print(df_parquet.count())

A df_parquet.count() prints zero because Parquet files need special schema.

B Both print the same number of rows because the data is identical.

C df_json.count() is slower and prints fewer rows due to parsing errors.

D df_json.count() raises an error because JSON is not supported.

💡 Hint

Think about how Spark reads data formats and counts rows.

Both JSON and Parquet readers load the data and count rows. If the data is the same and valid, counts will match. Performance differs but output count is the same.

❓ visualization

advanced

Which plot best shows performance difference by data format?

You run the same query on datasets stored as CSV, JSON, and Parquet. You want to visualize query execution time for each format. Which plot type best shows this comparison clearly?

A A line chart showing execution time over multiple runs for one format only.

B A pie chart showing percentage of total execution time per format.

C A bar chart with data formats on the x-axis and execution time on the y-axis.

D A scatter plot with random points representing execution times.

💡 Hint

Think about comparing categories with numeric values.

A bar chart clearly compares execution times across different data formats side by side. Pie charts and scatter plots are less clear for this purpose.

🔧 Debug

expert

Why does this Spark job run slowly with CSV but fast with Parquet?

You have this Spark code:

df = spark.read.csv('data.csv', header=True)
df_filtered = df.filter(df['age'] > 30)
df_filtered.show()

The same filter on a Parquet file runs much faster. What is the main reason for this difference?

A CSV format lacks column statistics and schema, so Spark cannot push down filters and must scan all data.

B Parquet files are stored in memory, so filtering is instant.

C CSV files are encrypted by default, slowing down reading.

D Spark applies filter only on Parquet files, ignoring filters on CSV.

💡 Hint

Consider how Spark optimizes queries differently for formats.

Parquet stores metadata and column statistics enabling filter pushdown, so Spark reads only relevant data. CSV lacks this, so Spark scans entire files, making it slower.

Practice

1. Why does using a columnar data format like Parquet improve performance in Apache Spark?

easy

A. Because it converts all data into JSON format for easy parsing.

B. Because it compresses data into a single large file without structure.

C. Because it stores data by columns, allowing faster reads of specific columns.

D. Because it stores data as plain text for better compatibility.

Solution

1. Step 1: Understand columnar storage

   Columnar formats store data by columns, not rows, so Spark reads only needed columns.

2. Step 2: Impact on performance

   Reading fewer columns reduces disk I/O and speeds up queries.

3. Final Answer:

   Because it stores data by columns, allowing faster reads of specific columns. -> Option C

4. Quick Check:

   Columnar storage = faster column reads [OK]

Hint: Columnar = faster column access, less data read [OK]

Common Mistakes:

• Thinking all formats compress data the same way
• Confusing row-based with column-based storage
• Assuming JSON is always faster due to readability

2. Which of the following is the correct way to read a Parquet file in Apache Spark using PySpark?

easy

A. spark.read.parquet('data.parquet')

B. spark.read.csv('data.parquet')

C. spark.load('data.parquet')

D. spark.read.json('data.parquet')

Solution

1. Step 1: Recall Spark read syntax

   Spark uses spark.read.format() or shortcut methods like parquet() to read files.

2. Step 2: Identify correct method for Parquet

   spark.read.parquet() is the correct method to read Parquet files.

3. Final Answer:

   spark.read.parquet('data.parquet') -> Option A

4. Quick Check:

   Use spark.read.parquet() for Parquet files [OK]

Hint: Use spark.read.parquet() to read Parquet files [OK]

Common Mistakes:

• Using csv() or json() methods for Parquet files
• Trying to use spark.load() which is invalid
• Confusing file format with method name

3. Given the following PySpark code, what will be the output type of df.count() when reading a Parquet file?

df = spark.read.parquet('data.parquet')
count = df.count()

medium

A. A DataFrame containing the count

B. An integer representing the number of rows

C. A string describing the count

D. A list of counts per column

Solution

1. Step 1: Understand df.count() method

   The count() method returns the total number of rows as an integer.

2. Step 2: Confirm output type

   Since count() returns a number, the output is an integer, not a DataFrame or list.

3. Final Answer:

   An integer representing the number of rows -> Option B

4. Quick Check:

   df.count() returns integer row count [OK]

Hint: count() returns integer row count [OK]

Common Mistakes:

• Expecting count() to return a DataFrame
• Thinking count() returns a string or list
• Confusing count() with collect()

4. You try to read a JSON file with spark.read.parquet('data.json') and get an error. What is the most likely cause?

medium

A. Spark cannot read any file formats other than Parquet

B. The file path is incorrect

C. The JSON file is empty

D. You are using the wrong read method for the file format

Solution

1. Step 1: Identify file format mismatch

   The code uses read.parquet() on a JSON file, which causes a format error.

2. Step 2: Correct method for JSON files

   Use spark.read.json() to read JSON files correctly.

3. Final Answer:

   You are using the wrong read method for the file format -> Option D

4. Quick Check:

   Use matching read method for file format [OK]

Hint: Match read method to file format (json, parquet, csv) [OK]

Common Mistakes:

• Assuming file path is always the problem
• Thinking Spark only reads Parquet files
• Ignoring file format when reading data

5. You have a large dataset stored as CSV and want to improve query speed in Spark. Which approach best improves performance while minimizing storage size?

hard

A. Convert CSV to Parquet format and use Spark SQL queries

B. Keep CSV but compress files with gzip

C. Convert CSV to JSON format for better readability

D. Split CSV into many small files without changing format

Solution

1. Step 1: Understand format impact on performance

   Parquet is columnar and compressed, improving query speed and reducing storage.

2. Step 2: Compare alternatives

   Compressing CSV helps storage but not query speed; JSON is slower; many small files hurt performance.

3. Final Answer:

   Convert CSV to Parquet format and use Spark SQL queries -> Option A

4. Quick Check:

   Parquet = faster queries + smaller size [OK]

Hint: Convert to Parquet for speed and compression [OK]

Common Mistakes:

• Thinking compression alone speeds queries
• Choosing JSON for performance
• Splitting files without format change improves speed