Parquet format and columnar storage in Apache Spark - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

Output of reading Parquet file with Spark

What will be the output of the following Spark code snippet when reading a Parquet file and showing the first 3 rows?

Apache Spark

from pyspark.sql import SparkSession
spark = SparkSession.builder.appName('Test').getOrCreate()
df = spark.read.parquet('data/sample.parquet')
df.show(3)

A Throws a SyntaxError due to incorrect method usage

B Displays first 3 rows of the DataFrame with all columns in tabular format

C Raises FileNotFoundError because 'data/sample.parquet' does not exist

D Prints schema of the DataFrame instead of data rows

💡 Hint

The show() method displays rows of the DataFrame in a table format.

The show(3) method prints the first 3 rows of the DataFrame read from the Parquet file. If the file exists and is readable, it outputs the data in tabular form.

🧠 Conceptual

intermediate

Why is Parquet format efficient for big data?

Which of the following best explains why Parquet format is efficient for big data processing?

A Parquet compresses data and stores it in a columnar format allowing faster reads of specific columns

B Parquet stores data in a row-based format making it faster for all queries

C Parquet requires no schema which speeds up data loading

D Parquet stores data as plain text making it easy to read

💡 Hint

Think about how columnar storage helps with reading only needed data.

Parquet stores data in a columnar format which allows reading only the columns needed for a query. It also compresses data efficiently, reducing storage and speeding up processing.

❓ data_output

advanced

Result of filtering Parquet data by column

Given a Parquet file with columns 'name', 'age', and 'city', what will be the output count after running this Spark code?

Apache Spark

df = spark.read.parquet('data/people.parquet')
filtered_df = df.filter(df.age > 30)
count = filtered_df.count()
print(count)

A Total number of rows in the DataFrame regardless of age

B Raises AnalysisException due to missing column 'age'

C Zero, because filter syntax is incorrect

D Number of rows where age is greater than 30

💡 Hint

Filter keeps only rows matching the condition.

The filter(df.age > 30) keeps rows where age is greater than 30. The count() returns how many such rows exist.

🔧 Debug

advanced

Identify the error in Parquet write code

What error will this Spark code produce when trying to write a DataFrame to Parquet?

Apache Spark

df = spark.createDataFrame([(1, 'Alice'), (2, 'Bob')], ['id', 'name'])
df.write.parquet('output/data.parquet', mode='overwrite', format='csv')

A TypeError because 'format' is not a valid argument for write.parquet()

B Writes the DataFrame successfully in CSV format

C AnalysisException due to missing output directory

D SyntaxError due to incorrect method chaining

💡 Hint

Check the method signature for write.parquet().

The write.parquet() method does not accept a 'format' argument. To write in CSV, use write.format('csv').save().

🚀 Application

expert

Optimizing query performance with Parquet and column pruning

You have a large Parquet dataset with many columns. You want to speed up a Spark query that only needs 'user_id' and 'purchase_amount'. Which approach will best improve performance?

A Read the entire Parquet file and then select 'user_id' and 'purchase_amount' columns

B Convert Parquet to CSV first, then read only needed columns

C Use Spark to read only 'user_id' and 'purchase_amount' columns from the Parquet file directly

D Load Parquet data into memory fully before filtering columns

💡 Hint

Think about how columnar storage helps avoid reading unnecessary data.

Parquet's columnar format allows Spark to read only the requested columns, reducing I/O and speeding up queries. Selecting columns after reading all data wastes resources.