Writing output with partitioning in Apache Spark - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

Output of partitioned write with multiple columns

What will be the folder structure created when writing a DataFrame partitioned by two columns?

Consider this Spark code:

df.write.partitionBy("year", "month").parquet("/data/events")

The DataFrame has data for years 2022 and 2023, and months 1 and 2.

Which folder paths will be created inside /data/events?

A /data/events/year=2022/month=1/, /data/events/year=2022/month=2/, /data/events/year=2023/month=1/, /data/events/year=2023/month=2/

B /data/events/month=1/year=2022/, /data/events/month=2/year=2022/, /data/events/month=1/year=2023/, /data/events/month=2/year=2023/

C /data/events/year=2022/, /data/events/year=2023/, /data/events/month=1/, /data/events/month=2/

D /data/events/year=2022_month=1/, /data/events/year=2022_month=2/, /data/events/year=2023_month=1/, /data/events/year=2023_month=2/

💡 Hint

Partition folders are created in the order of the columns specified in partitionBy.

When writing with partitionBy("year", "month"), Spark creates nested folders first by year, then by month inside each year folder.

❓ data_output

intermediate

Number of files created with partitioning

You write a DataFrame with 3 partitions using partitionBy("category"). The DataFrame has 2 unique categories: 'A' and 'B'.

If the DataFrame has 6 total partitions before writing, how many parquet files will be created inside the output folder?

A 2 files

B 6 files

C 3 files

D 1 file

💡 Hint

Each partition of the DataFrame writes one file per partition folder.

Even though there are 2 categories, the DataFrame has 6 partitions. Each partition writes one file, so total files = 6.

🔧 Debug

advanced

Error when writing partitioned data with unsupported column

What error will this Spark code produce?

df.write.partitionBy("nonexistent_column").parquet("/output/path")

The DataFrame df does not have a column named nonexistent_column.

A AnalysisException: 'Partition column nonexistent_column not found in schema'

B NullPointerException

C No error, writes data ignoring the partition column

D FileNotFoundException

💡 Hint

Partition columns must exist in the DataFrame schema.

Spark throws an AnalysisException if the partition column is missing from the DataFrame.

🚀 Application

advanced

Choosing partition columns for efficient queries

You have a large sales dataset with columns: date, region, product, and sales_amount.

You want to write the data partitioned to speed up queries filtering by region and date. Which partitioning strategy is best?

A Partition by <code>product</code>

B Partition by <code>date</code> only

C Partition by <code>region</code> only

D Partition by <code>region</code> and <code>date</code> (in that order)

💡 Hint

Partition columns should match common filter columns in queries.

Partitioning by both region and date helps queries filtering on these columns run faster by pruning partitions.

🧠 Conceptual

expert

Effect of partitioning on data skew and shuffle

Which statement best describes the impact of partitioning output data on data skew and shuffle during a Spark write?

A Partitioning always reduces shuffle and eliminates data skew

B Partitioning increases shuffle and always causes data skew

C Partitioning can reduce shuffle but may cause data skew if partition keys are unevenly distributed

D Partitioning has no effect on shuffle or data skew

💡 Hint

Think about how data distribution affects partitioning.

Partitioning can reduce shuffle by grouping data but if partition keys are uneven, some partitions become very large causing skew.