Partition tuning (repartition vs coalesce) in Apache Spark - Practice Questions

Challenge - 5 Problems

❓ Predict Output

intermediate

Output of repartition vs coalesce on DataFrame partitions

Consider a Spark DataFrame with 10 partitions. What will be the number of partitions after applying the following code?

df_repart = df.repartition(5)
df_coalesce = df.coalesce(5)

What are the number of partitions in df_repart and df_coalesce respectively?

Apache Spark

from pyspark.sql import SparkSession
spark = SparkSession.builder.getOrCreate()
data = [(i,) for i in range(100)]
df = spark.createDataFrame(data, ['num']).repartition(10)

print(df.rdd.getNumPartitions())
df_repart = df.repartition(5)
df_coalesce = df.coalesce(5)
print(df_repart.rdd.getNumPartitions())
print(df_coalesce.rdd.getNumPartitions())

A df_repart has 5 partitions, df_coalesce has 10 partitions

B df_repart has 10 partitions, df_coalesce has 5 partitions

C df_repart has 10 partitions, df_coalesce has 10 partitions

D df_repart has 5 partitions, df_coalesce has 5 partitions

💡 Hint

Both repartition and coalesce can change the number of partitions, but repartition always reshuffles data.

Both repartition(5) and coalesce(5) reduce the number of partitions to 5. Repartition reshuffles data fully, while coalesce tries to avoid shuffle but still reduces partitions.

🧠 Conceptual

intermediate

Difference in shuffle behavior between repartition and coalesce

Which statement correctly describes the shuffle behavior of repartition and coalesce in Spark?

A Both repartition and coalesce always cause shuffle regardless of partition count.

B Coalesce always causes a shuffle; repartition avoids shuffle when increasing partitions.

C Repartition always causes a shuffle; coalesce avoids shuffle when decreasing partitions.

D Neither repartition nor coalesce ever cause shuffle.

💡 Hint

Think about data movement when changing partitions.

Repartition always triggers a full shuffle to evenly distribute data. Coalesce tries to avoid shuffle when reducing partitions by merging existing partitions.

❓ data_output

advanced

Resulting partition sizes after coalesce without shuffle

Given a DataFrame with 8 partitions each containing 10 rows, what will be the approximate number of rows in each partition after applying df.coalesce(4) without shuffle?

Apache Spark

from pyspark.sql import SparkSession
spark = SparkSession.builder.getOrCreate()
data = [(i,) for i in range(80)]
df = spark.createDataFrame(data, ['num']).repartition(8)

print(df.rdd.glom().map(len).collect())
df_coalesce = df.coalesce(4)
print(df_coalesce.rdd.glom().map(len).collect())

A [20, 20, 20, 20]

B [10, 10, 10, 10, 10, 10, 10, 10]

C [40, 10, 10, 10]

D [15, 15, 15, 15]

💡 Hint

Coalesce merges partitions without reshuffling data.

Coalesce merges adjacent partitions, so 8 partitions with 10 rows each become 4 partitions with about 20 rows each.

🔧 Debug

advanced

Why does coalesce not increase partitions?

You run df.coalesce(20) on a DataFrame with 10 partitions. What will happen and why?

A The DataFrame will still have 10 partitions because coalesce cannot increase partitions.

B The DataFrame will have 5 partitions because coalesce halves the partitions automatically.

C The code will raise an error because coalesce cannot accept a number greater than current partitions.

D The DataFrame will have 20 partitions because coalesce can increase partitions.

💡 Hint

Check the behavior of coalesce when asked to increase partitions.

Coalesce only reduces partitions; it does not increase them. To increase partitions, repartition must be used.

🚀 Application

expert

Choosing repartition vs coalesce for performance optimization

You have a large Spark DataFrame with 100 partitions. You want to reduce partitions to 10 for faster downstream processing. You also want to minimize data shuffle to save time. Which approach is best?

A Use df.repartition(10) to evenly distribute data with shuffle.

B Use df.coalesce(10) to reduce partitions without shuffle.

C Use df.coalesce(10, shuffle=True) to reduce partitions with shuffle.

D Use df.repartition(10, shuffle=False) to reduce partitions without shuffle.

💡 Hint

Consider shuffle cost and partition reduction.

Coalesce reduces partitions without shuffle, which is faster but may cause uneven data distribution. Repartition always shuffles data.