Partition tuning (repartition vs coalesce) in Apache Spark - Practice Questions

Challenge - 5 Problems

❓ Predict Output

intermediate

Output of repartition vs coalesce on DataFrame partitions

Consider a Spark DataFrame with 10 partitions. What will be the number of partitions after applying the following code?

df_repart = df.repartition(5)
df_coalesce = df.coalesce(5)

What are the number of partitions in df_repart and df_coalesce respectively?

Apache Spark

from pyspark.sql import SparkSession
spark = SparkSession.builder.getOrCreate()
data = [(i,) for i in range(100)]
df = spark.createDataFrame(data, ['num']).repartition(10)

print(df.rdd.getNumPartitions())
df_repart = df.repartition(5)
df_coalesce = df.coalesce(5)
print(df_repart.rdd.getNumPartitions())
print(df_coalesce.rdd.getNumPartitions())

A df_repart has 5 partitions, df_coalesce has 10 partitions

B df_repart has 10 partitions, df_coalesce has 5 partitions

C df_repart has 10 partitions, df_coalesce has 10 partitions

D df_repart has 5 partitions, df_coalesce has 5 partitions

💡 Hint

Both repartition and coalesce can change the number of partitions, but repartition always reshuffles data.

Both repartition(5) and coalesce(5) reduce the number of partitions to 5. Repartition reshuffles data fully, while coalesce tries to avoid shuffle but still reduces partitions.

🧠 Conceptual

intermediate

Difference in shuffle behavior between repartition and coalesce

Which statement correctly describes the shuffle behavior of repartition and coalesce in Spark?

A Both repartition and coalesce always cause shuffle regardless of partition count.

B Coalesce always causes a shuffle; repartition avoids shuffle when increasing partitions.

C Repartition always causes a shuffle; coalesce avoids shuffle when decreasing partitions.

D Neither repartition nor coalesce ever cause shuffle.

💡 Hint

Think about data movement when changing partitions.

Repartition always triggers a full shuffle to evenly distribute data. Coalesce tries to avoid shuffle when reducing partitions by merging existing partitions.

❓ data_output

advanced

Resulting partition sizes after coalesce without shuffle

Given a DataFrame with 8 partitions each containing 10 rows, what will be the approximate number of rows in each partition after applying df.coalesce(4) without shuffle?

Apache Spark

from pyspark.sql import SparkSession
spark = SparkSession.builder.getOrCreate()
data = [(i,) for i in range(80)]
df = spark.createDataFrame(data, ['num']).repartition(8)

print(df.rdd.glom().map(len).collect())
df_coalesce = df.coalesce(4)
print(df_coalesce.rdd.glom().map(len).collect())

A [20, 20, 20, 20]

B [10, 10, 10, 10, 10, 10, 10, 10]

C [40, 10, 10, 10]

D [15, 15, 15, 15]

💡 Hint

Coalesce merges partitions without reshuffling data.

Coalesce merges adjacent partitions, so 8 partitions with 10 rows each become 4 partitions with about 20 rows each.

🔧 Debug

advanced

Why does coalesce not increase partitions?

You run df.coalesce(20) on a DataFrame with 10 partitions. What will happen and why?

A The DataFrame will still have 10 partitions because coalesce cannot increase partitions.

B The DataFrame will have 5 partitions because coalesce halves the partitions automatically.

C The code will raise an error because coalesce cannot accept a number greater than current partitions.

D The DataFrame will have 20 partitions because coalesce can increase partitions.

💡 Hint

Check the behavior of coalesce when asked to increase partitions.

Coalesce only reduces partitions; it does not increase them. To increase partitions, repartition must be used.

🚀 Application

expert

Choosing repartition vs coalesce for performance optimization

You have a large Spark DataFrame with 100 partitions. You want to reduce partitions to 10 for faster downstream processing. You also want to minimize data shuffle to save time. Which approach is best?

A Use df.repartition(10) to evenly distribute data with shuffle.

B Use df.coalesce(10) to reduce partitions without shuffle.

C Use df.coalesce(10, shuffle=True) to reduce partitions with shuffle.

D Use df.repartition(10, shuffle=False) to reduce partitions without shuffle.

💡 Hint

Consider shuffle cost and partition reduction.

Coalesce reduces partitions without shuffle, which is faster but may cause uneven data distribution. Repartition always shuffles data.

Practice

1. Which statement best describes the difference between repartition and coalesce in Apache Spark?

easy

A. coalesce only increases partitions with shuffle; repartition only decreases partitions without shuffle.

B. coalesce can increase or decrease partitions with shuffle; repartition only decreases partitions without shuffle.

C. repartition only increases partitions without shuffle; coalesce only decreases partitions with shuffle.

D. repartition can increase or decrease partitions with shuffle; coalesce only decreases partitions without shuffle.

Solution

1. Step 1: Understand repartition behavior

   repartition reshuffles data and can increase or decrease the number of partitions, causing a full shuffle.

2. Step 2: Understand coalesce behavior

   coalesce reduces the number of partitions without a full shuffle, making it faster but it cannot increase partitions.

3. Final Answer:

   repartition can increase or decrease partitions with shuffle; coalesce only decreases partitions without shuffle. -> Option D

4. Quick Check:

   repartition = shuffle & flexible, coalesce = no shuffle & reduce [OK]

Hint: Repartition reshuffles; coalesce only reduces partitions [OK]

Common Mistakes:

• Thinking coalesce can increase partitions
• Assuming repartition does not shuffle
• Confusing which method is faster

2. Which of the following is the correct syntax to reduce the number of partitions of a DataFrame df to 5 using coalesce?

easy

A. df.repartition(5)

B. df.coalesce(5)

C. df.coalesce().numPartitions(5)

D. df.reducePartitions(5)

Solution

1. Step 1: Recall coalesce syntax

   The correct method to reduce partitions is df.coalesce(numPartitions).

2. Step 2: Identify correct usage

   df.coalesce(5) uses df.coalesce(5), which correctly reduces partitions to 5.

3. Final Answer:

   df.coalesce(5) -> Option B

4. Quick Check:

   coalesce syntax = df.coalesce(number) [OK]

Hint: Use df.coalesce(number) to reduce partitions [OK]

Common Mistakes:

• Using repartition instead of coalesce to reduce partitions
• Calling non-existent methods like reducePartitions
• Incorrect method chaining syntax

3. Given the following Spark code, what will be the number of partitions after execution?

val df = spark.range(1000).repartition(10)
val df2 = df.coalesce(4)

medium

A. 4 partitions

B. 10 partitions

C. 14 partitions

D. 1000 partitions

Solution

1. Step 1: Analyze initial repartition

   df is repartitioned to 10 partitions explicitly.

2. Step 2: Apply coalesce

   df2 uses coalesce(4) which reduces partitions from 10 to 4 without shuffle.

3. Final Answer:

   4 partitions -> Option A

4. Quick Check:

   repartition to 10, then coalesce to 4 = 4 partitions [OK]

Hint: Coalesce reduces partitions to given number [OK]

Common Mistakes:

• Assuming partitions add up (10 + 4 = 14)
• Thinking coalesce increases partitions
• Confusing initial partition count with final

4. You want to increase the number of partitions of a DataFrame df from 4 to 8. You try df.coalesce(8) but it does not increase partitions. What is the issue?

medium

A. You need to call df.coalesce(8).shuffle() to increase partitions.

B. coalesce requires shuffle to increase partitions.

C. coalesce cannot increase partitions; use repartition instead.

D. The DataFrame must be cached before increasing partitions.

Solution

1. Step 1: Understand coalesce limitations

   coalesce only reduces partitions and does not increase them.

2. Step 2: Choose correct method to increase partitions

   To increase partitions, repartition must be used as it performs a shuffle.

3. Final Answer:

   coalesce cannot increase partitions; use repartition instead. -> Option C

4. Quick Check:

   coalesce only reduces, repartition reshuffles [OK]

Hint: Use repartition to increase partitions, not coalesce [OK]

Common Mistakes:

• Trying to increase partitions with coalesce
• Assuming coalesce triggers shuffle
• Believing caching affects partition count

5. You have a large DataFrame with 100 partitions. You want to reduce partitions to 10 for faster processing but want to avoid a full shuffle to save time. Which method should you use and why?

hard

A. Use df.coalesce(10) because it reduces partitions without a full shuffle.

B. Use df.repartition(10) because it reduces partitions without shuffle.

C. Use df.repartition(10) because it avoids shuffle and reduces partitions.

D. Use df.coalesce(10) because it increases partitions with shuffle.

Solution

1. Step 1: Identify partition reduction without shuffle

   coalesce reduces partitions without a full shuffle, making it faster.

2. Step 2: Compare with repartition

   repartition always causes a full shuffle, which is slower but can increase or decrease partitions.

3. Final Answer:

   Use df.coalesce(10) because it reduces partitions without a full shuffle. -> Option A

4. Quick Check:

   Reduce partitions fast = coalesce without shuffle [OK]

Hint: Coalesce reduces partitions fast without shuffle [OK]

Common Mistakes:

• Using repartition and expecting no shuffle
• Thinking coalesce increases partitions
• Confusing shuffle behavior of methods