Partition tuning (repartition vs coalesce) in Apache Spark - Visual Side-by-Side Comparison

Concept Flow - Partition tuning (repartition vs coalesce)

Start with DataFrame

↓

Choose partition tuning method

↓

repartition

↓

Shuffle data

↓

New partitions

↓

Balanced partitions

↓

Use tuned DataFrame for processing

Shows the choice between repartition and coalesce, their shuffle behavior, and resulting partitions.

Execution Sample

Apache Spark

df = spark.range(20)
df2 = df.repartition(5)
df3 = df2.coalesce(2)
df3.rdd.getNumPartitions()

Create a DataFrame, repartition to 5 partitions, then coalesce to 2 partitions, and check partition count.

Execution Table

Step	Action	Partitions Before	Partitions After	Shuffle Occurs	Result
1	Create DataFrame with default partitions	N/A	Default (usually 200)	No	DataFrame with default partitions
2	Repartition to 5 partitions	Default (~200)	5	Yes	Data shuffled evenly into 5 partitions
3	Coalesce to 2 partitions	5	2	No or minimal	Partitions merged without full shuffle
4	Check number of partitions	2	2	No	Returns 2 partitions
5	End	2	2	No	Partition tuning complete

💡 Partition tuning stops after coalesce reduces partitions to 2 without shuffle.

Variable Tracker

Variable	Start	After repartition(5)	After coalesce(2)	Final
df	DataFrame with default partitions	Same	Same	Same
df2	N/A	DataFrame with 5 partitions	Same	Same
df3	N/A	N/A	DataFrame with 2 partitions	Same
num_partitions	N/A	N/A	N/A	2

Key Moments - 3 Insights

Why does repartition cause a shuffle but coalesce usually does not?

Repartition reshuffles all data to evenly distribute across new partitions (see step 2 in execution_table). Coalesce merges existing partitions without full shuffle, so it avoids expensive data movement (step 3).

Can coalesce increase the number of partitions?

No, coalesce only reduces partitions or keeps them the same. To increase partitions, repartition must be used (see step 2 vs step 3).

Why might coalesce result in uneven partition sizes?

Because coalesce merges partitions without reshuffling data, some partitions may become larger than others, causing imbalance (explained in concept_flow).

Visual Quiz - 3 Questions

Look at the execution_table at step 2, how many partitions does the DataFrame have after repartition?

A 2

B 5

C 200

D Default number

💡 Hint

Check the 'Partitions After' column at step 2 in the execution_table.

At which step does the shuffle occur according to the execution_table?

A Step 2

B Step 1

C Step 3

D Step 4

💡 Hint

Look at the 'Shuffle Occurs' column in the execution_table.

If you want to increase partitions from 2 to 5, which method should you use?

A coalesce

B Neither, partitions cannot increase

C repartition

D Both coalesce and repartition

💡 Hint

Refer to key_moments about which method increases partitions.

Concept Snapshot

Partition tuning in Spark:
- repartition(n): reshuffles data, creates n balanced partitions
- coalesce(n): merges partitions, reduces to n without full shuffle
- repartition can increase or decrease partitions
- coalesce only reduces partitions
- repartition is costlier but balances data
- coalesce is cheaper but may cause uneven partitions

Full Transcript

This visual execution shows how Spark partition tuning works using repartition and coalesce. Starting with a DataFrame, repartition reshuffles data to create a specified number of balanced partitions, causing a shuffle. Coalesce merges existing partitions to reduce their number without a full shuffle, which is cheaper but can cause uneven partition sizes. The example code creates a DataFrame, repartitions it to 5 partitions, then coalesces to 2 partitions. The execution table traces each step, showing partition counts and shuffle occurrence. Key moments clarify common confusions like why repartition shuffles and coalesce does not, and that coalesce cannot increase partitions. The visual quiz tests understanding of partition counts and method effects. The snapshot summarizes key points for quick recall.

Practice

1. Which statement best describes the difference between repartition and coalesce in Apache Spark?

easy

A. coalesce only increases partitions with shuffle; repartition only decreases partitions without shuffle.

B. coalesce can increase or decrease partitions with shuffle; repartition only decreases partitions without shuffle.

C. repartition only increases partitions without shuffle; coalesce only decreases partitions with shuffle.

D. repartition can increase or decrease partitions with shuffle; coalesce only decreases partitions without shuffle.

Solution

1. Step 1: Understand repartition behavior

   repartition reshuffles data and can increase or decrease the number of partitions, causing a full shuffle.

2. Step 2: Understand coalesce behavior

   coalesce reduces the number of partitions without a full shuffle, making it faster but it cannot increase partitions.

3. Final Answer:

   repartition can increase or decrease partitions with shuffle; coalesce only decreases partitions without shuffle. -> Option D

4. Quick Check:

   repartition = shuffle & flexible, coalesce = no shuffle & reduce [OK]

Hint: Repartition reshuffles; coalesce only reduces partitions [OK]

Common Mistakes:

• Thinking coalesce can increase partitions
• Assuming repartition does not shuffle
• Confusing which method is faster

2. Which of the following is the correct syntax to reduce the number of partitions of a DataFrame df to 5 using coalesce?

easy

A. df.repartition(5)

B. df.coalesce(5)

C. df.coalesce().numPartitions(5)

D. df.reducePartitions(5)

Solution

1. Step 1: Recall coalesce syntax

   The correct method to reduce partitions is df.coalesce(numPartitions).

2. Step 2: Identify correct usage

   df.coalesce(5) uses df.coalesce(5), which correctly reduces partitions to 5.

3. Final Answer:

   df.coalesce(5) -> Option B

4. Quick Check:

   coalesce syntax = df.coalesce(number) [OK]

Hint: Use df.coalesce(number) to reduce partitions [OK]

Common Mistakes:

• Using repartition instead of coalesce to reduce partitions
• Calling non-existent methods like reducePartitions
• Incorrect method chaining syntax

3. Given the following Spark code, what will be the number of partitions after execution?

val df = spark.range(1000).repartition(10)
val df2 = df.coalesce(4)

medium

A. 4 partitions

B. 10 partitions

C. 14 partitions

D. 1000 partitions

Solution

1. Step 1: Analyze initial repartition

   df is repartitioned to 10 partitions explicitly.

2. Step 2: Apply coalesce

   df2 uses coalesce(4) which reduces partitions from 10 to 4 without shuffle.

3. Final Answer:

   4 partitions -> Option A

4. Quick Check:

   repartition to 10, then coalesce to 4 = 4 partitions [OK]

Hint: Coalesce reduces partitions to given number [OK]

Common Mistakes:

• Assuming partitions add up (10 + 4 = 14)
• Thinking coalesce increases partitions
• Confusing initial partition count with final

4. You want to increase the number of partitions of a DataFrame df from 4 to 8. You try df.coalesce(8) but it does not increase partitions. What is the issue?

medium

A. You need to call df.coalesce(8).shuffle() to increase partitions.

B. coalesce requires shuffle to increase partitions.

C. coalesce cannot increase partitions; use repartition instead.

D. The DataFrame must be cached before increasing partitions.

Solution

1. Step 1: Understand coalesce limitations

   coalesce only reduces partitions and does not increase them.

2. Step 2: Choose correct method to increase partitions

   To increase partitions, repartition must be used as it performs a shuffle.

3. Final Answer:

   coalesce cannot increase partitions; use repartition instead. -> Option C

4. Quick Check:

   coalesce only reduces, repartition reshuffles [OK]

Hint: Use repartition to increase partitions, not coalesce [OK]

Common Mistakes:

• Trying to increase partitions with coalesce
• Assuming coalesce triggers shuffle
• Believing caching affects partition count

5. You have a large DataFrame with 100 partitions. You want to reduce partitions to 10 for faster processing but want to avoid a full shuffle to save time. Which method should you use and why?

hard

A. Use df.coalesce(10) because it reduces partitions without a full shuffle.

B. Use df.repartition(10) because it reduces partitions without shuffle.

C. Use df.repartition(10) because it avoids shuffle and reduces partitions.

D. Use df.coalesce(10) because it increases partitions with shuffle.

Solution

1. Step 1: Identify partition reduction without shuffle

   coalesce reduces partitions without a full shuffle, making it faster.

2. Step 2: Compare with repartition

   repartition always causes a full shuffle, which is slower but can increase or decrease partitions.

3. Final Answer:

   Use df.coalesce(10) because it reduces partitions without a full shuffle. -> Option A

4. Quick Check:

   Reduce partitions fast = coalesce without shuffle [OK]

Hint: Coalesce reduces partitions fast without shuffle [OK]

Common Mistakes:

• Using repartition and expecting no shuffle
• Thinking coalesce increases partitions
• Confusing shuffle behavior of methods