Partition tuning (repartition vs coalesce) in Apache Spark - Hands-On Comparison

Partition tuning with repartition vs coalesce in Apache Spark

📖 Scenario: You work with a large dataset of sales records. You want to optimize how Spark processes this data by changing the number of partitions. This helps Spark run faster and use resources better.

🎯 Goal: Learn how to change the number of partitions in a Spark DataFrame using repartition() and coalesce(). See how these methods affect the data partitions.

📋 What You'll Learn

Create a Spark DataFrame with sample sales data

Set a target number of partitions

Use repartition() to increase partitions

Use coalesce() to decrease partitions

Print the number of partitions after each operation

💡 Why This Matters

🌍 Real World

Data engineers and data scientists often tune partitions in Spark to improve job speed and resource use when processing big data.

💼 Career

Knowing how to repartition and coalesce data is important for optimizing Spark jobs in roles like data engineering, big data analytics, and machine learning pipelines.

1

Create a Spark DataFrame with sales data

Create a Spark DataFrame called sales_df with these exact rows: ("2023-01-01", "StoreA", 100), ("2023-01-02", "StoreB", 150), ("2023-01-03", "StoreA", 200), ("2023-01-04", "StoreC", 50). Use columns named date, store, and sales.

Apache Spark

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("PartitionTuning").getOrCreate()

# Create sales_df DataFrame with the exact rows and columns
# Your code here

Hint

Use spark.createDataFrame() with a list of tuples and a list of column names.

2

Set a target number of partitions

Create a variable called target_partitions and set it to 6. This will be the number of partitions we want to use.

Apache Spark

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("PartitionTuning").getOrCreate()

sales_data = [("2023-01-01", "StoreA", 100), ("2023-01-02", "StoreB", 150), ("2023-01-03", "StoreA", 200), ("2023-01-04", "StoreC", 50)]
sales_df = spark.createDataFrame(sales_data, ["date", "store", "sales"])

# Create target_partitions variable and set to 6
# Your code here

Hint

Just create a variable named target_partitions and assign the number 6.

3

Use repartition() and coalesce() to change partitions

Create a new DataFrame called repartitioned_df by calling sales_df.repartition(target_partitions). Then create another DataFrame called coalesced_df by calling repartitioned_df.coalesce(2).

Apache Spark

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("PartitionTuning").getOrCreate()

sales_data = [("2023-01-01", "StoreA", 100), ("2023-01-02", "StoreB", 150), ("2023-01-03", "StoreA", 200), ("2023-01-04", "StoreC", 50)]
sales_df = spark.createDataFrame(sales_data, ["date", "store", "sales"])

target_partitions = 6

# Use repartition() to create repartitioned_df with target_partitions
# Use coalesce() on repartitioned_df to create coalesced_df with 2 partitions
# Your code here

Hint

Use repartition() to increase partitions and coalesce() to reduce partitions.

4

Print the number of partitions after repartition and coalesce

Print the number of partitions in sales_df, repartitioned_df, and coalesced_df by using .rdd.getNumPartitions(). Use three separate print statements with the exact text: "Original partitions: ", "After repartition: ", and "After coalesce: " followed by the number.

Apache Spark

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("PartitionTuning").getOrCreate()

sales_data = [("2023-01-01", "StoreA", 100), ("2023-01-02", "StoreB", 150), ("2023-01-03", "StoreA", 200), ("2023-01-04", "StoreC", 50)]
sales_df = spark.createDataFrame(sales_data, ["date", "store", "sales"])

target_partitions = 6

repartitioned_df = sales_df.repartition(target_partitions)
coalesced_df = repartitioned_df.coalesce(2)

# Print the number of partitions for sales_df, repartitioned_df, and coalesced_df
# Your code here

Hint

Use print() and .rdd.getNumPartitions() on each DataFrame.

Practice

1. Which statement best describes the difference between repartition and coalesce in Apache Spark?

easy

A. coalesce only increases partitions with shuffle; repartition only decreases partitions without shuffle.

B. coalesce can increase or decrease partitions with shuffle; repartition only decreases partitions without shuffle.

C. repartition only increases partitions without shuffle; coalesce only decreases partitions with shuffle.

D. repartition can increase or decrease partitions with shuffle; coalesce only decreases partitions without shuffle.

Solution

1. Step 1: Understand repartition behavior

   repartition reshuffles data and can increase or decrease the number of partitions, causing a full shuffle.

2. Step 2: Understand coalesce behavior

   coalesce reduces the number of partitions without a full shuffle, making it faster but it cannot increase partitions.

3. Final Answer:

   repartition can increase or decrease partitions with shuffle; coalesce only decreases partitions without shuffle. -> Option D

4. Quick Check:

   repartition = shuffle & flexible, coalesce = no shuffle & reduce [OK]

Hint: Repartition reshuffles; coalesce only reduces partitions [OK]

Common Mistakes:

• Thinking coalesce can increase partitions
• Assuming repartition does not shuffle
• Confusing which method is faster

2. Which of the following is the correct syntax to reduce the number of partitions of a DataFrame df to 5 using coalesce?

easy

A. df.repartition(5)

B. df.coalesce(5)

C. df.coalesce().numPartitions(5)

D. df.reducePartitions(5)

Solution

1. Step 1: Recall coalesce syntax

   The correct method to reduce partitions is df.coalesce(numPartitions).

2. Step 2: Identify correct usage

   df.coalesce(5) uses df.coalesce(5), which correctly reduces partitions to 5.

3. Final Answer:

   df.coalesce(5) -> Option B

4. Quick Check:

   coalesce syntax = df.coalesce(number) [OK]

Hint: Use df.coalesce(number) to reduce partitions [OK]

Common Mistakes:

• Using repartition instead of coalesce to reduce partitions
• Calling non-existent methods like reducePartitions
• Incorrect method chaining syntax

3. Given the following Spark code, what will be the number of partitions after execution?

val df = spark.range(1000).repartition(10)
val df2 = df.coalesce(4)

medium

A. 4 partitions

B. 10 partitions

C. 14 partitions

D. 1000 partitions

Solution

1. Step 1: Analyze initial repartition

   df is repartitioned to 10 partitions explicitly.

2. Step 2: Apply coalesce

   df2 uses coalesce(4) which reduces partitions from 10 to 4 without shuffle.

3. Final Answer:

   4 partitions -> Option A

4. Quick Check:

   repartition to 10, then coalesce to 4 = 4 partitions [OK]

Hint: Coalesce reduces partitions to given number [OK]

Common Mistakes:

• Assuming partitions add up (10 + 4 = 14)
• Thinking coalesce increases partitions
• Confusing initial partition count with final

4. You want to increase the number of partitions of a DataFrame df from 4 to 8. You try df.coalesce(8) but it does not increase partitions. What is the issue?

medium

A. You need to call df.coalesce(8).shuffle() to increase partitions.

B. coalesce requires shuffle to increase partitions.

C. coalesce cannot increase partitions; use repartition instead.

D. The DataFrame must be cached before increasing partitions.

Solution

1. Step 1: Understand coalesce limitations

   coalesce only reduces partitions and does not increase them.

2. Step 2: Choose correct method to increase partitions

   To increase partitions, repartition must be used as it performs a shuffle.

3. Final Answer:

   coalesce cannot increase partitions; use repartition instead. -> Option C

4. Quick Check:

   coalesce only reduces, repartition reshuffles [OK]

Hint: Use repartition to increase partitions, not coalesce [OK]

Common Mistakes:

• Trying to increase partitions with coalesce
• Assuming coalesce triggers shuffle
• Believing caching affects partition count

5. You have a large DataFrame with 100 partitions. You want to reduce partitions to 10 for faster processing but want to avoid a full shuffle to save time. Which method should you use and why?

hard

A. Use df.coalesce(10) because it reduces partitions without a full shuffle.

B. Use df.repartition(10) because it reduces partitions without shuffle.

C. Use df.repartition(10) because it avoids shuffle and reduces partitions.

D. Use df.coalesce(10) because it increases partitions with shuffle.

Solution

1. Step 1: Identify partition reduction without shuffle

   coalesce reduces partitions without a full shuffle, making it faster.

2. Step 2: Compare with repartition

   repartition always causes a full shuffle, which is slower but can increase or decrease partitions.

3. Final Answer:

   Use df.coalesce(10) because it reduces partitions without a full shuffle. -> Option A

4. Quick Check:

   Reduce partitions fast = coalesce without shuffle [OK]

Hint: Coalesce reduces partitions fast without shuffle [OK]

Common Mistakes:

• Using repartition and expecting no shuffle
• Thinking coalesce increases partitions
• Confusing shuffle behavior of methods