Handling skewed joins in Apache Spark - Cheat Sheet & Quick Revision

Recall & Review

beginner

What is a skewed join in Apache Spark?

A skewed join happens when one or more keys in the join have a very large number of records, causing some tasks to take much longer and slow down the whole job.

beginner

Why do skewed joins cause performance problems?

Because the data for some keys is much larger, the tasks handling those keys take longer, causing uneven workload and delays in the join operation.

intermediate

Name one common technique to handle skewed joins in Spark.

One common technique is to use a 'salting' method, which adds a random number to the join key to spread out the large key's data across multiple tasks.

intermediate

What is the 'salting' technique in handling skewed joins?

Salting adds a random number to the join key on both sides of the join, splitting the large key's data into smaller parts to balance the workload.

advanced

How does Spark's built-in skew join optimization work?

Spark detects skewed keys automatically and splits the join into two parts: one for normal keys and one for skewed keys, processing skewed keys separately to improve performance.

What causes a skewed join in Spark?

A One or more keys have many more records than others

B All keys have equal number of records

C Data is sorted before join

D Join keys are missing

Skewed joins happen when some keys have a lot more data, causing uneven task workloads.

Which technique helps to balance data in skewed joins by modifying join keys?

A Broadcasting

B Caching

C Salting

D Filtering

Salting adds a random number to join keys to spread large keys across tasks.

What does Spark do when using built-in skew join optimization?

A Ignores skewed keys

B Processes skewed keys separately

C Drops skewed keys

D Sorts all data

Spark splits the join to handle skewed keys separately for better performance.

Which of these is NOT a way to handle skewed joins?

A Filtering skewed keys

B Broadcast join for small table

C Salting keys

D Ignoring skew

Ignoring skew causes slow joins; handling skew is necessary.

Why is salting done on both sides of the join?

A To keep keys matching after modification

B To increase data size

C To remove duplicates

D To sort data

Salting both sides ensures the join keys still match after adding random numbers.

Explain what a skewed join is and why it causes problems in Spark.

Think about how some keys have much more data than others.

Describe the salting technique and how it helps fix skewed joins.

Imagine adding a small tag to keys to split big groups.

Practice

1.

What is the main problem caused by skewed joins in Apache Spark?

easy

A. Spark cannot read the input data format

B. Spark runs out of memory due to too many partitions

C. Some keys have many rows, causing slow join performance

D. Data gets duplicated across all nodes

Solution

1. Step 1: Understand skewed join definition

   Skewed joins happen when some keys have a lot more rows than others, causing uneven data distribution.

2. Step 2: Identify the impact on performance

   This uneven distribution causes some tasks to take much longer, slowing down the entire join operation.

3. Final Answer:

   Some keys have many rows, causing slow join performance -> Option C

4. Quick Check:

   Skewed joins = slow join due to uneven keys [OK]

Hint: Skewed join means uneven key distribution slows join [OK]

Common Mistakes:

• Thinking skewed joins cause data duplication
• Confusing skew with memory errors
• Assuming all keys have equal rows

2.

Which of the following is the correct way to add a salt column for handling skewed joins in Spark using PySpark?

from pyspark.sql.functions import rand, floor

# Add salt column
salted_df = df.withColumn('salt', floor(rand() * 10))

easy

A. salted_df = df.withColumn('salt', floor(rand() * 10))

B. salted_df = df.withColumn('salt', rand() + 10)

C. salted_df = df.withColumn('salt', floor(rand()))

D. salted_df = df.withColumn('salt', rand() * 10)

Solution

1. Step 1: Understand salt column creation

   Salt column should be an integer from 0 to 9 to spread keys evenly.

2. Step 2: Check the code correctness

   Using floor(rand() * 10) creates integers 0-9, which is correct for salting.

3. Final Answer:

   salted_df = df.withColumn('salt', floor(rand() * 10)) -> Option A

4. Quick Check:

   floor(rand()*10) creates salt 0-9 [OK]

Hint: Use floor(rand()*N) for integer salt column [OK]

Common Mistakes:

• Using rand() without floor gives float salt
• Adding 10 instead of multiplying
• Not converting to integer with floor

3.

Given the following PySpark code snippet, what will be the output count of joined_df?

df1 = spark.createDataFrame([(1, 'A'), (1, 'B'), (2, 'C')], ['id', 'val1'])
df2 = spark.createDataFrame([(1, 'X'), (1, 'Y'), (3, 'Z')], ['id', 'val2'])

joined_df = df1.join(df2, 'id')
print(joined_df.count())

medium

A. 2

B. 4

C. 3

D. 5

Solution

1. Step 1: Analyze join keys and rows

   df1 has id=1 twice, df2 has id=1 twice, so join on id=1 produces 2*2=4 rows. id=2 and id=3 do not match both sides.

2. Step 2: Calculate total joined rows

   Only id=1 matches, producing 4 rows in joined_df.

3. Final Answer:

   4 -> Option B

4. Quick Check:

   2x2 matches for id=1 = 4 rows [OK]

Hint: Multiply matching rows per key for join count [OK]

Common Mistakes:

• Counting only unique keys, ignoring duplicates
• Assuming join count equals smaller table size
• Ignoring multiple matches per key

4.

Identify the error in the following PySpark code snippet intended to handle skewed joins by salting:

from pyspark.sql.functions import rand, floor

salted_df1 = df1.withColumn('salt', floor(rand() * 5))
salted_df2 = df2.withColumn('salt', floor(rand() * 5))

joined_df = salted_df1.join(salted_df2, ['id', 'salt'])

medium

A. rand() function is not imported correctly

B. floor(rand() * 5) produces values outside 0-4 range

C. Join should be on 'id' only, not on 'salt'

D. Salt columns are added independently, so keys may not match on salt

Solution

1. Step 1: Understand salting logic

   Salt must be added consistently to both dataframes so matching keys have the same salt value.

2. Step 2: Identify error in salt generation

   Here, salt is generated independently with rand(), so salt values won't match for same keys, causing join to fail.

3. Final Answer:

   Salt columns are added independently, so keys may not match on salt -> Option D

4. Quick Check:

   Independent salt columns cause join mismatch [OK]

Hint: Salt must be consistent on both sides for join [OK]

Common Mistakes:

• Adding salt independently without coordination
• Joining only on 'id' when salting is used
• Misunderstanding rand() output range

5.

You have two large datasets with a highly skewed join key. Which approach best handles the skew to improve join performance?

1. Add a random salt column to the skewed dataset and replicate the smaller dataset across all salt values, then join on both key and salt.
2. Disable broadcast joins to force shuffle join.
3. Use map-side join without salting.
4. Increase executor memory without changing join logic.

hard

A. Add random salt to skewed dataset and replicate smaller dataset across salt values, join on key and salt

B. Disable broadcast joins to force shuffle join

C. Use map-side join without salting

D. Increase executor memory without changing join logic

Solution

1. Step 1: Understand skewed join handling

   Salting spreads skewed keys across partitions by adding a random salt column to the skewed dataset and replicating the smaller dataset across all possible salt values.

2. Step 2: Evaluate options

   Disabling broadcast or increasing memory alone won't fix skew. Map-side join without salting won't handle skew properly.

3. Final Answer:

   Add random salt to skewed dataset and replicate smaller dataset across salt values, join on key and salt -> Option A

4. Quick Check:

   Random salt skewed + replicate small fixes skew [OK]

Hint: Random salt skewed + replicate small fixes skew [OK]

Common Mistakes:

• Disabling broadcast join alone fixes skew
• Not replicating the smaller dataset when salting
• Relying only on memory increase