Handling skewed joins in Apache Spark - Cheat Sheet & Quick Revision

Recall & Review

beginner

What is a skewed join in Apache Spark?

A skewed join happens when one or more keys in the join have a very large number of records, causing some tasks to take much longer and slow down the whole job.

beginner

Why do skewed joins cause performance problems?

Because the data for some keys is much larger, the tasks handling those keys take longer, causing uneven workload and delays in the join operation.

intermediate

Name one common technique to handle skewed joins in Spark.

One common technique is to use a 'salting' method, which adds a random number to the join key to spread out the large key's data across multiple tasks.

intermediate

What is the 'salting' technique in handling skewed joins?

Salting adds a random number to the join key on both sides of the join, splitting the large key's data into smaller parts to balance the workload.

advanced

How does Spark's built-in skew join optimization work?

Spark detects skewed keys automatically and splits the join into two parts: one for normal keys and one for skewed keys, processing skewed keys separately to improve performance.

What causes a skewed join in Spark?

A One or more keys have many more records than others

B All keys have equal number of records

C Data is sorted before join

D Join keys are missing

Skewed joins happen when some keys have a lot more data, causing uneven task workloads.

Which technique helps to balance data in skewed joins by modifying join keys?

A Broadcasting

B Caching

C Salting

D Filtering

Salting adds a random number to join keys to spread large keys across tasks.

What does Spark do when using built-in skew join optimization?

A Ignores skewed keys

B Processes skewed keys separately

C Drops skewed keys

D Sorts all data

Spark splits the join to handle skewed keys separately for better performance.

Which of these is NOT a way to handle skewed joins?

A Filtering skewed keys

B Broadcast join for small table

C Salting keys

D Ignoring skew

Ignoring skew causes slow joins; handling skew is necessary.

Why is salting done on both sides of the join?

A To keep keys matching after modification

B To increase data size

C To remove duplicates

D To sort data

Salting both sides ensures the join keys still match after adding random numbers.

Explain what a skewed join is and why it causes problems in Spark.

Think about how some keys have much more data than others.

Describe the salting technique and how it helps fix skewed joins.

Imagine adding a small tag to keys to split big groups.