Handling data skew at scale in Apache Spark - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

Output of join with skewed key using salting

Consider two Spark DataFrames df1 and df2 where df1 has a highly skewed key 'user_id'. The code below applies salting to handle skew during join. What is the count of rows in the joined DataFrame joined_df?

Apache Spark

from pyspark.sql import functions as F

# Sample data
skewed_data = [(1, 'A')] * 1000 + [(2, 'B')] * 10
other_data = [(1, 'X')] * 1000 + [(3, 'Y')] * 5

df1 = spark.createDataFrame(skewed_data, ['user_id', 'val1'])
df2 = spark.createDataFrame(other_data, ['user_id', 'val2'])

# Add salt column to df1
salt_range = 10
salted_df1 = df1.withColumn('salt', (F.rand() * salt_range).cast('int'))

# Add salt column to df2 by exploding salt values
salt_values = spark.range(salt_range).withColumnRenamed('id', 'salt')
salted_df2 = df2.crossJoin(salt_values)

# Join on user_id and salt
joined_df = salted_df1.join(salted_df2, ['user_id', 'salt'])

result_count = joined_df.count()

A 2000

B 10000

C 1010

D 1000

💡 Hint

Think about how salting multiplies the number of matching rows for the skewed key.

The skewed key 'user_id' = 1 appears 1000 times in df1 and 1000 times in df2. After salting with 10 salt values, each row in df2 is duplicated 10 times with different salt values. The join matches on both user_id and salt, so for user_id=1, the join produces approximately 1000 rows in df1 distributed across 10 salt values, and 1000 rows in df2 duplicated 10 times with salt values. The expected total join rows for user_id=1 is about 10000. For other keys, the join count is smaller but does not affect the total significantly.

🧠 Conceptual

intermediate

Identifying data skew in Spark jobs

Which of the following is the most reliable sign of data skew causing performance issues in a Spark job?

A All tasks have roughly equal execution times

B The Spark UI shows no shuffle read or write metrics

C The job fails with a memory error on the driver node

D Some tasks take significantly longer than others during shuffle stages

💡 Hint

Think about how skew affects task durations during shuffle.

Data skew causes some partitions to be much larger, making tasks processing them take longer. This results in uneven task durations visible in shuffle stages in Spark UI.

🔧 Debug

advanced

Debugging skew handling with broadcast join

The code below attempts to fix skew by broadcasting the smaller DataFrame. However, it still runs slowly. What is the main reason?

Apache Spark

from pyspark.sql import functions as F

large_df = spark.range(1000000).withColumn('key', (F.rand() * 10).cast('int'))
small_df = spark.range(10).withColumnRenamed('id', 'key')

# Attempt broadcast join
joined = large_df.join(F.broadcast(small_df), 'key')
result = joined.count()

A Broadcast join does not fix skew because large_df has skewed keys causing some partitions to be large

B Broadcast join fails because small_df is too large

C Broadcast join causes shuffle on both sides, increasing runtime

D Broadcast join is not supported on integer keys

💡 Hint

Consider what causes skew and how broadcast join works.

Broadcast join avoids shuffle on the small DataFrame, but if the large DataFrame has skewed keys, some partitions remain large and cause slow tasks.

🚀 Application

advanced

Choosing the best skew mitigation technique

You have a Spark job joining two large DataFrames on a key with heavy skew. The skewed key appears in 50% of the data. Which approach is best to reduce job runtime?

A Use broadcast join on both DataFrames

B Use repartition with a large number of partitions without salting

C Filter out skewed keys before join and process separately

D Use salting on the skewed key and join on salted keys

💡 Hint

Think about isolating skewed keys to handle them differently.

Filtering out skewed keys and processing them separately allows tailored handling, reducing skew impact. Salting helps but is less effective when skewed keys dominate half the data.

❓ visualization

expert

Interpreting Spark UI task duration histogram for skew detection

You see the following histogram of task durations for a shuffle stage in Spark UI: most tasks finish in under 10 seconds, but a few tasks take over 5 minutes. What does this indicate?

A There is data skew causing some partitions to be much larger

B The job has no skew and runs efficiently

C The cluster is under-provisioned causing all tasks to run slowly

D The long tasks are caused by network failures

💡 Hint

Look at the distribution of task durations and what skew causes.

A few very long tasks among many short ones is a classic sign of data skew, where some partitions have much more data.

Practice

1. What is the main problem caused by data skew in Apache Spark jobs?

easy

A. Some tasks become much larger and slower than others

B. All tasks run equally fast

C. Data is lost during processing

D. Spark automatically fixes skew without user action

Solution

1. Step 1: Understand data skew impact

   Data skew means some keys have many more records, causing uneven task sizes.

2. Step 2: Identify effect on tasks

   Tasks processing large keys take longer, slowing the whole job.

3. Final Answer:

   Some tasks become much larger and slower than others -> Option A

4. Quick Check:

   Data skew = uneven task sizes [OK]

Hint: Data skew means uneven task sizes slowing jobs [OK]

Common Mistakes:

• Thinking all tasks run equally fast
• Believing Spark fixes skew automatically
• Confusing data loss with skew

2. Which of the following is the correct way to add a salt key to a DataFrame column named user_id in PySpark?

easy

A. df.withColumn('salted_key', concat(col('user_id'), lit('_'), rand()))

B. df.withColumn('salted_key', col('user_id') + rand())

C. df.withColumn('salted_key', col('user_id').cast('int'))

D. df.withColumn('salted_key', col('user_id').substr(0, 3))

Solution

1. Step 1: Understand salting technique

   Salting adds a random value to the key to spread data across partitions.

2. Step 2: Check correct syntax for salting

   Concatenating user_id with a random number as string is correct.

3. Final Answer:

   df.withColumn('salted_key', concat(col('user_id'), lit('_'), rand())) -> Option A

4. Quick Check:

   Salting = concat key + random [OK]

Hint: Use concat with random value to salt keys [OK]

Common Mistakes:

• Adding random float directly to string key
• Casting key without salting
• Using substring instead of salting

3. Given the following PySpark code snippet to reduce skew by salting, what will be the output count of salted_df if df has 1000 rows?

from pyspark.sql.functions import col, concat, lit, rand
salted_df = df.withColumn('salted_key', concat(col('key'), lit('_'), (rand() * 10).cast('int')))
count = salted_df.count()

medium

A. 0

B. 1000

C. 10

D. 100

Solution

1. Step 1: Understand salting effect on row count

   Salting adds a new column but does not change the number of rows.

2. Step 2: Count rows after salting

   The count remains the same as original DataFrame, 1000 rows.

3. Final Answer:

   1000 -> Option B

4. Quick Check:

   Salting adds column, count unchanged [OK]

Hint: Salting adds column, row count stays same [OK]

Common Mistakes:

• Thinking salting duplicates rows
• Confusing salt range with row count
• Assuming count changes after withColumn

4. You have a join in Spark that is very slow due to data skew on the join key. Which of the following code snippets correctly applies salting to both DataFrames to fix the skew?

df1_salted = df1.withColumn('salt', (rand() * 5).cast('int'))
df2_salted = df2.withColumn('salt', (rand() * 5).cast('int'))
joined = df1_salted.join(df2_salted, ['key', 'salt'])

medium

A. Incorrect: salting with fixed value instead of random

B. Incorrect: salting only one DataFrame

C. Incorrect: joining without salt column

D. Correct: salting both sides and joining on key and salt

Solution

1. Step 1: Understand salting for joins

   To fix skew, salt both DataFrames on join key with same salt range.

2. Step 2: Verify join condition

   Joining on both original key and salt column ensures balanced partitions.

3. Final Answer:

   Correct: salting both sides and joining on key and salt -> Option D

4. Quick Check:

   Salt both sides + join on key & salt [OK]

Hint: Salt both DataFrames and join on key plus salt [OK]

Common Mistakes:

• Salting only one DataFrame
• Joining without salt column
• Using fixed salt value instead of random

5. You have a Spark job with a very large skewed key causing one partition to be huge. You want to reduce skew by salting and also control shuffle partitions. Which approach below best balances load and parallelism?

hard

A. Add salt column but keep shuffle partitions very low

B. Do not add salt, just increase shuffle partitions

C. Add a salt column with random values, join on salted keys, and increase shuffle partitions

D. Add salt column with fixed value and reduce shuffle partitions

Solution

1. Step 1: Apply salting to spread skewed key

   Adding a salt column with random values breaks large groups into smaller ones.

2. Step 2: Adjust shuffle partitions to increase parallelism

   Increasing shuffle partitions allows more tasks to run in parallel, reducing bottlenecks.

3. Step 3: Combine salting and partition tuning

   Both salting and increasing partitions together balance load and improve job speed.

4. Final Answer:

   Add a salt column with random values, join on salted keys, and increase shuffle partitions -> Option C

5. Quick Check:

   Salt + more partitions = balanced load [OK]

Hint: Salt keys and increase shuffle partitions for best balance [OK]

Common Mistakes:

• Keeping shuffle partitions too low after salting
• Not salting and only increasing partitions
• Using fixed salt value causing no skew fix