Handling data skew at scale in Apache Spark - Mini Project: Build & Apply

Handling data skew at scale

📖 Scenario: You work with a large sales dataset from multiple stores. Some stores have many more sales records than others, causing data skew when you try to join sales data with store information. This skew slows down your Spark job and wastes resources.Your task is to handle this data skew by applying a common technique called salting to balance the data during the join.

🎯 Goal: Build a Spark program that reads two datasets: sales and stores. Then apply salting to the store_id key in the sales dataset to reduce skew before joining with stores. Finally, combine the results and show the total sales per store.

📋 What You'll Learn

Create a Spark DataFrame called sales with given sales records including store_id and amount

Create a Spark DataFrame called stores with given store details including store_id and store_name

Add a salting column called salt to the sales DataFrame with random integers from 0 to 2

Add a corresponding salt column to stores DataFrame by cross joining with a small range

Perform the join on both store_id and salt columns

Aggregate the joined data to get total sales amount per store_name

Print the final aggregated result

💡 Why This Matters

🌍 Real World

Data skew is common in big data when some keys have many more records. Handling skew improves performance and resource use in distributed systems like Spark.

💼 Career

Data engineers and data scientists often need to optimize Spark jobs by handling skewed data to ensure faster and cost-effective processing.

1

Create initial sales and stores DataFrames

Create a Spark DataFrame called sales with these exact rows: (1, 100), (1, 150), (1, 200), (2, 300), (3, 400). The columns should be store_id and amount. Also create a Spark DataFrame called stores with these exact rows: (1, 'Store A'), (2, 'Store B'), (3, 'Store C'). The columns should be store_id and store_name.

Apache Spark

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("DataSkewHandling").getOrCreate()
# Create sales DataFrame with columns store_id and amount
# Create stores DataFrame with columns store_id and store_name
# Your code here

Hint

Use spark.createDataFrame with a list of tuples and specify the column names as a list.

2

Add salting column to sales DataFrame

Create a new column called salt in the sales DataFrame. This column should contain random integers from 0 to 2 (inclusive). Use the Spark SQL function floor(rand() * 3) to generate the salt values.

Apache Spark

from pyspark.sql.functions import floor, rand

# Add a salt column to sales DataFrame with random integers 0 to 2
# Your code here

Hint

Use withColumn with floor(rand() * 3) and cast to integer.

3

Add salt column to stores DataFrame by cross joining

Create a DataFrame called salt_values with a single column salt containing values 0, 1, and 2. Then cross join stores with salt_values to add the salt column to stores. Assign the result back to stores.

Apache Spark

# Create salt_values DataFrame with salt values 0, 1, 2
# Cross join stores with salt_values to add salt column
# Your code here

Hint

Create salt_values with spark.createDataFrame and cross join with stores.

4

Join sales and stores on store_id and salt, then aggregate

Join sales and stores DataFrames on both store_id and salt columns. Then group by store_name and calculate the sum of amount as total_sales. Finally, print the result sorted by store_name.

Apache Spark

# Join sales and stores on store_id and salt
# Group by store_name and sum amount as total_sales
# Print the result sorted by store_name
# Your code here

Hint

Join on both store_id and salt. Use groupBy and agg with sum. Use show() to print.

Practice

1. What is the main problem caused by data skew in Apache Spark jobs?

easy

A. Some tasks become much larger and slower than others

B. All tasks run equally fast

C. Data is lost during processing

D. Spark automatically fixes skew without user action

Solution

1. Step 1: Understand data skew impact

   Data skew means some keys have many more records, causing uneven task sizes.

2. Step 2: Identify effect on tasks

   Tasks processing large keys take longer, slowing the whole job.

3. Final Answer:

   Some tasks become much larger and slower than others -> Option A

4. Quick Check:

   Data skew = uneven task sizes [OK]

Hint: Data skew means uneven task sizes slowing jobs [OK]

Common Mistakes:

• Thinking all tasks run equally fast
• Believing Spark fixes skew automatically
• Confusing data loss with skew

2. Which of the following is the correct way to add a salt key to a DataFrame column named user_id in PySpark?

easy

A. df.withColumn('salted_key', concat(col('user_id'), lit('_'), rand()))

B. df.withColumn('salted_key', col('user_id') + rand())

C. df.withColumn('salted_key', col('user_id').cast('int'))

D. df.withColumn('salted_key', col('user_id').substr(0, 3))

Solution

1. Step 1: Understand salting technique

   Salting adds a random value to the key to spread data across partitions.

2. Step 2: Check correct syntax for salting

   Concatenating user_id with a random number as string is correct.

3. Final Answer:

   df.withColumn('salted_key', concat(col('user_id'), lit('_'), rand())) -> Option A

4. Quick Check:

   Salting = concat key + random [OK]

Hint: Use concat with random value to salt keys [OK]

Common Mistakes:

• Adding random float directly to string key
• Casting key without salting
• Using substring instead of salting

3. Given the following PySpark code snippet to reduce skew by salting, what will be the output count of salted_df if df has 1000 rows?

from pyspark.sql.functions import col, concat, lit, rand
salted_df = df.withColumn('salted_key', concat(col('key'), lit('_'), (rand() * 10).cast('int')))
count = salted_df.count()

medium

A. 0

B. 1000

C. 10

D. 100

Solution

1. Step 1: Understand salting effect on row count

   Salting adds a new column but does not change the number of rows.

2. Step 2: Count rows after salting

   The count remains the same as original DataFrame, 1000 rows.

3. Final Answer:

   1000 -> Option B

4. Quick Check:

   Salting adds column, count unchanged [OK]

Hint: Salting adds column, row count stays same [OK]

Common Mistakes:

• Thinking salting duplicates rows
• Confusing salt range with row count
• Assuming count changes after withColumn

4. You have a join in Spark that is very slow due to data skew on the join key. Which of the following code snippets correctly applies salting to both DataFrames to fix the skew?

df1_salted = df1.withColumn('salt', (rand() * 5).cast('int'))
df2_salted = df2.withColumn('salt', (rand() * 5).cast('int'))
joined = df1_salted.join(df2_salted, ['key', 'salt'])

medium

A. Incorrect: salting with fixed value instead of random

B. Incorrect: salting only one DataFrame

C. Incorrect: joining without salt column

D. Correct: salting both sides and joining on key and salt

Solution

1. Step 1: Understand salting for joins

   To fix skew, salt both DataFrames on join key with same salt range.

2. Step 2: Verify join condition

   Joining on both original key and salt column ensures balanced partitions.

3. Final Answer:

   Correct: salting both sides and joining on key and salt -> Option D

4. Quick Check:

   Salt both sides + join on key & salt [OK]

Hint: Salt both DataFrames and join on key plus salt [OK]

Common Mistakes:

• Salting only one DataFrame
• Joining without salt column
• Using fixed salt value instead of random

5. You have a Spark job with a very large skewed key causing one partition to be huge. You want to reduce skew by salting and also control shuffle partitions. Which approach below best balances load and parallelism?

hard

A. Add salt column but keep shuffle partitions very low

B. Do not add salt, just increase shuffle partitions

C. Add a salt column with random values, join on salted keys, and increase shuffle partitions

D. Add salt column with fixed value and reduce shuffle partitions

Solution

1. Step 1: Apply salting to spread skewed key

   Adding a salt column with random values breaks large groups into smaller ones.

2. Step 2: Adjust shuffle partitions to increase parallelism

   Increasing shuffle partitions allows more tasks to run in parallel, reducing bottlenecks.

3. Step 3: Combine salting and partition tuning

   Both salting and increasing partitions together balance load and improve job speed.

4. Final Answer:

   Add a salt column with random values, join on salted keys, and increase shuffle partitions -> Option C

5. Quick Check:

   Salt + more partitions = balanced load [OK]

Hint: Salt keys and increase shuffle partitions for best balance [OK]

Common Mistakes:

• Keeping shuffle partitions too low after salting
• Not salting and only increasing partitions
• Using fixed salt value causing no skew fix