Avoiding shuffle operations in Apache Spark - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

Output of join without shuffle

Consider two Spark DataFrames df1 and df2, both partitioned by the same column 'id'. What will be the output count of the following join operation?

Apache Spark

df1 = spark.createDataFrame([(1, 'a'), (2, 'b'), (3, 'c')], ['id', 'val1']).repartition('id')
df2 = spark.createDataFrame([(1, 'x'), (2, 'y'), (4, 'z')], ['id', 'val2']).repartition('id')
joined = df1.join(df2, 'id')
result_count = joined.count()
print(result_count)

A 2

B 3

C 1

D 4

💡 Hint

Think about which ids are common in both DataFrames.

The join matches rows with the same 'id'. Only ids 1 and 2 are common, so the result has 2 rows.

🧠 Conceptual

intermediate

Why avoid shuffle in Spark?

Why is it important to avoid shuffle operations in Apache Spark when possible?

A Shuffle operations increase the number of partitions automatically, which is always bad.

B Shuffle operations are expensive because they involve disk and network I/O, slowing down the job.

C Shuffle operations cause Spark to lose data, leading to incorrect results.

D Shuffle operations prevent Spark from caching data in memory.

💡 Hint

Think about what happens behind the scenes during a shuffle.

Shuffle moves data across the cluster, involving disk writes and network transfer, which slows down processing.

🔧 Debug

advanced

Identify shuffle causing operation

Given the following Spark code, which line causes a shuffle operation?

Apache Spark

df = spark.createDataFrame([(1, 'a'), (2, 'b'), (3, 'c')], ['id', 'val'])
df2 = df.filter(df.id > 1)
df3 = df2.groupBy('val').count()
df4 = df3.orderBy('count')
df4.show()

A df3 = df2.groupBy('val').count()

B df2 = df.filter(df.id > 1)

C df4 = df3.orderBy('count')

D df4.show()

💡 Hint

GroupBy operations usually cause shuffle.

groupBy triggers shuffle to group data by keys across partitions.

🚀 Application

advanced

Avoid shuffle in join by partitioning

You have two large DataFrames df1 and df2 both partitioned by 'user_id'. You want to join them on 'user_id' without causing shuffle. Which approach is correct?

A Sort both DataFrames by 'user_id' before join.

B Use df1.join(df2, 'user_id') directly without repartitioning.

C Collect df2 to driver and broadcast join with df1.

D Repartition both DataFrames by 'user_id' before join.

💡 Hint

Matching partitioning keys avoids shuffle in join.

Repartitioning both DataFrames by the join key ensures data is co-located, avoiding shuffle during join.

❓ visualization

expert

Visualize shuffle stages in Spark UI

You run a Spark job with multiple transformations including groupBy and join. Which visualization in Spark UI helps you identify shuffle stages and their cost?

A The Executors tab showing memory usage.

B The SQL tab showing executed queries.

C The DAG visualization showing stages and shuffle dependencies.

D The Storage tab showing cached RDDs.

💡 Hint

Look for visual representation of stages and data movement.

The DAG visualization shows stages and shuffle dependencies, helping identify costly shuffle operations.