Broadcast joins for small tables in Apache Spark - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

Output of broadcast join with small table

What is the output count of the following Spark code snippet that uses a broadcast join?

Apache Spark

from pyspark.sql import SparkSession
from pyspark.sql.functions import broadcast

spark = SparkSession.builder.getOrCreate()

# Large DataFrame
df_large = spark.createDataFrame([(1, 'A'), (2, 'B'), (3, 'C')], ['id', 'value'])

# Small DataFrame to broadcast
small_df = spark.createDataFrame([(1, 'X'), (3, 'Y')], ['id', 'desc'])

# Broadcast join
joined_df = df_large.join(broadcast(small_df), 'id', 'inner')

print(joined_df.count())

A 3

B 1

C 0

D 2

💡 Hint

Think about how many matching keys exist in both DataFrames.

The join is an inner join on 'id'. The small table has ids 1 and 3, which match two rows in the large table. So the result has 2 rows.

❓ data_output

intermediate

Resulting DataFrame after broadcast join

What is the content of the DataFrame after this broadcast join?

Apache Spark

from pyspark.sql import SparkSession
from pyspark.sql.functions import broadcast

spark = SparkSession.builder.getOrCreate()

df1 = spark.createDataFrame([(10, 'apple'), (20, 'banana'), (30, 'cherry')], ['key', 'fruit'])
small_df = spark.createDataFrame([(10, 'red'), (30, 'dark red')], ['key', 'color'])

result = df1.join(broadcast(small_df), 'key', 'left')
result.show()

A [Row(key=10, fruit='apple', color='red'), Row(key=20, fruit='banana', color='yellow'), Row(key=30, fruit='cherry', color='dark red')]

B [Row(key=10, fruit='apple', color=None), Row(key=20, fruit='banana', color=None), Row(key=30, fruit='cherry', color=None)]

C [Row(key=10, fruit='apple', color='red'), Row(key=20, fruit='banana', color=None), Row(key=30, fruit='cherry', color='dark red')]

D [Row(key=10, fruit='apple', color='red'), Row(key=20, fruit='banana', color='red'), Row(key=30, fruit='cherry', color='dark red')]

💡 Hint

Remember that a left join keeps all rows from the left DataFrame and adds matching rows from the right.

The left join keeps all keys from df1. Keys 10 and 30 have matching colors, key 20 has no match so color is None.

🔧 Debug

advanced

Identify the error in broadcast join code

What error will this Spark code raise when executed?

Apache Spark

from pyspark.sql import SparkSession
from pyspark.sql.functions import broadcast

spark = SparkSession.builder.getOrCreate()

large_df = spark.createDataFrame([(1, 'A'), (2, 'B')], ['id', 'val'])
small_df = spark.createDataFrame([(1, 'X'), (2, 'Y')], ['key', 'desc'])

joined = large_df.join(broadcast(small_df), large_df.id == small_df.key)
joined.show()

A No error, outputs joined DataFrame with 2 rows

B AnalysisException: 'Reference 'key' is ambiguous, could be: key, key.'

C TypeError: join condition must be a Column

D ValueError: Cannot broadcast a DataFrame with more than 100MB

💡 Hint

Check if the join columns have the same name or if they are ambiguous.

The join condition uses columns with different names but the result has duplicate column names causing ambiguity in output.

🚀 Application

advanced

Choosing broadcast join for performance

You have a large DataFrame with 10 million rows and a small DataFrame with 100 rows. Which Spark join strategy is best for performance?

A Use cartesian join to combine all rows from both DataFrames

B Use broadcast join to send the small DataFrame to all worker nodes

C Use shuffle join to redistribute both DataFrames across the cluster

D Use cross join with filter to reduce data size

💡 Hint

Think about minimizing data movement for small tables.

Broadcast join sends the small DataFrame to all nodes, avoiding expensive shuffles and improving performance.

🧠 Conceptual

expert

Effect of broadcast join on shuffle partitions

When using a broadcast join in Spark, what happens to the shuffle partitions compared to a regular shuffle join?

A Shuffle partitions are eliminated for the broadcasted table, reducing shuffle overhead

B Shuffle partitions increase because data is duplicated across nodes

C Shuffle partitions remain the same as in a regular shuffle join

D Shuffle partitions are replaced by a cartesian product

💡 Hint

Consider how broadcast join avoids shuffling the small table.

Broadcast join sends the small table to all executors, so no shuffle is needed for it, reducing shuffle partitions and overhead.