Cross joins and when to avoid them in Apache Spark - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

Output of a simple cross join in Spark

What is the output of the following Spark code snippet?

Apache Spark

from pyspark.sql import SparkSession
spark = SparkSession.builder.getOrCreate()
data1 = [(1, "A"), (2, "B")]
data2 = [("X", 10), ("Y", 20)]
df1 = spark.createDataFrame(data1, ["id", "val1"])
df2 = spark.createDataFrame(data2, ["val2", "num"])
cross_joined = df1.crossJoin(df2)
cross_joined.show()

A [Row(id=1, val1='A', val2='X', num=10), Row(id=1, val1='A', val2='Y', num=20), Row(id=2, val1='B', val2='X', num=10), Row(id=2, val1='B', val2='Y', num=20)]

B [Row(id=1, val1='A', val2='X', num=10), Row(id=2, val1='B', val2='Y', num=20)]

C [Row(id=1, val1='A', val2='X', num=10), Row(id=1, val1='A', val2='Y', num=20)]

D SyntaxError

💡 Hint

Remember that a cross join pairs every row of the first DataFrame with every row of the second DataFrame.

A cross join returns the Cartesian product of the two DataFrames. Since df1 has 2 rows and df2 has 2 rows, the result has 2*2=4 rows combining all pairs.

🧠 Conceptual

intermediate

When to avoid cross joins in Spark

Which of the following is the best reason to avoid cross joins in Spark?

A Cross joins are deprecated and not supported in Spark.

B Cross joins always produce empty results, so they are useless.

C Cross joins automatically filter rows, which can cause data loss.

D Cross joins can produce very large datasets that consume excessive memory and slow down processing.

💡 Hint

Think about what happens when you combine every row with every other row.

Cross joins create the Cartesian product, which can explode the number of rows and cause performance and memory issues.

🔧 Debug

advanced

Identify the error in cross join usage

What error will this Spark code raise?

Apache Spark

from pyspark.sql import SparkSession
spark = SparkSession.builder.getOrCreate()
data1 = [(1, "A")]
data2 = [("X", 10)]
df1 = spark.createDataFrame(data1, ["id", "val1"])
df2 = spark.createDataFrame(data2, ["val2", "num"])
result = df1.join(df2)
result.show()

A No error, outputs the cross join result

B AnalysisException: 'Detected cartesian product for INNER join, use CROSS JOIN if intended.'

C TypeError: join() missing required argument

D SyntaxError

💡 Hint

Check the join method usage without join condition.

Spark requires explicit crossJoin() for cross joins. Using join() without condition causes an AnalysisException.

❓ data_output

advanced

Number of rows after cross join

Given two DataFrames df1 with 3 rows and df2 with 4 rows, how many rows will the DataFrame have after a cross join?

A 12

B 1

C 7

D 0

💡 Hint

Multiply the number of rows in each DataFrame.

Cross join produces the Cartesian product, so total rows = rows in df1 * rows in df2 = 3 * 4 = 12.

🚀 Application

expert

Avoiding cross join explosion in Spark

You have two large DataFrames and need to join them without a common key. Which approach best avoids the performance problems of a cross join?

A Use join() without condition to let Spark optimize automatically.

B Use crossJoin() directly to get all combinations.

C Use broadcast join to send the smaller DataFrame to all nodes and join on a condition.

D Convert both DataFrames to Pandas and join locally.

💡 Hint

Think about how to reduce data shuffle and avoid Cartesian explosion.

Broadcast join sends the smaller DataFrame to all nodes, allowing efficient join without cross join explosion.