Understanding the Catalyst optimizer in Apache Spark - Practice Questions & Exercises

Challenge - 5 Problems

🧠 Conceptual

intermediate

How Catalyst Optimizer improves query execution

Which of the following best describes the main role of the Catalyst optimizer in Apache Spark?

A It manages cluster resources and schedules tasks across nodes.

B It compiles Spark code into machine language for faster execution.

C It transforms and optimizes logical query plans to improve execution efficiency.

D It stores data in a distributed file system for fault tolerance.

💡 Hint

Think about what happens between writing a query and running it efficiently.

The Catalyst optimizer takes the logical plan of a query and applies rules to optimize it before execution, improving performance.

❓ Predict Output

intermediate

Output of optimized query plan

Given the following Spark SQL code, what will be the output of df.explain(true) regarding the optimization?

Apache Spark

val df = spark.read.json("people.json")
val filtered = df.filter("age > 21")
filtered.explain(true)

A Prints the data content of the DataFrame instead of the plan.

B Shows only the raw logical plan without any optimization steps.

C Throws an error because explain(true) is not valid syntax.

D Shows a physical plan with a Filter node pushed down to data source if supported.

💡 Hint

Explain(true) shows detailed plans including optimized physical plans.

The Catalyst optimizer pushes filters down to the data source when possible, which is visible in the physical plan output by explain(true).

❓ data_output

advanced

Result of query after Catalyst optimization

Consider a DataFrame df with columns name and age. After applying df.filter("age > 30").select("name"), what will be the schema of the resulting DataFrame?

A Both <code>name</code> and <code>age</code> columns will be present.

B Only the <code>name</code> column with string type will be present.

C Only the <code>age</code> column will be present.

D The DataFrame will be empty with no columns.

💡 Hint

Think about what columns are selected after filtering.

The filter keeps rows where age > 30, but the select keeps only the name column, so the resulting schema has only name.

🔧 Debug

advanced

Identifying error in Catalyst optimization stage

What error will occur if a user tries to filter a DataFrame using a non-existent column like df.filter("salary > 50000") when salary does not exist?

A AnalysisException indicating unresolved attribute 'salary'.

B NullPointerException during query execution.

C No error; the filter will be ignored silently.

D SyntaxError due to invalid filter syntax.

💡 Hint

Think about how Catalyst checks column names before running queries.

Catalyst performs analysis and throws an AnalysisException if a referenced column does not exist in the schema.

🚀 Application

expert

Effect of Catalyst optimizer on join order

Given two DataFrames df1 and df2, both large, what does the Catalyst optimizer do when you write df1.join(df2, "id")?

Choose the most accurate description of Catalyst's behavior regarding join order and optimization.

A Catalyst automatically reorders joins to minimize data shuffling and improve performance.

B Catalyst executes joins in the exact order written without any changes.

C Catalyst converts all joins to broadcast joins regardless of size.

D Catalyst disables join optimization if DataFrames are large.

💡 Hint

Think about how query optimizers improve join performance.

Catalyst uses cost-based optimization to reorder joins and choose join strategies that reduce data movement and improve speed.