Lazy evaluation in Spark in Apache Spark - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

Understanding Lazy Evaluation in Spark

Which statement best describes lazy evaluation in Apache Spark?

A Spark delays execution of transformations until an action is called.

B Spark immediately executes all transformations as soon as they are called.

C Spark executes transformations in parallel without waiting for actions.

D Spark caches all data automatically to speed up computations.

💡 Hint

Think about when Spark actually runs the computations.

Lazy evaluation means Spark waits to run transformations until an action triggers execution.

❓ Predict Output

intermediate

Output of Spark Transformations Without Action

What will be the output of the following Spark code snippet?

rdd = sc.parallelize([1, 2, 3, 4])
mapped_rdd = rdd.map(lambda x: x * 2)
print(mapped_rdd.collect())

Apache Spark

rdd = sc.parallelize([1, 2, 3, 4])
mapped_rdd = rdd.map(lambda x: x * 2)
print(mapped_rdd.collect())

A [2, 4, 6, 8]

B SyntaxError due to missing action

C An empty list []

D None

💡 Hint

Consider what triggers execution in Spark.

The collect() action triggers execution, so the doubled values are returned.

❓ data_output

advanced

Number of Jobs Triggered by Actions

Given the following Spark code, how many Spark jobs will be triggered?

rdd = sc.parallelize([1, 2, 3, 4])
mapped = rdd.map(lambda x: x + 1)
filtered = mapped.filter(lambda x: x % 2 == 0)
count = filtered.count()
collected = filtered.collect()

Apache Spark

rdd = sc.parallelize([1, 2, 3, 4])
mapped = rdd.map(lambda x: x + 1)
filtered = mapped.filter(lambda x: x % 2 == 0)
count = filtered.count()
collected = filtered.collect()

A 1

B 2

C 0

D 3

💡 Hint

Each action triggers a job. How many actions are there?

There are two actions: count() and collect(), so two jobs run.

🔧 Debug

advanced

Identifying the Cause of No Output

Why does the following Spark code produce no output?

rdd = sc.parallelize([10, 20, 30])
rdd.map(lambda x: x * 3)

Apache Spark

rdd = sc.parallelize([10, 20, 30])
rdd.map(lambda x: x * 3)

A The RDD is empty so no output is produced.

B The lambda function syntax is incorrect causing silent failure.

C The map transformation is lazy and no action was called to trigger execution.

D Spark requires caching before transformations to produce output.

💡 Hint

Think about what triggers Spark to run transformations.

Transformations like map are lazy and need an action like collect() to run.

🚀 Application

expert

Optimizing Spark Job Execution

You have a Spark job with multiple transformations and two actions on the same RDD. How can you optimize to avoid running the same transformations twice?

A Call <code>collect()</code> after each transformation to save intermediate results.

B Rewrite the transformations as actions to force immediate execution.

C Split the RDD into two separate RDDs to run actions independently.

D Use <code>cache()</code> or <code>persist()</code> on the RDD before the actions to reuse computed data.

💡 Hint

Think about how Spark can reuse data between actions.

Caching or persisting stores the RDD in memory or disk to avoid recomputing transformations for multiple actions.

Practice

1. What does lazy evaluation mean in Apache Spark?

easy

A. Spark runs all transformations immediately when defined

B. Spark delays running transformations until an action is called

C. Spark only runs actions if there are no errors

D. Spark saves all data permanently after each step

Solution

1. Step 1: Understand lazy evaluation concept

   Lazy evaluation means Spark does not run transformations right away but waits.

2. Step 2: Identify when Spark runs computations

   Spark runs the actual work only when an action like collect() or count() is called.

3. Final Answer:

   Spark delays running transformations until an action is called -> Option B

4. Quick Check:

   Lazy evaluation = Delay until action [OK]

Hint: Remember: transformations wait, actions run [OK]

Common Mistakes:

• Thinking transformations run immediately
• Confusing actions with transformations
• Believing Spark saves data after each step

2. Which of the following is the correct way to trigger execution in Spark?

easy

A. df.count()

B. df.select("name")

C. df.filter("age > 30")

D. df.withColumn("new", df["age"] + 1)

Solution

1. Step 1: Identify transformations vs actions

   filter(), select(), and withColumn() are transformations (lazy).

2. Step 2: Recognize actions that trigger execution

   count() is an action that forces Spark to run all prior transformations.

3. Final Answer:

   df.count() -> Option A

4. Quick Check:

   Action = count() triggers execution [OK]

Hint: Actions like count() run Spark jobs [OK]

Common Mistakes:

• Choosing transformations as actions
• Confusing withColumn as action
• Thinking select triggers execution

3. What will be the output of this Spark code snippet?

val data = spark.range(1, 5)
val filtered = data.filter(_ % 2 == 0)
filtered.show()

medium

A. No output, code has error

B. [1, 3, 5]

C. [1, 2, 3, 4]

D. [2, 4]

Solution

1. Step 1: Understand the data range

   spark.range(1, 5) creates numbers 1, 2, 3, 4 (end exclusive).

2. Step 2: Apply filter for even numbers

   filter(_ % 2 == 0) keeps only even numbers: 2 and 4.

3. Final Answer:

   [2, 4] -> Option D

4. Quick Check:

   Filter even numbers = [2, 4] [OK]

Hint: Filter keeps only matching rows, here even numbers [OK]

Common Mistakes:

• Including 5 in range output
• Confusing filter condition
• Assuming inclusive end in range

4. Identify the error in this Spark code:

val df = spark.read.csv("data.csv")
df.filter(col("age") > 30)

medium

A. csv() cannot read files

B. filter() is not a valid method

C. Missing import for col function

D. No error, code runs fine

Solution

1. Step 1: Check usage of col function

   col("age") requires import org.apache.spark.sql.functions.col.

2. Step 2: Verify other code parts

   filter() and csv() are valid methods; no syntax error there.

3. Final Answer:

   Missing import for col function -> Option C

4. Quick Check:

   col() needs import [OK]

Hint: Import col before using it [OK]

Common Mistakes:

• Assuming filter is invalid
• Thinking csv() can't read files
• Ignoring missing imports

5. You want to optimize a Spark job that chains multiple transformations but runs slowly. Which approach uses lazy evaluation to improve performance?

hard

A. Chain all transformations first, then call a single action at the end

B. Use collect() after every transformation to view data

C. Save intermediate results to disk after each transformation

D. Call an action after each transformation to run them step-by-step

Solution

1. Step 1: Understand lazy evaluation benefits

   Lazy evaluation lets Spark optimize by combining transformations before running.

2. Step 2: Identify best practice for chaining

   Chaining all transformations then calling one action lets Spark optimize execution plan.

3. Step 3: Evaluate other options

   Calling actions early or saving intermediate data breaks lazy evaluation and slows job.

4. Final Answer:

   Chain all transformations first, then call a single action at the end -> Option A

5. Quick Check:

   Chain transformations + one action = optimized [OK]

Hint: One action after all transformations optimizes Spark job [OK]

Common Mistakes:

• Calling actions too early
• Saving intermediate data unnecessarily
• Using collect() too often