Lazy evaluation in Spark in Apache Spark - Step-by-Step Execution

Concept Flow - Lazy evaluation in Spark

Define transformations

↓

Build DAG (Directed Acyclic Graph)

↓

Trigger action

↓

Execute DAG

↓

Return results

Spark waits to run computations until an action is called, building a plan (DAG) first, then executing it.

Execution Sample

Apache Spark

rdd = sc.parallelize([1, 2, 3, 4])
mapped = rdd.map(lambda x: x * 2)
filtered = mapped.filter(lambda x: x > 4)
result = filtered.collect()

Defines transformations on an RDD but only runs them when collect() action is called.

Execution Table

Step	Operation	Action Triggered?	DAG State	Execution	Output
1	Create RDD from list [1,2,3,4]	No	DAG with 1 node (source)	No execution	No output
2	Map: multiply each element by 2	No	DAG with 2 nodes (source -> map)	No execution	No output
3	Filter: keep elements > 4	No	DAG with 3 nodes (source -> map -> filter)	No execution	No output
4	Collect action called	Yes	DAG ready	Execute all transformations	[6, 8]

💡 Execution happens only at step 4 when collect() triggers the DAG run

Variable Tracker

Variable	Start	After Step 1	After Step 2	After Step 3	After Step 4
rdd	undefined	RDD with data [1, 2, 3, 4]	RDD with data [1, 2, 3, 4]	RDD with data [1, 2, 3, 4]	RDD with data [1, 2, 3, 4]
mapped	undefined	undefined	RDD with map transformation	RDD with map transformation	RDD with map transformation
filtered	undefined	undefined	undefined	RDD with filter transformation	RDD with filter transformation
result	undefined	undefined	undefined	undefined	[6, 8]

Key Moments - 3 Insights

Why don't the transformations run immediately when defined?

Because Spark uses lazy evaluation, it builds a plan (DAG) of transformations but waits to run them until an action like collect() is called (see execution_table step 4).

What triggers the actual computation in Spark?

An action such as collect(), count(), or save triggers execution of all prior transformations in the DAG (see execution_table step 4).

What is the benefit of building a DAG before execution?

It allows Spark to optimize the whole computation plan before running, improving efficiency and reducing unnecessary work.

Visual Quiz - 3 Questions

Look at the execution table, at which step does Spark actually run the transformations?

A Step 3

B Step 4

C Step 2

D Step 1

💡 Hint

Check the 'Action Triggered?' and 'Execution' columns in the execution_table.

According to the variable tracker, what is the value of 'result' before step 4?

A [2, 4, 6, 8]

B [6, 8]

C Undefined

D [1, 2, 3, 4]

💡 Hint

Look at the 'result' row in variable_tracker before step 4.

If we remove the collect() action, what happens to the DAG execution?

A It never runs

B It runs immediately after each transformation

C It runs only for the first transformation

D It runs twice

💡 Hint

Refer to the exit_note and the concept of lazy evaluation in the concept_flow.

Concept Snapshot

Lazy evaluation in Spark means transformations build a plan (DAG) but do not run immediately.
Actions like collect() trigger execution of all transformations.
This allows Spark to optimize and run efficiently.
Transformations are 'lazy'; actions are 'eager'.

Full Transcript

In Spark, when you write code to transform data, Spark does not run those steps right away. Instead, it remembers the steps you want to do and builds a plan called a DAG. This plan shows how data flows through each transformation. Only when you ask for a result with an action like collect(), Spark runs all the steps together. This is called lazy evaluation. It helps Spark run faster by optimizing the whole plan before doing any work. For example, if you create an RDD, map it, and filter it, Spark waits until you call collect() to actually process the data and give you the output.

Practice

1. What does lazy evaluation mean in Apache Spark?

easy

A. Spark runs all transformations immediately when defined

B. Spark delays running transformations until an action is called

C. Spark only runs actions if there are no errors

D. Spark saves all data permanently after each step

Solution

1. Step 1: Understand lazy evaluation concept

   Lazy evaluation means Spark does not run transformations right away but waits.

2. Step 2: Identify when Spark runs computations

   Spark runs the actual work only when an action like collect() or count() is called.

3. Final Answer:

   Spark delays running transformations until an action is called -> Option B

4. Quick Check:

   Lazy evaluation = Delay until action [OK]

Hint: Remember: transformations wait, actions run [OK]

Common Mistakes:

• Thinking transformations run immediately
• Confusing actions with transformations
• Believing Spark saves data after each step

2. Which of the following is the correct way to trigger execution in Spark?

easy

A. df.count()

B. df.select("name")

C. df.filter("age > 30")

D. df.withColumn("new", df["age"] + 1)

Solution

1. Step 1: Identify transformations vs actions

   filter(), select(), and withColumn() are transformations (lazy).

2. Step 2: Recognize actions that trigger execution

   count() is an action that forces Spark to run all prior transformations.

3. Final Answer:

   df.count() -> Option A

4. Quick Check:

   Action = count() triggers execution [OK]

Hint: Actions like count() run Spark jobs [OK]

Common Mistakes:

• Choosing transformations as actions
• Confusing withColumn as action
• Thinking select triggers execution

3. What will be the output of this Spark code snippet?

val data = spark.range(1, 5)
val filtered = data.filter(_ % 2 == 0)
filtered.show()

medium

A. No output, code has error

B. [1, 3, 5]

C. [1, 2, 3, 4]

D. [2, 4]

Solution

1. Step 1: Understand the data range

   spark.range(1, 5) creates numbers 1, 2, 3, 4 (end exclusive).

2. Step 2: Apply filter for even numbers

   filter(_ % 2 == 0) keeps only even numbers: 2 and 4.

3. Final Answer:

   [2, 4] -> Option D

4. Quick Check:

   Filter even numbers = [2, 4] [OK]

Hint: Filter keeps only matching rows, here even numbers [OK]

Common Mistakes:

• Including 5 in range output
• Confusing filter condition
• Assuming inclusive end in range

4. Identify the error in this Spark code:

val df = spark.read.csv("data.csv")
df.filter(col("age") > 30)

medium

A. csv() cannot read files

B. filter() is not a valid method

C. Missing import for col function

D. No error, code runs fine

Solution

1. Step 1: Check usage of col function

   col("age") requires import org.apache.spark.sql.functions.col.

2. Step 2: Verify other code parts

   filter() and csv() are valid methods; no syntax error there.

3. Final Answer:

   Missing import for col function -> Option C

4. Quick Check:

   col() needs import [OK]

Hint: Import col before using it [OK]

Common Mistakes:

• Assuming filter is invalid
• Thinking csv() can't read files
• Ignoring missing imports

5. You want to optimize a Spark job that chains multiple transformations but runs slowly. Which approach uses lazy evaluation to improve performance?

hard

A. Chain all transformations first, then call a single action at the end

B. Use collect() after every transformation to view data

C. Save intermediate results to disk after each transformation

D. Call an action after each transformation to run them step-by-step

Solution

1. Step 1: Understand lazy evaluation benefits

   Lazy evaluation lets Spark optimize by combining transformations before running.

2. Step 2: Identify best practice for chaining

   Chaining all transformations then calling one action lets Spark optimize execution plan.

3. Step 3: Evaluate other options

   Calling actions early or saving intermediate data breaks lazy evaluation and slows job.

4. Final Answer:

   Chain all transformations first, then call a single action at the end -> Option A

5. Quick Check:

   Chain transformations + one action = optimized [OK]

Hint: One action after all transformations optimizes Spark job [OK]

Common Mistakes:

• Calling actions too early
• Saving intermediate data unnecessarily
• Using collect() too often