Lazy evaluation in Spark in Apache Spark - Mini Project: Build & Apply

Lazy evaluation in Spark

📖 Scenario: You work as a data analyst at a retail company. You want to analyze sales data using Apache Spark. Spark uses lazy evaluation, which means it waits to run your commands until it really needs to. This helps save time and resources.

🎯 Goal: Learn how to create a Spark DataFrame, apply transformations, and see when Spark actually runs the work using lazy evaluation.

📋 What You'll Learn

Create a Spark DataFrame with sales data

Define a filter condition as a configuration variable

Apply a filter transformation using lazy evaluation

Trigger the execution by showing the filtered data

💡 Why This Matters

🌍 Real World

Data analysts use lazy evaluation in Spark to write efficient data processing code that only runs when needed, saving time and computing resources.

💼 Career

Understanding lazy evaluation is key for roles like data engineer, data analyst, and data scientist working with big data tools like Apache Spark.

1

Create the sales DataFrame

Create a Spark DataFrame called sales_df with these exact rows: ("Store1", 100), ("Store2", 200), ("Store3", 150). Use columns named store and sales.

Apache Spark

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("LazyEval").getOrCreate()
# Create sales_df DataFrame with given rows and columns
# Your code here

Hint

Use spark.createDataFrame() with a list of tuples and a list of column names.

2

Set the sales threshold

Create a variable called threshold and set it to 150. This will be used to filter stores with sales above this number.

Apache Spark

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("LazyEval").getOrCreate()
sales_data = [("Store1", 100), ("Store2", 200), ("Store3", 150)]
sales_df = spark.createDataFrame(sales_data, ["store", "sales"])
# Create threshold variable with value 150
# Your code here

Hint

Just assign the number 150 to a variable named threshold.

3

Filter the DataFrame using lazy evaluation

Create a new DataFrame called filtered_df by filtering sales_df to keep only rows where sales is greater than threshold. Use the DataFrame filter() method.

Apache Spark

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("LazyEval").getOrCreate()
sales_data = [("Store1", 100), ("Store2", 200), ("Store3", 150)]
sales_df = spark.createDataFrame(sales_data, ["store", "sales"])
threshold = 150
# Filter sales_df where sales > threshold and assign to filtered_df
# Your code here

Hint

Use filtered_df = sales_df.filter(sales_df.sales > threshold).

4

Show the filtered results

Use print() to display the result of filtered_df.show(). This will trigger Spark to run the filter and show the rows with sales above the threshold.

Apache Spark

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("LazyEval").getOrCreate()
sales_data = [("Store1", 100), ("Store2", 200), ("Store3", 150)]
sales_df = spark.createDataFrame(sales_data, ["store", "sales"])
threshold = 150
filtered_df = sales_df.filter(sales_df.sales > threshold)
# Print the filtered DataFrame using filtered_df.show()
# Your code here

Hint

Use filtered_df.show() inside a print() to display the filtered rows.

Practice

1. What does lazy evaluation mean in Apache Spark?

easy

A. Spark runs all transformations immediately when defined

B. Spark delays running transformations until an action is called

C. Spark only runs actions if there are no errors

D. Spark saves all data permanently after each step

Solution

1. Step 1: Understand lazy evaluation concept

   Lazy evaluation means Spark does not run transformations right away but waits.

2. Step 2: Identify when Spark runs computations

   Spark runs the actual work only when an action like collect() or count() is called.

3. Final Answer:

   Spark delays running transformations until an action is called -> Option B

4. Quick Check:

   Lazy evaluation = Delay until action [OK]

Hint: Remember: transformations wait, actions run [OK]

Common Mistakes:

• Thinking transformations run immediately
• Confusing actions with transformations
• Believing Spark saves data after each step

2. Which of the following is the correct way to trigger execution in Spark?

easy

A. df.count()

B. df.select("name")

C. df.filter("age > 30")

D. df.withColumn("new", df["age"] + 1)

Solution

1. Step 1: Identify transformations vs actions

   filter(), select(), and withColumn() are transformations (lazy).

2. Step 2: Recognize actions that trigger execution

   count() is an action that forces Spark to run all prior transformations.

3. Final Answer:

   df.count() -> Option A

4. Quick Check:

   Action = count() triggers execution [OK]

Hint: Actions like count() run Spark jobs [OK]

Common Mistakes:

• Choosing transformations as actions
• Confusing withColumn as action
• Thinking select triggers execution

3. What will be the output of this Spark code snippet?

val data = spark.range(1, 5)
val filtered = data.filter(_ % 2 == 0)
filtered.show()

medium

A. No output, code has error

B. [1, 3, 5]

C. [1, 2, 3, 4]

D. [2, 4]

Solution

1. Step 1: Understand the data range

   spark.range(1, 5) creates numbers 1, 2, 3, 4 (end exclusive).

2. Step 2: Apply filter for even numbers

   filter(_ % 2 == 0) keeps only even numbers: 2 and 4.

3. Final Answer:

   [2, 4] -> Option D

4. Quick Check:

   Filter even numbers = [2, 4] [OK]

Hint: Filter keeps only matching rows, here even numbers [OK]

Common Mistakes:

• Including 5 in range output
• Confusing filter condition
• Assuming inclusive end in range

4. Identify the error in this Spark code:

val df = spark.read.csv("data.csv")
df.filter(col("age") > 30)

medium

A. csv() cannot read files

B. filter() is not a valid method

C. Missing import for col function

D. No error, code runs fine

Solution

1. Step 1: Check usage of col function

   col("age") requires import org.apache.spark.sql.functions.col.

2. Step 2: Verify other code parts

   filter() and csv() are valid methods; no syntax error there.

3. Final Answer:

   Missing import for col function -> Option C

4. Quick Check:

   col() needs import [OK]

Hint: Import col before using it [OK]

Common Mistakes:

• Assuming filter is invalid
• Thinking csv() can't read files
• Ignoring missing imports

5. You want to optimize a Spark job that chains multiple transformations but runs slowly. Which approach uses lazy evaluation to improve performance?

hard

A. Chain all transformations first, then call a single action at the end

B. Use collect() after every transformation to view data

C. Save intermediate results to disk after each transformation

D. Call an action after each transformation to run them step-by-step

Solution

1. Step 1: Understand lazy evaluation benefits

   Lazy evaluation lets Spark optimize by combining transformations before running.

2. Step 2: Identify best practice for chaining

   Chaining all transformations then calling one action lets Spark optimize execution plan.

3. Step 3: Evaluate other options

   Calling actions early or saving intermediate data breaks lazy evaluation and slows job.

4. Final Answer:

   Chain all transformations first, then call a single action at the end -> Option A

5. Quick Check:

   Chain transformations + one action = optimized [OK]

Hint: One action after all transformations optimizes Spark job [OK]

Common Mistakes:

• Calling actions too early
• Saving intermediate data unnecessarily
• Using collect() too often