Transformations vs actions in Apache Spark - Hands-On Comparison

Understanding Transformations vs Actions in Apache Spark

📖 Scenario: Imagine you work at a company that collects sales data from different stores. You want to analyze this data using Apache Spark to find out how many sales were made in total and which stores had sales above a certain number.

🎯 Goal: Build a simple Spark program that creates a dataset of sales, sets a threshold for high sales, applies a transformation to filter stores with sales above the threshold, and then uses an action to count how many such stores there are.

📋 What You'll Learn

Create an RDD with store sales data

Define a sales threshold variable

Use a transformation to filter stores with sales above the threshold

Use an action to count the filtered stores

Print the count result

💡 Why This Matters

🌍 Real World

Companies use Apache Spark to process large datasets efficiently. Understanding transformations and actions helps in writing optimized data processing pipelines.

💼 Career

Data engineers and data scientists use Spark transformations and actions daily to clean, filter, and analyze big data.

1

Create the sales data RDD

Create an RDD called sales_rdd from the list [('StoreA', 150), ('StoreB', 80), ('StoreC', 200), ('StoreD', 50)] using sc.parallelize().

Apache Spark

# Create the sales_rdd from the given list
# Your code here

Hint

Use sc.parallelize() to create an RDD from a Python list.

2

Set the sales threshold

Create a variable called threshold and set it to 100.

Apache Spark

sales_rdd = sc.parallelize([('StoreA', 150), ('StoreB', 80), ('StoreC', 200), ('StoreD', 50)])
# Create the threshold variable and set it to 100
# Your code here

Hint

Just assign the number 100 to a variable named threshold.

3

Filter stores with sales above threshold

Use a transformation to create a new RDD called high_sales_rdd by filtering sales_rdd for stores where the sales value is greater than threshold. Use filter() with a lambda function that checks if the second item in the tuple is greater than threshold.

Apache Spark

sales_rdd = sc.parallelize([('StoreA', 150), ('StoreB', 80), ('StoreC', 200), ('StoreD', 50)])
threshold = 100
# Create high_sales_rdd by filtering sales_rdd for sales > threshold
# Your code here

Hint

Use filter() with a lambda that checks if store[1] is greater than threshold.

4

Count and print the number of high sales stores

Use an action to count the number of elements in high_sales_rdd and print the result using print(). Store the count in a variable called count_high_sales.

Apache Spark

sales_rdd = sc.parallelize([('StoreA', 150), ('StoreB', 80), ('StoreC', 200), ('StoreD', 50)])
threshold = 100
high_sales_rdd = sales_rdd.filter(lambda store: store[1] > threshold)
# Count the elements in high_sales_rdd and print the count
# Your code here

Hint

Use count() on high_sales_rdd and print the result.

Practice

1. Which of the following best describes a transformation in Apache Spark?

easy

A. It creates a new dataset from an existing one without running the job immediately.

B. It runs the job and returns the final result immediately.

C. It saves the dataset to an external storage system.

D. It deletes data from the dataset permanently.

Solution

1. Step 1: Understand what transformations do

   Transformations in Spark create a new dataset from an existing one but do not execute the job immediately.

2. Step 2: Differentiate from actions

   Actions trigger the execution and return results or save data, unlike transformations.

3. Final Answer:

   It creates a new dataset from an existing one without running the job immediately. -> Option A

4. Quick Check:

   Transformation = Lazy evaluation [OK]

Hint: Transformations are lazy; they don't run until an action triggers them [OK]

Common Mistakes:

• Confusing transformations with actions
• Thinking transformations execute immediately
• Believing transformations save or delete data

2. Which of the following is the correct syntax to perform an action that counts the number of elements in an RDD named data?

easy

A. data.map()

B. data.count()

C. data.filter()

D. data.transform()

Solution

1. Step 1: Identify the action to count elements

   The count() method is an action that returns the number of elements in an RDD.

2. Step 2: Check syntax correctness

   Calling data.count() correctly triggers the action and returns the count.

3. Final Answer:

   data.count() -> Option B

4. Quick Check:

   Action to count = data.count() [OK]

Hint: Actions like count() end with parentheses and return results [OK]

Common Mistakes:

• Using transformations like map() as actions
• Calling non-existent methods like transform()
• Forgetting parentheses on actions

3. Given the following Spark code, what will be the output?

rdd = sc.parallelize([1, 2, 3, 4])
mapped = rdd.map(lambda x: x * 2)
result = mapped.collect()

medium

A. An error because collect() is not an action

B. [1, 2, 3, 4]

C. [2, 4, 6, 8]

D. An empty list [] because map is lazy

Solution

1. Step 1: Understand the transformation

   The map transformation multiplies each element by 2 but does not run immediately.

2. Step 2: Understand the action

   The collect() action triggers execution and returns all elements as a list.

3. Final Answer:

   [2, 4, 6, 8] -> Option C

4. Quick Check:

   Transformation + collect() = output list [OK]

Hint: collect() runs the job and returns all data as a list [OK]

Common Mistakes:

• Thinking map() runs immediately
• Believing collect() returns nothing
• Confusing transformations with actions

4. Identify the error in the following Spark code snippet:

rdd = sc.parallelize([10, 20, 30])
filtered = rdd.filter(lambda x: x > 15)
filtered.count

medium

A. filter is not a valid transformation

B. Lambda function syntax is incorrect

C. parallelize is not defined

D. Missing parentheses after count action

Solution

1. Step 1: Check the use of count

   The count method is an action and requires parentheses to execute.

2. Step 2: Identify the error

   Missing parentheses means filtered.count is a method reference, not a call, so no action runs.

3. Final Answer:

   Missing parentheses after count action -> Option D

4. Quick Check:

   Actions need () to run [OK]

Hint: Always add () to call actions like count() [OK]

Common Mistakes:

• Forgetting parentheses on actions
• Thinking filter is an action
• Assuming parallelize is undefined without context

5. You want to find the sum of all even numbers in an RDD numbers. Which sequence of operations correctly uses transformations and actions?

hard

A. Use filter to select even numbers, then reduce to sum them.

B. Use reduce first, then filter to select even numbers.

C. Use map to sum numbers, then collect to filter evens.

D. Use collect first, then filter and reduce on the driver.

Solution

1. Step 1: Apply transformation to filter even numbers

   Use filter transformation to keep only even numbers in the RDD.

2. Step 2: Use action to sum filtered numbers

   Use reduce action to sum all filtered even numbers efficiently on the cluster.

3. Final Answer:

   Use filter to select even numbers, then reduce to sum them. -> Option A

4. Quick Check:

   Transformations then actions = correct Spark pattern [OK]

Hint: Filter first, then reduce to sum -- transformations before actions [OK]

Common Mistakes:

• Trying to reduce before filtering
• Using collect() too early causing driver overload
• Misusing map() for summing