Accumulator variables in Apache Spark - Mini Project: Build & Apply

Using Accumulator Variables in Apache Spark

📖 Scenario: You work at a retail company analyzing sales data. You want to count how many sales are above a certain amount using Apache Spark.

🎯 Goal: Build a Spark program that uses an accumulator variable to count sales above a threshold.

📋 What You'll Learn

Create an RDD with given sales amounts

Create an accumulator variable to count sales above threshold

Use a Spark action to process the RDD and update the accumulator

Print the final count of sales above the threshold

💡 Why This Matters

🌍 Real World

Counting specific events or conditions in large datasets during distributed processing is common in data science and big data analytics.

💼 Career

Understanding accumulators helps data engineers and data scientists track metrics and debug distributed Spark jobs efficiently.

1

Create the sales data RDD

Create a Spark RDD called sales_rdd from the list [100, 250, 300, 150, 50, 400] using sc.parallelize().

Apache Spark

# Create sales_rdd from the list using sc.parallelize()
# Your code here

Need a hint?

Use sc.parallelize() to create an RDD from a Python list.

2

Create an accumulator variable

Create an accumulator variable called high_sales_count initialized to 0 using sc.accumulator(0).

Apache Spark

sales_rdd = sc.parallelize([100, 250, 300, 150, 50, 400])
# Create accumulator high_sales_count initialized to 0
# Your code here

Need a hint?

Use sc.accumulator(0) to create an accumulator starting at zero.

3

Use accumulator in RDD processing

Use sales_rdd.foreach() with a function that adds 1 to high_sales_count if the sale is greater than 200.

Apache Spark

sales_rdd = sc.parallelize([100, 250, 300, 150, 50, 400])
high_sales_count = sc.accumulator(0)
# Use foreach to add 1 to high_sales_count for sales > 200
# Your code here

Need a hint?

Define a function that checks if sale > 200 and adds 1 to the accumulator. Then call foreach with that function.

4

Print the accumulator result

Print the value of high_sales_count.value to show how many sales were above 200.

Apache Spark

sales_rdd = sc.parallelize([100, 250, 300, 150, 50, 400])
high_sales_count = sc.accumulator(0)

def count_high_sales(sale):
    if sale > 200:
        high_sales_count.add(1)

sales_rdd.foreach(count_high_sales)
# Print the accumulator value
# Your code here

Need a hint?

The accumulator counts sales above 200. Print high_sales_count.value to see the count.