Accumulator variables in Apache Spark - Cheat Sheet & Quick Revision

Recall & Review

beginner

What is an accumulator variable in Apache Spark?

An accumulator variable is a special variable used to count or sum values across multiple tasks in a distributed Spark job. It helps track information like counts or sums safely from many workers.

beginner

How do accumulator variables behave in Spark tasks?

Accumulator variables can only be added to inside tasks. They do not change the original value in the driver until the job finishes. This prevents errors from multiple updates in parallel.

intermediate

Why should accumulator variables only be used for adding or counting?

Because accumulators are designed to safely add values from many tasks, using them for other operations like subtraction or multiplication can cause incorrect results due to task retries or parallelism.

beginner

Show a simple example of creating and using an accumulator in Spark with Python.

In PySpark, you create an accumulator with sc.accumulator(0). Then inside an RDD operation, you add to it like accum.add(1). After the job, you get the value with accum.value.

intermediate

What happens if you try to read an accumulator's value inside a Spark task?

You cannot reliably read the accumulator's value inside tasks because it is only updated in the driver after the job completes. Reading it inside tasks may give wrong or zero values.

What is the main use of accumulator variables in Spark?

A To shuffle data between nodes

B To store large datasets in memory

C To safely count or sum values across distributed tasks

D To replace broadcast variables

Accumulators are designed to safely aggregate counts or sums from many parallel tasks.

Which operation is safe to perform on accumulators inside Spark tasks?

A Addition

B Subtraction

C Multiplication

D Division

Only addition (or commutative operations like sum) is safe with accumulators to avoid errors from retries.

When can you reliably read the value of an accumulator in Spark?

A Before starting the job

B After the Spark job finishes

C Inside each task

D During task execution

Accumulator values are updated in the driver after job completion, not during task execution.

What happens if a Spark task is retried when using accumulators?

A Accumulator resets automatically

B Task fails immediately

C Accumulator values are ignored

D Accumulator updates may be counted multiple times

Retries can cause accumulator additions to be counted more than once, so only addition is safe.

Which Spark context method is used to create an accumulator in PySpark?

A sc.accumulator()

B sc.broadcast()

C sc.parallelize()

D sc.collect()

The sc.accumulator() method creates an accumulator variable.

Explain what accumulator variables are and why they are useful in Spark.

Think about how you count things safely when many people work in parallel.

Describe best practices and limitations when using accumulator variables in Spark.

Consider what can go wrong if you try to do other math or read values too early.

Practice

1. What is the main purpose of an accumulator variable in Apache Spark?

easy

A. To define the schema of a DataFrame

B. To store large datasets in memory for faster access

C. To cache RDDs for repeated use

D. To count or sum values across multiple nodes during a Spark job

Solution

1. Step 1: Understand accumulator role

   Accumulators are special variables used to aggregate information like counts or sums across many worker nodes.

2. Step 2: Compare with other options

   Options B, C, and D describe caching, storage, or schema tasks, which are unrelated to accumulators.

3. Final Answer:

   To count or sum values across multiple nodes during a Spark job -> Option D

4. Quick Check:

   Accumulator = distributed counting/summing [OK]

Hint: Accumulators add values across nodes during actions [OK]

Common Mistakes:

• Confusing accumulators with broadcast variables
• Thinking accumulators cache data
• Assuming accumulators define data schema

2. Which of the following is the correct way to create an accumulator variable in Spark using Python?

easy

A. accum = sparkContext.accumulate(0)

B. accum = sparkContext.accumulator(0)

C. accum = sparkContext.accumulator()

D. accum = sparkContext.createAccumulator(0)

Solution

1. Step 1: Recall accumulator creation syntax

   The correct method to create an accumulator is sparkContext.accumulator(initial_value).

2. Step 2: Check each option

   accum = sparkContext.accumulator(0) uses the correct method and initial value. Options B, C, and D use incorrect method names or missing parameters.

3. Final Answer:

   accum = sparkContext.accumulator(0) -> Option B

4. Quick Check:

   Use .accumulator(initial_value) to create [OK]

Hint: Use .accumulator() with initial value to create [OK]

Common Mistakes:

• Using .accumulate() instead of .accumulator()
• Omitting the initial value
• Using non-existent methods like createAccumulator

3. Consider the following Spark code snippet:

accum = sc.accumulator(0)
rdd = sc.parallelize([1, 2, 3, 4])
rdd.foreach(lambda x: accum.add(x))
print(accum.value)

What will be the output?

medium

A. Error: accum.add() not allowed in transformations

B. 0

C. 10

D. 4

Solution

1. Step 1: Understand accumulator usage in foreach

   The foreach action applies the lambda to each element, adding each value to the accumulator.

2. Step 2: Calculate the sum of elements

   Sum of [1, 2, 3, 4] is 10, so accum.value will be 10 after foreach completes.

3. Final Answer:

   10 -> Option C

4. Quick Check:

   Sum of list = accumulator value = 10 [OK]

Hint: Accumulator sums values added inside actions like foreach [OK]

Common Mistakes:

• Expecting accumulator to update inside transformations
• Thinking accumulator value stays zero without action
• Confusing accumulator with variable inside map

4. What is wrong with this Spark code snippet?

accum = sc.accumulator(0)
rdd = sc.parallelize([1, 2, 3])
rdd.map(lambda x: accum.add(x))
print(accum.value)

medium

A. Accumulator updates inside map are not guaranteed to run

B. Accumulator must be initialized with a string, not 0

C. Accumulator cannot be printed directly

D. RDD must be cached before using accumulator

Solution

1. Step 1: Identify accumulator usage in map

   map is a transformation and lazy; accumulator updates inside map may not execute unless an action triggers it.

2. Step 2: Understand why accumulator value remains unchanged

   Since no action is called, accum.value remains 0 or unchanged, so this code does not update accumulator as expected.

3. Final Answer:

   Accumulator updates inside map are not guaranteed to run -> Option A

4. Quick Check:

   Accumulator updates must be inside actions [OK]

Hint: Accumulator updates only guaranteed inside actions like foreach [OK]

Common Mistakes:

• Expecting accumulator to update inside lazy transformations
• Initializing accumulator with wrong data type
• Printing accumulator without triggering action

5. You want to count how many numbers in an RDD are greater than 5 using an accumulator. Which code snippet correctly achieves this?

hard

A. accum = sc.accumulator(0)
rdd.filter(lambda x: x > 5).foreach(lambda x: accum.add(1))
print(accum.value)

B. accum = sc.accumulator(0)
rdd.map(lambda x: accum.add(1) if x > 5 else 0)
print(accum.value)

C. accum = sc.accumulator(0)
rdd.reduce(lambda x, y: accum.add(1) if x > 5 else 0)
print(accum.value)

D. accum = sc.accumulator(0)
rdd.foreach(lambda x: accum.add(1) if x > 5 else None)
print(accum.value)

Solution

1. Step 1: Identify correct accumulator update inside action

   accum = sc.accumulator(0)
   rdd.filter(lambda x: x > 5).foreach(lambda x: accum.add(1))
   print(accum.value)

   filters numbers > 5, then uses foreach (an action) to add 1 to accumulator for each such number.

2. Step 2: Check other options for errors

   accum = sc.accumulator(0)
   rdd.foreach(lambda x: accum.add(1) if x > 5 else None)
   print(accum.value)

   uses foreach with correct syntax; the original

   accum = sc.accumulator(0)
   rdd.foreach(lambda x: accum.add(1) if x > 5 else None)
   print(accum.value)

   had a syntax error missing else clause, corrected here;

   accum = sc.accumulator(0)
   rdd.map(lambda x: accum.add(1) if x > 5 else 0)
   print(accum.value)

   uses map (lazy transformation) so accumulator won't update;

   accum = sc.accumulator(0)
   rdd.reduce(lambda x, y: accum.add(1) if x > 5 else 0)
   print(accum.value)

   misuses reduce which is not for side effects.

3. Final Answer:

   accum = sc.accumulator(0) rdd.filter(lambda x: x > 5).foreach(lambda x: accum.add(1)) print(accum.value) -> Option A

4. Quick Check:

   Use filter + foreach to update accumulator correctly [OK]

Hint: Use filter then foreach to update accumulator for condition [OK]

Common Mistakes:

• Updating accumulator inside map or reduce instead of foreach
• Incorrect lambda syntax for conditional add
• Not triggering an action to update accumulator