Reduce and aggregate actions in Apache Spark - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

What is the output of this Spark reduce operation?

Given the following Spark code, what will be the output printed?

Apache Spark

from pyspark.sql import SparkSession
spark = SparkSession.builder.master('local').appName('Test').getOrCreate()
rdd = spark.sparkContext.parallelize([1, 2, 3, 4])
result = rdd.reduce(lambda x, y: x + y)
print(result)

A TypeError

B 24

C 10

D None

💡 Hint

Think about what reduce does with the lambda function summing two numbers.

The reduce action applies the lambda function cumulatively to the elements of the RDD, summing all numbers: 1+2+3+4 = 10.

❓ data_output

intermediate

What is the result of this aggregate action?

Consider this Spark code using aggregate to compute sum and count. What is the final output?

Apache Spark

from pyspark.sql import SparkSession
spark = SparkSession.builder.master('local').appName('Test').getOrCreate()
rdd = spark.sparkContext.parallelize([1, 2, 3, 4])
zero_value = (0, 0)
seq_op = lambda acc, x: (acc[0] + x, acc[1] + 1)
comb_op = lambda acc1, acc2: (acc1[0] + acc2[0], acc1[1] + acc2[1])
result = rdd.aggregate(zero_value, seq_op, comb_op)
print(result)

A (10, 4)

B (24, 4)

C TypeError

D (10, 10)

💡 Hint

Aggregate returns a combined result of sum and count.

The aggregate sums all elements (10) and counts them (4), returning (10, 4).

❓ Predict Output

advanced

What is the output of this Spark reduceByKey operation?

Given the following Spark code, what will be the output printed?

Apache Spark

from pyspark.sql import SparkSession
spark = SparkSession.builder.master('local').appName('Test').getOrCreate()
rdd = spark.sparkContext.parallelize([('a', 1), ('b', 2), ('a', 3)])
result = rdd.reduceByKey(lambda x, y: x + y).collect()
print(result)

A [('a', 4), ('b', 2)]

B AttributeError

C TypeError

D ValueError

💡 Hint

reduceByKey sums values for each key.

reduceByKey sums values for each key: 'a' -> 1+3=4, 'b' -> 2.

❓ visualization

advanced

Visualize the result of countByValue action

Given this Spark code, what is the output of countByValue?

Apache Spark

from pyspark.sql import SparkSession
spark = SparkSession.builder.master('local').appName('Test').getOrCreate()
rdd = spark.sparkContext.parallelize(['apple', 'banana', 'apple', 'orange', 'banana', 'apple'])
result = rdd.countByValue()
print(result)

A {'apple': 1, 'banana': 1, 'orange': 1}

B {'apple': 2, 'banana': 2, 'orange': 2}

C TypeError

D {'apple': 3, 'banana': 2, 'orange': 1}

💡 Hint

countByValue counts how many times each value appears.

The countByValue returns a dictionary with counts of each fruit: apple 3, banana 2, orange 1.

🧠 Conceptual

expert

Which Spark action returns a single value by combining all elements?

Among these Spark actions, which one returns a single value by combining all elements of an RDD using a function?

A take

B reduce

C countByKey

D collect

💡 Hint

Think about which action combines all elements into one result.

The reduce action combines all elements using a function and returns a single value.

Practice

1.

What does the reduce action do in Apache Spark?

easy

A. It filters elements based on a condition.

B. It creates a new RDD by mapping each element.

C. It combines all elements of an RDD using a specified function to produce a single result.

D. It sorts the elements in the RDD.

Solution

1. Step 1: Understand the purpose of reduce

   The reduce action combines all elements of an RDD by applying a function that takes two elements and returns one, repeating until one result remains.

2. Step 2: Compare with other actions

   Filtering selects elements, mapping transforms elements, and sorting orders elements, none of which combine all elements into one.

3. Final Answer:

   It combines all elements of an RDD using a specified function to produce a single result. -> Option C

4. Quick Check:

   Reduce = combine all elements [OK]

Hint: Reduce combines all data into one value using a function [OK]

Common Mistakes:

• Confusing reduce with filter or map actions
• Thinking reduce sorts data
• Assuming reduce returns an RDD instead of a single value

2.

Which of the following is the correct syntax to use reduce on an RDD named rdd to sum all elements?

rdd.reduce(_____)

easy

A. lambda x: x * 2

B. lambda x, y: x + y

C. lambda x, y: x * y

D. lambda x: x + 1

Solution

1. Step 1: Identify the function signature for reduce

   The function passed to reduce must take two arguments and return one combined result.

2. Step 2: Choose the function that sums two elements

   Only lambda x, y: x + y takes two arguments and sums them, suitable for summing all elements.

3. Final Answer:

   lambda x, y: x + y -> Option B

4. Quick Check:

   Reduce needs two-arg function for combining [OK]

Hint: Reduce function must take two inputs and combine them [OK]

Common Mistakes:

• Using single-argument lambda instead of two-argument
• Using multiplication when sum is needed
• Passing a function that does not return a single value

3.

Given the following code, what is the output?

rdd = sc.parallelize([1, 2, 3, 4])
result = rdd.aggregate(0, (lambda acc, x: acc + x), (lambda acc1, acc2: acc1 + acc2))
print(result)

medium

A. 24

B. 0

C. Error

D. 10

Solution

1. Step 1: Understand aggregate parameters

   The initial value is 0. The first lambda adds each element to the accumulator. The second lambda combines accumulators from partitions by summing.

2. Step 2: Calculate the sum of elements

   Sum of [1, 2, 3, 4] is 10. The aggregate returns this sum.

3. Final Answer:

   10 -> Option D

4. Quick Check:

   Aggregate sums elements starting at 0 [OK]

Hint: Aggregate with zero start and sum functions returns total sum [OK]

Common Mistakes:

• Confusing aggregate with reduce output
• Ignoring initial zero value
• Expecting product instead of sum

4.

Identify the error in this Spark code snippet:

rdd = sc.parallelize([1, 2, 3])
result = rdd.reduce(lambda x: x + 1)
print(result)

medium

A. The lambda function should take two arguments, not one.

B. The RDD is empty, so reduce will fail.

C. The reduce function cannot be used with integers.

D. The print statement is missing parentheses.

Solution

1. Step 1: Check lambda function signature for reduce

   The reduce function requires a function with two parameters to combine elements pairwise.

2. Step 2: Identify the incorrect lambda

   The lambda lambda x: x + 1 takes only one argument, causing a runtime error.

3. Final Answer:

   The lambda function should take two arguments, not one. -> Option A

4. Quick Check:

   Reduce needs two-argument function [OK]

Hint: Reduce lambda must have two parameters to combine elements [OK]

Common Mistakes:

• Using single-argument lambda in reduce
• Assuming reduce works on empty RDD without error
• Misreading print syntax in Python 3

5.

You have an RDD of tuples representing (category, value):

data = [("A", 2), ("B", 3), ("A", 5), ("B", 7), ("C", 1)]
rdd = sc.parallelize(data)

Which approach correctly uses aggregateByKey to find the sum of values per category?

hard

A. rdd.aggregateByKey(0, lambda acc, v: acc + v, lambda acc1, acc2: acc1 + acc2).collect()

B. rdd.reduce(lambda x, y: x + y).collect()

C. rdd.aggregate(0, lambda acc, x: acc + x[1], lambda acc1, acc2: acc1 + acc2).collect()

D. rdd.mapValues(lambda v: v).reduceByKey(lambda x, y: x * y).collect()

Solution

1. Step 1: Understand aggregateByKey usage

   aggregateByKey starts with zero per key, adds values with first lambda, then merges partitions with second lambda.

2. Step 2: Check each option

   rdd.aggregateByKey(0, lambda acc, v: acc + v, lambda acc1, acc2: acc1 + acc2).collect() correctly uses aggregateByKey with zero start and sum functions per key. rdd.reduce(lambda x, y: x + y).collect() is invalid syntax for key-value RDD. rdd.aggregate(0, lambda acc, x: acc + x[1], lambda acc1, acc2: acc1 + acc2).collect() uses aggregate on whole RDD, not per key. rdd.mapValues(lambda v: v).reduceByKey(lambda x, y: x * y).collect() multiplies values, not sums.

3. Final Answer:

   rdd.aggregateByKey(0, lambda acc, v: acc + v, lambda acc1, acc2: acc1 + acc2).collect() -> Option A

4. Quick Check:

   aggregateByKey sums values per key [OK]

Hint: Use aggregateByKey with zero start and sum lambdas for per-key sums [OK]

Common Mistakes:

• Using aggregate instead of aggregateByKey for key-value RDD
• Using reduce without key awareness
• Multiplying instead of summing values