Reduce and aggregate actions in Apache Spark - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

What is the output of this Spark reduce operation?

Given the following Spark code, what will be the output printed?

Apache Spark

from pyspark.sql import SparkSession
spark = SparkSession.builder.master('local').appName('Test').getOrCreate()
rdd = spark.sparkContext.parallelize([1, 2, 3, 4])
result = rdd.reduce(lambda x, y: x + y)
print(result)

A TypeError

B 24

C 10

D None

💡 Hint

Think about what reduce does with the lambda function summing two numbers.

The reduce action applies the lambda function cumulatively to the elements of the RDD, summing all numbers: 1+2+3+4 = 10.

❓ data_output

intermediate

What is the result of this aggregate action?

Consider this Spark code using aggregate to compute sum and count. What is the final output?

Apache Spark

from pyspark.sql import SparkSession
spark = SparkSession.builder.master('local').appName('Test').getOrCreate()
rdd = spark.sparkContext.parallelize([1, 2, 3, 4])
zero_value = (0, 0)
seq_op = lambda acc, x: (acc[0] + x, acc[1] + 1)
comb_op = lambda acc1, acc2: (acc1[0] + acc2[0], acc1[1] + acc2[1])
result = rdd.aggregate(zero_value, seq_op, comb_op)
print(result)

A (10, 4)

B (24, 4)

C TypeError

D (10, 10)

💡 Hint

Aggregate returns a combined result of sum and count.

The aggregate sums all elements (10) and counts them (4), returning (10, 4).

❓ Predict Output

advanced

What is the output of this Spark reduceByKey operation?

Given the following Spark code, what will be the output printed?

Apache Spark

from pyspark.sql import SparkSession
spark = SparkSession.builder.master('local').appName('Test').getOrCreate()
rdd = spark.sparkContext.parallelize([('a', 1), ('b', 2), ('a', 3)])
result = rdd.reduceByKey(lambda x, y: x + y).collect()
print(result)

A [('a', 4), ('b', 2)]

B AttributeError

C TypeError

D ValueError

💡 Hint

reduceByKey sums values for each key.

reduceByKey sums values for each key: 'a' -> 1+3=4, 'b' -> 2.

❓ visualization

advanced

Visualize the result of countByValue action

Given this Spark code, what is the output of countByValue?

Apache Spark

from pyspark.sql import SparkSession
spark = SparkSession.builder.master('local').appName('Test').getOrCreate()
rdd = spark.sparkContext.parallelize(['apple', 'banana', 'apple', 'orange', 'banana', 'apple'])
result = rdd.countByValue()
print(result)

A {'apple': 1, 'banana': 1, 'orange': 1}

B {'apple': 2, 'banana': 2, 'orange': 2}

C TypeError

D {'apple': 3, 'banana': 2, 'orange': 1}

💡 Hint

countByValue counts how many times each value appears.

The countByValue returns a dictionary with counts of each fruit: apple 3, banana 2, orange 1.

🧠 Conceptual

expert

Which Spark action returns a single value by combining all elements?

Among these Spark actions, which one returns a single value by combining all elements of an RDD using a function?

A take

B reduce

C countByKey

D collect

💡 Hint

Think about which action combines all elements into one result.

The reduce action combines all elements using a function and returns a single value.