Reduce and aggregate actions in Apache Spark - Cheat Sheet & Quick Revision

Recall & Review

beginner

What does the reduce() action do in Apache Spark?

The reduce() action combines all elements of an RDD using a specified function that takes two arguments and returns one. It aggregates the data into a single result.

intermediate

Explain the difference between reduce() and aggregate() in Spark.

reduce() combines elements using one function and returns a single value. aggregate() allows different functions for combining within partitions and across partitions, and can return a different type than the input.

beginner

What is the purpose of the zero value in the aggregate() action?

The zero value is the initial value for the aggregation. It is used as a starting point for combining elements within each partition and across partitions.

intermediate

How does fold() differ from reduce() in Spark?

fold() is like reduce() but it uses a zero value as a starting point. This makes fold() safer when the RDD might be empty.

beginner

Give a simple example of using reduce() to sum numbers in an RDD.

If you have an RDD of numbers, you can sum them with: rdd.reduce(lambda a, b: a + b). This adds all numbers together and returns the total.

What does the reduce() action return when applied to an RDD?

A A single aggregated value

B A new RDD with transformed elements

C A list of all elements

D The count of elements

reduce() combines all elements into one single value using the provided function.

Which action allows you to use different functions for combining data within partitions and across partitions?

A aggregate()

B fold()

C reduce()

D map()

aggregate() lets you specify separate functions for within-partition and across-partition aggregation.

Why is a zero value needed in fold() and aggregate()?

A To cache the data

B To filter elements

C To sort the RDD

D To initialize the aggregation

The zero value initializes the aggregation and is used as a starting point.

Which action is safer to use on an empty RDD?

A reduce()

B map()

C fold()

D filter()

fold() uses a zero value, so it can safely handle empty RDDs without errors.

What type of function does reduce() require?

A A function that takes one input and returns one output

B A function that takes two inputs and returns one output

C A function that takes no inputs

D A function that returns a list

reduce() needs a function that combines two elements into one.

Describe how the reduce() action works in Apache Spark and give a simple example.

Think about how you add all numbers in a list using a function.

Explain the difference between aggregate() and fold() actions in Spark.

Consider how aggregation can be customized versus a simpler fold with a starting value.