Map, filter, and flatMap operations in Apache Spark - Cheat Sheet & Quick Revision

Recall & Review

beginner

What does the map operation do in Apache Spark?

The map operation applies a function to each element of an RDD or DataFrame column and returns a new RDD or column with the transformed elements.

beginner

Explain the purpose of the filter operation in Spark.

The filter operation selects elements from an RDD or DataFrame column that satisfy a given condition, removing all others.

intermediate

What is the difference between map and flatMap in Spark?

map transforms each element into exactly one element, while flatMap can transform each element into zero or more elements, flattening the results into a single list.

beginner

Give a real-life example of using filter in data processing.

Filtering a list of customer orders to keep only those with an amount greater than $100 is a real-life example of using filter.

intermediate

Why is flatMap useful when working with text data in Spark?

flatMap is useful for splitting lines of text into words because it can turn one line into many words, flattening them into a single list for easier processing.

What does the map operation return when applied to an RDD?

A A new RDD with each element transformed by a function

B Only elements that satisfy a condition

C A flattened list of elements

D The original RDD unchanged

map applies a function to each element and returns a new RDD with transformed elements.

Which operation would you use to keep only even numbers from an RDD?

A filter

B map

C flatMap

D reduce

filter selects elements that meet a condition, such as being even.

What is the key feature of flatMap compared to map?

A It returns fewer elements

B It can return zero or more elements per input element

C It only works on numbers

D It filters elements

flatMap can return multiple elements per input, flattening the results.

If you want to split sentences into words in Spark, which operation is best?

A map

B filter

C flatMap

D groupBy

flatMap splits each sentence into multiple words and flattens them.

Which operation removes elements from an RDD?

A collect

B map

C flatMap

D filter

filter removes elements that do not meet the condition.

Describe how you would use map, filter, and flatMap to process a list of sentences to get all words longer than 3 letters.

Think about splitting, selecting, and transforming steps.

Explain the difference between map and flatMap with an example.

Focus on output size per input element.