Map, filter, and flatMap operations in Apache Spark - Step-by-Step Execution

Concept Flow - Map, filter, and flatMap operations

Start with RDD/DataFrame

↓

Map

↓

Transformed RDD/DataFrame

↓

Result

Start with data, then apply map to transform each item, filter to keep items by condition, or flatMap to transform and flatten lists, finally collect results.

Execution Sample

Apache Spark

rdd = sc.parallelize([1, 2, 3, 4])
map_rdd = rdd.map(lambda x: x * 2)
filter_rdd = map_rdd.filter(lambda x: x > 4)
flatmap_rdd = filter_rdd.flatMap(lambda x: [x, x+1])
result = flatmap_rdd.collect()

This code doubles numbers, keeps those greater than 4, then expands each to two numbers, and collects the final list.

Execution Table

Step	RDD Content	Operation	Transformation Detail	Resulting RDD Content
1	[1, 2, 3, 4]	Start	Initial RDD	[1, 2, 3, 4]
2	[1, 2, 3, 4]	map	Multiply each by 2	[2, 4, 6, 8]
3	[2, 4, 6, 8]	filter	Keep elements > 4	[6, 8]
4	[6, 8]	flatMap	For each x, create [x, x+1]	[6, 7, 8, 9]
5	[6, 7, 8, 9]	collect	Gather all elements to driver	[6, 7, 8, 9]

💡 After collect, data is gathered and processing stops.

Variable Tracker

Variable	Start	After map	After filter	After flatMap	After collect
rdd	[1, 2, 3, 4]	[1, 2, 3, 4]	[1, 2, 3, 4]	[1, 2, 3, 4]	[1, 2, 3, 4]
map_rdd		[2, 4, 6, 8]	[2, 4, 6, 8]	[2, 4, 6, 8]	[2, 4, 6, 8]
filter_rdd			[6, 8]	[6, 8]	[6, 8]
flatmap_rdd				[6, 7, 8, 9]	[6, 7, 8, 9]
result					[6, 7, 8, 9]

Key Moments - 3 Insights

Why does flatMap produce more elements than filter or map?

Because flatMap takes each element and returns a list, then flattens all lists into one big list, increasing total elements (see execution_table step 4).

Does filter change the values of elements?

No, filter only keeps or removes elements based on a condition without changing their values (see execution_table step 3).

When is the actual computation done in Spark for these operations?

Computation happens only when an action like collect() is called, not during map, filter, or flatMap (see execution_table step 5).

Visual Quiz - 3 Questions

Look at the execution table, what is the content of filter_rdd after step 3?

A [1, 2, 3, 4]

B [2, 4, 6, 8]

C [6, 8]

D [6, 7, 8, 9]

💡 Hint

Check the 'Resulting RDD Content' column at step 3 in the execution_table.

At which step does the RDD first contain more elements than the previous step?

A Step 4 (flatMap)

B Step 2 (map)

C Step 3 (filter)

D Step 5 (collect)

💡 Hint

Look at the 'Resulting RDD Content' sizes in execution_table rows.

If the filter condition changed to x > 7, what would be the content of filter_rdd after step 3?

A [6, 8]

B [8]

C [2, 4, 6, 8]

D []

💡 Hint

Filter keeps elements strictly greater than 7, check values after map in execution_table step 2.

Concept Snapshot

Map, filter, and flatMap are Spark transformations.
Map changes each element.
Filter keeps elements by condition.
FlatMap changes and flattens lists.
Actions like collect trigger execution.

Full Transcript

We start with an RDD of numbers. Map doubles each number. Filter keeps only numbers greater than 4. FlatMap takes each number and creates a list with the number and the next number, then flattens all lists into one RDD. Finally, collect gathers all elements to the driver program. Map changes values, filter removes some elements, flatMap can increase the number of elements. Actual computation happens only when collect is called.

Practice

1. What does the filter operation do in Apache Spark?

easy

A. Splits each item into multiple items and flattens the result

B. Changes each item in the dataset to a new item

C. Selects items from the dataset based on a condition

D. Combines two datasets into one

Solution

1. Step 1: Understand the purpose of filter

   The filter operation keeps only those items that meet a specific condition or rule.

2. Step 2: Compare with other operations

   Unlike map or flatMap, filter does not change items but selects some based on a test.

3. Final Answer:

   Selects items from the dataset based on a condition -> Option C

4. Quick Check:

   filter = select items [OK]

Hint: Filter picks items by condition, map changes items, flatMap expands items [OK]

Common Mistakes:

• Confusing filter with map operation
• Thinking filter changes items instead of selecting
• Mixing flatMap behavior with filter

2. Which of the following is the correct syntax to apply a map operation on an RDD named rdd to multiply each element by 2?

easy

A. rdd.flatMap(x => x * 2)

B. rdd.filter(x => x * 2)

C. rdd.map(x, y => x * 2)

D. rdd.map(x => x * 2)

Solution

1. Step 1: Identify correct map syntax

   The map function takes one argument function: x => x * 2 applies multiplication to each element.

2. Step 2: Check other options

   filter expects a boolean condition, flatMap expects a function returning iterable, and map with two parameters is invalid.

3. Final Answer:

   rdd.map(x => x * 2) -> Option D

4. Quick Check:

   map syntax = rdd.map(x => x * 2) [OK]

Hint: map uses one argument function: x => new_value [OK]

Common Mistakes:

• Using filter instead of map for transformation
• Passing two parameters to map function
• Using flatMap when map is needed

3. Given the following Spark code:

val rdd = sc.parallelize(List("apple", "banana", "pear"))
val result = rdd.flatMap(word => word.toCharArray)
result.collect()

What is the output of result.collect()?

medium

A. ["apple", "banana", "pear"]

B. ["a", "p", "p", "l", "e", "b", "a", "n", "a", "n", "a", "p", "e", "a", "r"]

C. [["a", "p", "p", "l", "e"], ["b", "a", "n", "a", "n", "a"], ["p", "e", "a", "r"]]

D. ["applebananapear"]

Solution

1. Step 1: Understand flatMap with toCharArray

   flatMap applies toCharArray to each word, turning each word into a list of characters.

2. Step 2: Flatten the lists into one sequence

   flatMap flattens all character arrays into a single list of characters.

3. Final Answer:

   ["a", "p", "p", "l", "e", "b", "a", "n", "a", "n", "a", "p", "e", "a", "r"] -> Option B

4. Quick Check:

   flatMap + toCharArray = flat list of chars [OK]

Hint: flatMap flattens lists from each item into one list [OK]

Common Mistakes:

• Expecting flatMap to keep nested lists
• Confusing map output with flatMap output
• Thinking flatMap returns original words

4. Identify the error in this Spark code snippet:

val rdd = sc.parallelize(List(1, 2, 3, 4))
val filtered = rdd.filter(x => x > 2)
val mapped = filtered.map(x => x * 2)
val result = mapped.flatMap(x => x)
result.collect()

medium

A. flatMap function argument should return an iterable, but it returns an integer

B. filter condition syntax is incorrect

C. map function cannot multiply integers

D. parallelize method is used incorrectly

Solution

1. Step 1: Analyze flatMap argument

   flatMap expects a function returning a list or iterable, but here it returns a single integer.

2. Step 2: Check other parts

   filter and map are correctly used; parallelize is correct.

3. Final Answer:

   flatMap function argument should return an iterable, but it returns an integer -> Option A

4. Quick Check:

   flatMap needs iterable return [OK]

Hint: flatMap needs iterable output, not single values [OK]

Common Mistakes:

• Passing single value to flatMap instead of iterable
• Misreading filter syntax as error
• Thinking map cannot multiply numbers

5. You have an RDD of sentences:

val sentences = sc.parallelize(List("hello world", "apache spark", "map filter flatMap"))

How would you use flatMap and filter to get all words longer than 4 characters?

hard

A. sentences.flatMap(sentence => sentence.split(" ")).filter(word => word.length > 4)

B. sentences.map(sentence => sentence.split(" ")).filter(word => word.length > 4)

C. sentences.filter(sentence => sentence.length > 4).flatMap(word => word.split(" "))

D. sentences.flatMap(sentence => sentence.length > 4)

Solution

1. Step 1: Use flatMap to split sentences into words

   flatMap splits each sentence into words and flattens them into one list.

2. Step 2: Use filter to select words longer than 4 characters

   filter keeps only words where length is greater than 4.

3. Final Answer:

   sentences.flatMap(sentence => sentence.split(" ")).filter(word => word.length > 4) -> Option A

4. Quick Check:

   flatMap split + filter length > 4 = correct [OK]

Hint: flatMap splits, filter selects by condition [OK]

Common Mistakes:

• Using map instead of flatMap to split words
• Filtering sentences instead of words
• Passing wrong argument types to flatMap