Map, filter, and flatMap operations in Apache Spark - Mini Project: Build & Apply

Map, filter, and flatMap operations

📖 Scenario: You work at a small online bookstore. You have a list of book titles and want to process them to find interesting information.

🎯 Goal: Learn how to use map, filter, and flatMap operations in Apache Spark to transform and analyze book titles.

📋 What You'll Learn

Create an RDD with given book titles

Create a filter condition variable

Use map to convert titles to uppercase

Use filter to keep titles longer than a threshold

Use flatMap to split titles into words

Print the final results

💡 Why This Matters

🌍 Real World

Processing text data like book titles is common in data science to prepare data for analysis or search indexing.

💼 Career

Understanding <code>map</code>, <code>filter</code>, and <code>flatMap</code> is essential for working with big data frameworks like Apache Spark in data engineering and data science roles.

1

Create the initial RDD with book titles

Create an RDD called books_rdd from the list ["The Hobbit", "War and Peace", "Pride and Prejudice", "The Catcher in the Rye"] using sc.parallelize().

Apache Spark

# Create books_rdd from the list of book titles
# Your code here

Hint

Use sc.parallelize() to create an RDD from a Python list.

2

Set the minimum title length filter

Create a variable called min_length and set it to 12. This will be used to filter book titles longer than this length.

Apache Spark

books = ["The Hobbit", "War and Peace", "Pride and Prejudice", "The Catcher in the Rye"]
books_rdd = sc.parallelize(books)
# Create min_length variable
# Your code here

Hint

Just create a variable min_length and assign the number 12.

3

Apply map, filter, and flatMap operations

Use map on books_rdd to convert each title to uppercase and save as upper_rdd. Then use filter on upper_rdd to keep titles with length greater than min_length and save as filtered_rdd. Finally, use flatMap on filtered_rdd to split each title into words and save as words_rdd.

Apache Spark

books = ["The Hobbit", "War and Peace", "Pride and Prejudice", "The Catcher in the Rye"]
books_rdd = sc.parallelize(books)
min_length = 12
# Convert titles to uppercase using map
# Filter titles longer than min_length
# Split titles into words using flatMap
# Your code here

Hint

Use map to uppercase, filter with len(title) > min_length, and flatMap with split().

4

Print the final list of words

Collect the words_rdd into a list and print it using print().

Apache Spark

books = ["The Hobbit", "War and Peace", "Pride and Prejudice", "The Catcher in the Rye"]
books_rdd = sc.parallelize(books)
min_length = 12
upper_rdd = books_rdd.map(lambda title: title.upper())
filtered_rdd = upper_rdd.filter(lambda title: len(title) > min_length)
words_rdd = filtered_rdd.flatMap(lambda title: title.split())
# Collect words_rdd and print the list
# Your code here

Hint

Use collect() to get the list and then print it.

Practice

1. What does the filter operation do in Apache Spark?

easy

A. Splits each item into multiple items and flattens the result

B. Changes each item in the dataset to a new item

C. Selects items from the dataset based on a condition

D. Combines two datasets into one

Solution

1. Step 1: Understand the purpose of filter

   The filter operation keeps only those items that meet a specific condition or rule.

2. Step 2: Compare with other operations

   Unlike map or flatMap, filter does not change items but selects some based on a test.

3. Final Answer:

   Selects items from the dataset based on a condition -> Option C

4. Quick Check:

   filter = select items [OK]

Hint: Filter picks items by condition, map changes items, flatMap expands items [OK]

Common Mistakes:

• Confusing filter with map operation
• Thinking filter changes items instead of selecting
• Mixing flatMap behavior with filter

2. Which of the following is the correct syntax to apply a map operation on an RDD named rdd to multiply each element by 2?

easy

A. rdd.flatMap(x => x * 2)

B. rdd.filter(x => x * 2)

C. rdd.map(x, y => x * 2)

D. rdd.map(x => x * 2)

Solution

1. Step 1: Identify correct map syntax

   The map function takes one argument function: x => x * 2 applies multiplication to each element.

2. Step 2: Check other options

   filter expects a boolean condition, flatMap expects a function returning iterable, and map with two parameters is invalid.

3. Final Answer:

   rdd.map(x => x * 2) -> Option D

4. Quick Check:

   map syntax = rdd.map(x => x * 2) [OK]

Hint: map uses one argument function: x => new_value [OK]

Common Mistakes:

• Using filter instead of map for transformation
• Passing two parameters to map function
• Using flatMap when map is needed

3. Given the following Spark code:

val rdd = sc.parallelize(List("apple", "banana", "pear"))
val result = rdd.flatMap(word => word.toCharArray)
result.collect()

What is the output of result.collect()?

medium

A. ["apple", "banana", "pear"]

B. ["a", "p", "p", "l", "e", "b", "a", "n", "a", "n", "a", "p", "e", "a", "r"]

C. [["a", "p", "p", "l", "e"], ["b", "a", "n", "a", "n", "a"], ["p", "e", "a", "r"]]

D. ["applebananapear"]

Solution

1. Step 1: Understand flatMap with toCharArray

   flatMap applies toCharArray to each word, turning each word into a list of characters.

2. Step 2: Flatten the lists into one sequence

   flatMap flattens all character arrays into a single list of characters.

3. Final Answer:

   ["a", "p", "p", "l", "e", "b", "a", "n", "a", "n", "a", "p", "e", "a", "r"] -> Option B

4. Quick Check:

   flatMap + toCharArray = flat list of chars [OK]

Hint: flatMap flattens lists from each item into one list [OK]

Common Mistakes:

• Expecting flatMap to keep nested lists
• Confusing map output with flatMap output
• Thinking flatMap returns original words

4. Identify the error in this Spark code snippet:

val rdd = sc.parallelize(List(1, 2, 3, 4))
val filtered = rdd.filter(x => x > 2)
val mapped = filtered.map(x => x * 2)
val result = mapped.flatMap(x => x)
result.collect()

medium

A. flatMap function argument should return an iterable, but it returns an integer

B. filter condition syntax is incorrect

C. map function cannot multiply integers

D. parallelize method is used incorrectly

Solution

1. Step 1: Analyze flatMap argument

   flatMap expects a function returning a list or iterable, but here it returns a single integer.

2. Step 2: Check other parts

   filter and map are correctly used; parallelize is correct.

3. Final Answer:

   flatMap function argument should return an iterable, but it returns an integer -> Option A

4. Quick Check:

   flatMap needs iterable return [OK]

Hint: flatMap needs iterable output, not single values [OK]

Common Mistakes:

• Passing single value to flatMap instead of iterable
• Misreading filter syntax as error
• Thinking map cannot multiply numbers

5. You have an RDD of sentences:

val sentences = sc.parallelize(List("hello world", "apache spark", "map filter flatMap"))

How would you use flatMap and filter to get all words longer than 4 characters?

hard

A. sentences.flatMap(sentence => sentence.split(" ")).filter(word => word.length > 4)

B. sentences.map(sentence => sentence.split(" ")).filter(word => word.length > 4)

C. sentences.filter(sentence => sentence.length > 4).flatMap(word => word.split(" "))

D. sentences.flatMap(sentence => sentence.length > 4)

Solution

1. Step 1: Use flatMap to split sentences into words

   flatMap splits each sentence into words and flattens them into one list.

2. Step 2: Use filter to select words longer than 4 characters

   filter keeps only words where length is greater than 4.

3. Final Answer:

   sentences.flatMap(sentence => sentence.split(" ")).filter(word => word.length > 4) -> Option A

4. Quick Check:

   flatMap split + filter length > 4 = correct [OK]

Hint: flatMap splits, filter selects by condition [OK]

Common Mistakes:

• Using map instead of flatMap to split words
• Filtering sentences instead of words
• Passing wrong argument types to flatMap