Map, filter, and flatMap operations in Apache Spark - Mini Project: Build & Apply

Map, filter, and flatMap operations

📖 Scenario: You work at a small online bookstore. You have a list of book titles and want to process them to find interesting information.

🎯 Goal: Learn how to use map, filter, and flatMap operations in Apache Spark to transform and analyze book titles.

📋 What You'll Learn

Create an RDD with given book titles

Create a filter condition variable

Use map to convert titles to uppercase

Use filter to keep titles longer than a threshold

Use flatMap to split titles into words

Print the final results

💡 Why This Matters

🌍 Real World

Processing text data like book titles is common in data science to prepare data for analysis or search indexing.

💼 Career

Understanding <code>map</code>, <code>filter</code>, and <code>flatMap</code> is essential for working with big data frameworks like Apache Spark in data engineering and data science roles.

1

Create the initial RDD with book titles

Create an RDD called books_rdd from the list ["The Hobbit", "War and Peace", "Pride and Prejudice", "The Catcher in the Rye"] using sc.parallelize().

Apache Spark

# Create books_rdd from the list of book titles
# Your code here

Need a hint?

Use sc.parallelize() to create an RDD from a Python list.

2

Set the minimum title length filter

Create a variable called min_length and set it to 12. This will be used to filter book titles longer than this length.

Apache Spark

books = ["The Hobbit", "War and Peace", "Pride and Prejudice", "The Catcher in the Rye"]
books_rdd = sc.parallelize(books)
# Create min_length variable
# Your code here

Need a hint?

Just create a variable min_length and assign the number 12.

3

Apply map, filter, and flatMap operations

Use map on books_rdd to convert each title to uppercase and save as upper_rdd. Then use filter on upper_rdd to keep titles with length greater than min_length and save as filtered_rdd. Finally, use flatMap on filtered_rdd to split each title into words and save as words_rdd.

Apache Spark

books = ["The Hobbit", "War and Peace", "Pride and Prejudice", "The Catcher in the Rye"]
books_rdd = sc.parallelize(books)
min_length = 12
# Convert titles to uppercase using map
# Filter titles longer than min_length
# Split titles into words using flatMap
# Your code here

Need a hint?

Use map to uppercase, filter with len(title) > min_length, and flatMap with split().

4

Print the final list of words

Collect the words_rdd into a list and print it using print().

Apache Spark

books = ["The Hobbit", "War and Peace", "Pride and Prejudice", "The Catcher in the Rye"]
books_rdd = sc.parallelize(books)
min_length = 12
upper_rdd = books_rdd.map(lambda title: title.upper())
filtered_rdd = upper_rdd.filter(lambda title: len(title) > min_length)
words_rdd = filtered_rdd.flatMap(lambda title: title.split())
# Collect words_rdd and print the list
# Your code here

Need a hint?

Use collect() to get the list and then print it.