Map, filter, and flatMap operations in Apache Spark - Interactive Code Practice

Practice - 5 Tasks

1fill in blank

easy

Complete the code to apply a map operation that doubles each number in the RDD.

Apache Spark

rdd2 = rdd.[1](lambda x: x * 2)

Drag options to blanks, or click blank then click option'

Amap

Bfilter

CflatMap

Dreduce

Attempts:

3 left

💡 Hint

Common Mistakes

Using filter instead of map, which would remove elements instead of transforming them.

Using reduce, which combines all elements into one value.

The map operation applies a function to each element in the RDD, here doubling each number.

2fill in blank

medium

Complete the code to filter out all numbers less than 5 from the RDD.

Apache Spark

filtered_rdd = rdd.[1](lambda x: x >= 5)

Drag options to blanks, or click blank then click option'

Amap

Bfilter

CflatMap

Dreduce

Attempts:

3 left

💡 Hint

Common Mistakes

Using map instead of filter, which would transform elements but not remove any.

Using flatMap, which expects a function returning a list or iterable.

The filter operation keeps only elements that satisfy the condition, here numbers greater or equal to 5.

3fill in blank

hard

Fix the error in the code to correctly flatten a list of words from sentences using flatMap.

Apache Spark

words = sentences.[1](lambda sentence: sentence.split(' '))

Drag options to blanks, or click blank then click option'

Afilter

Bmap

CflatMap

Dreduce

Attempts:

3 left

💡 Hint

Common Mistakes

Using map, which results in an RDD of lists instead of a flat list of words.

Using filter or reduce, which do not flatten lists.

flatMap applies the function and flattens the resulting lists into a single RDD of words.

4fill in blank

hard

Complete the code to create a dictionary of words and their lengths, but only for words longer than 3 characters.

Apache Spark

word_lengths = {word: len(word) for word in words if len(word) [1] 3}

Drag options to blanks, or click blank then click option'

A:

B>

C<

D=

Attempts:

3 left

💡 Hint

Common Mistakes

Using '=' instead of ':' in dictionary comprehension.

Using '<' instead of '>' in the condition, which would select shorter words.

The colon : separates keys and values in a dictionary. The condition > filters words longer than 3 characters.

5fill in blank

hard

Fill all three blanks to create a dictionary with uppercase words as keys, their original words as values, but only for words longer than 4 characters.

Apache Spark

result = [1]: [2] for word in words if len(word) [3] 4}

Drag options to blanks, or click blank then click option'

Aword.upper()

Bword

C>

Dlen(word)

Attempts:

3 left

💡 Hint

Common Mistakes

Using length as a key or value instead of the word or its uppercase.

Using '<' instead of '>' in the condition.

The key is the uppercase word (word.upper()), the value is the original word (word), and the condition keeps words longer than 4 characters (>).

Practice

1. What does the filter operation do in Apache Spark?

easy

A. Splits each item into multiple items and flattens the result

B. Changes each item in the dataset to a new item

C. Selects items from the dataset based on a condition

D. Combines two datasets into one

Solution

1. Step 1: Understand the purpose of filter

   The filter operation keeps only those items that meet a specific condition or rule.

2. Step 2: Compare with other operations

   Unlike map or flatMap, filter does not change items but selects some based on a test.

3. Final Answer:

   Selects items from the dataset based on a condition -> Option C

4. Quick Check:

   filter = select items [OK]

Hint: Filter picks items by condition, map changes items, flatMap expands items [OK]

Common Mistakes:

• Confusing filter with map operation
• Thinking filter changes items instead of selecting
• Mixing flatMap behavior with filter

2. Which of the following is the correct syntax to apply a map operation on an RDD named rdd to multiply each element by 2?

easy

A. rdd.flatMap(x => x * 2)

B. rdd.filter(x => x * 2)

C. rdd.map(x, y => x * 2)

D. rdd.map(x => x * 2)

Solution

1. Step 1: Identify correct map syntax

   The map function takes one argument function: x => x * 2 applies multiplication to each element.

2. Step 2: Check other options

   filter expects a boolean condition, flatMap expects a function returning iterable, and map with two parameters is invalid.

3. Final Answer:

   rdd.map(x => x * 2) -> Option D

4. Quick Check:

   map syntax = rdd.map(x => x * 2) [OK]

Hint: map uses one argument function: x => new_value [OK]

Common Mistakes:

• Using filter instead of map for transformation
• Passing two parameters to map function
• Using flatMap when map is needed

3. Given the following Spark code:

val rdd = sc.parallelize(List("apple", "banana", "pear"))
val result = rdd.flatMap(word => word.toCharArray)
result.collect()

What is the output of result.collect()?

medium

A. ["apple", "banana", "pear"]

B. ["a", "p", "p", "l", "e", "b", "a", "n", "a", "n", "a", "p", "e", "a", "r"]

C. [["a", "p", "p", "l", "e"], ["b", "a", "n", "a", "n", "a"], ["p", "e", "a", "r"]]

D. ["applebananapear"]

Solution

1. Step 1: Understand flatMap with toCharArray

   flatMap applies toCharArray to each word, turning each word into a list of characters.

2. Step 2: Flatten the lists into one sequence

   flatMap flattens all character arrays into a single list of characters.

3. Final Answer:

   ["a", "p", "p", "l", "e", "b", "a", "n", "a", "n", "a", "p", "e", "a", "r"] -> Option B

4. Quick Check:

   flatMap + toCharArray = flat list of chars [OK]

Hint: flatMap flattens lists from each item into one list [OK]

Common Mistakes:

• Expecting flatMap to keep nested lists
• Confusing map output with flatMap output
• Thinking flatMap returns original words

4. Identify the error in this Spark code snippet:

val rdd = sc.parallelize(List(1, 2, 3, 4))
val filtered = rdd.filter(x => x > 2)
val mapped = filtered.map(x => x * 2)
val result = mapped.flatMap(x => x)
result.collect()

medium

A. flatMap function argument should return an iterable, but it returns an integer

B. filter condition syntax is incorrect

C. map function cannot multiply integers

D. parallelize method is used incorrectly

Solution

1. Step 1: Analyze flatMap argument

   flatMap expects a function returning a list or iterable, but here it returns a single integer.

2. Step 2: Check other parts

   filter and map are correctly used; parallelize is correct.

3. Final Answer:

   flatMap function argument should return an iterable, but it returns an integer -> Option A

4. Quick Check:

   flatMap needs iterable return [OK]

Hint: flatMap needs iterable output, not single values [OK]

Common Mistakes:

• Passing single value to flatMap instead of iterable
• Misreading filter syntax as error
• Thinking map cannot multiply numbers

5. You have an RDD of sentences:

val sentences = sc.parallelize(List("hello world", "apache spark", "map filter flatMap"))

How would you use flatMap and filter to get all words longer than 4 characters?

hard

A. sentences.flatMap(sentence => sentence.split(" ")).filter(word => word.length > 4)

B. sentences.map(sentence => sentence.split(" ")).filter(word => word.length > 4)

C. sentences.filter(sentence => sentence.length > 4).flatMap(word => word.split(" "))

D. sentences.flatMap(sentence => sentence.length > 4)

Solution

1. Step 1: Use flatMap to split sentences into words

   flatMap splits each sentence into words and flattens them into one list.

2. Step 2: Use filter to select words longer than 4 characters

   filter keeps only words where length is greater than 4.

3. Final Answer:

   sentences.flatMap(sentence => sentence.split(" ")).filter(word => word.length > 4) -> Option A

4. Quick Check:

   flatMap split + filter length > 4 = correct [OK]

Hint: flatMap splits, filter selects by condition [OK]

Common Mistakes:

• Using map instead of flatMap to split words
• Filtering sentences instead of words
• Passing wrong argument types to flatMap