Why transformations build processing pipelines in Apache Spark - Test Your Understanding

Practice - 5 Tasks

1fill in blank

easy

Complete the code to create a new RDD by applying a transformation.

Apache Spark

rdd2 = rdd.[1](lambda x: x * 2)

Drag options to blanks, or click blank then click option'

Afilter

Breduce

Ccollect

Dmap

Attempts:

3 left

💡 Hint

Common Mistakes

Using 'collect' which returns data to the driver, not a transformation.

Using 'reduce' which aggregates data, not element-wise transformation.

The map transformation applies a function to each element and returns a new RDD.

2fill in blank

medium

Complete the code to filter RDD elements greater than 10.

Apache Spark

filtered_rdd = rdd.[1](lambda x: x > 10)

Drag options to blanks, or click blank then click option'

Amap

BflatMap

Cfilter

DreduceByKey

Attempts:

3 left

💡 Hint

Common Mistakes

Using 'map' which transforms all elements but does not filter.

Using 'reduceByKey' which is for key-value RDDs.

The filter transformation keeps only elements that satisfy the condition.

3fill in blank

hard

Fix the error in the code to chain transformations correctly.

Apache Spark

result = rdd.[1](lambda x: x + 1).[2](lambda x: x * 2)

Drag options to blanks, or click blank then click option'

Amap

Bfilter

Creduce

Dcollect

Attempts:

3 left

💡 Hint

Common Mistakes

Using 'filter' in the second step which expects a boolean function.

Using 'collect' which triggers action, not transformation.

Both transformations should be map to apply functions element-wise and chain properly.

4fill in blank

hard

Fill both blanks to build a pipeline that filters and then maps the RDD.

Apache Spark

pipeline = rdd.[1](lambda x: x % 2 == 0).[2](lambda x: x * 10)

Drag options to blanks, or click blank then click option'

Afilter

Bmap

Creduce

DflatMap

Attempts:

3 left

💡 Hint

Common Mistakes

Reversing the order of transformations.

Using 'reduce' which aggregates instead of transforming.

First filter keeps even numbers, then map multiplies them by 10.

5fill in blank

hard

Fill all three blanks to create a pipeline that filters, maps, and then collects results.

Apache Spark

final_result = rdd.[1](lambda x: x > 5).[2](lambda x: x - 1).[3]()

Drag options to blanks, or click blank then click option'

Afilter

Bmap

Ccollect

Dreduce

Attempts:

3 left

💡 Hint

Common Mistakes

Using 'reduce' instead of 'collect' which returns a single value.

Calling 'collect' before transformations.

The pipeline filters elements greater than 5, maps by subtracting 1, then collects the results to the driver.

Practice

1. What does it mean when we say Spark transformations are lazy?

easy

A. They do not run immediately but build a plan to run later

B. They run instantly as soon as they are called

C. They save data permanently after each step

D. They only work on small datasets

Solution

1. Step 1: Understand the meaning of lazy evaluation in Spark

   Lazy means Spark does not execute transformations right away but waits until an action is called.

2. Step 2: Connect lazy evaluation to building a processing plan

   Transformations build a pipeline or plan that Spark will run later when needed.

3. Final Answer:

   They do not run immediately but build a plan to run later -> Option A

4. Quick Check:

   Lazy evaluation = build plan, not run immediately [OK]

Hint: Lazy means wait to run until action triggers [OK]

Common Mistakes:

• Thinking transformations run immediately
• Confusing lazy with saving data
• Believing transformations only work on small data

2. Which of the following is the correct way to chain two transformations in Spark?

easy

A. rdd.map(...); filter(...)

B. rdd.filter(...).map(...)()

C. rdd.filter(...).map

D. rdd.map(...).filter(...)

Solution

1. Step 1: Review correct chaining syntax in Spark

   Transformations are chained by calling one after another with dot notation, like rdd.map(...).filter(...).

2. Step 2: Identify incorrect syntax in other options

   rdd.map(...); filter(...) uses semicolon which breaks chaining, rdd.filter(...).map(...)() has extra parentheses, rdd.filter(...).map misses parentheses after map.

3. Final Answer:

   rdd.map(...).filter(...) -> Option D

4. Quick Check:

   Chaining uses dot notation with parentheses [OK]

Hint: Use dot and parentheses to chain transformations [OK]

Common Mistakes:

• Adding extra parentheses after chain
• Using semicolons instead of chaining
• Forgetting parentheses after function names

3. Given the code below, what will be the output after the action is called?

rdd = sc.parallelize([1, 2, 3, 4])
result = rdd.map(lambda x: x * 2).filter(lambda x: x > 4).collect()

medium

A. [2, 4, 6, 8]

B. [6, 8]

C. [1, 2, 3, 4]

D. Error due to missing action

Solution

1. Step 1: Apply the map transformation

   Each element is multiplied by 2: [2, 4, 6, 8]

2. Step 2: Apply the filter transformation

   Keep elements greater than 4: [6, 8]

3. Final Answer:

   [6, 8] -> Option B

4. Quick Check:

   Map then filter results in [6, 8] [OK]

Hint: Apply transformations step-by-step before action [OK]

Common Mistakes:

• Forgetting filter condition
• Assuming map runs after collect
• Thinking collect is a transformation

4. Identify the error in this Spark code snippet:

rdd = sc.parallelize([1, 2, 3])
new_rdd = rdd.map(lambda x: x + 1)
new_rdd.collect

medium

A. map function syntax is wrong

B. parallelize method is deprecated

C. Missing parentheses after collect

D. collect should be called before map

Solution

1. Step 1: Check usage of collect method

   collect is a function and must be called with parentheses: collect().

2. Step 2: Verify other parts of the code

   map syntax and parallelize usage are correct; collect must come after transformations.

3. Final Answer:

   Missing parentheses after collect -> Option C

4. Quick Check:

   collect() needs parentheses to run [OK]

Hint: Remember collect is a function, use () [OK]

Common Mistakes:

• Calling collect without parentheses
• Thinking map syntax is incorrect
• Calling collect before transformations

5. You want to optimize a Spark job by chaining multiple transformations before an action. Which benefit does this pipeline approach provide?

hard

A. Spark can optimize the entire chain and reduce data scans

B. Each transformation saves intermediate results to disk

C. Transformations run in parallel immediately

D. Actions are skipped if transformations are chained

Solution

1. Step 1: Understand pipeline optimization in Spark

   Chaining transformations lets Spark analyze the whole plan and optimize execution.

2. Step 2: Contrast with other options

   Transformations do not save intermediate results by default, do not run immediately, and actions are never skipped.

3. Final Answer:

   Spark can optimize the entire chain and reduce data scans -> Option A

4. Quick Check:

   Pipeline builds optimization plan [OK]

Hint: Pipeline lets Spark optimize all steps together [OK]

Common Mistakes:

• Thinking transformations save intermediate data
• Believing transformations run immediately
• Assuming actions can be skipped