Why transformations build processing pipelines in Apache Spark - Challenge Your Understanding

Challenge - 5 Problems

🧠 Conceptual

intermediate

Why are Spark transformations lazy?

In Apache Spark, transformations like map() and filter() are called lazy. What is the main reason for this laziness?

A To build a processing pipeline that optimizes execution before running any computation

B To immediately execute each transformation and store intermediate results

C To prevent any data from being processed at all

D To automatically cache all data in memory after each transformation

💡 Hint

Think about how Spark plans work before running jobs.

Spark delays execution of transformations to build a pipeline of operations. This allows it to optimize the entire workflow before running any actual computation.

❓ Predict Output

intermediate

Output of chained transformations in Spark

Given the following Spark code, what is the output when result.collect() is called?

Apache Spark

rdd = sc.parallelize([1, 2, 3, 4, 5])
result = rdd.filter(lambda x: x % 2 == 0).map(lambda x: x * 10)

A [2, 4]

B [20, 40]

C [10, 20, 30, 40, 50]

D []

💡 Hint

Filter keeps even numbers, then map multiplies by 10.

The filter keeps only even numbers [2,4], then map multiplies each by 10 resulting in [20,40].

❓ data_output

advanced

Number of stages in a Spark pipeline

Consider this Spark code snippet. How many stages will Spark create when result.count() is called?

Apache Spark

rdd = sc.parallelize([1, 2, 3, 4, 5])
result = rdd.map(lambda x: x + 1).filter(lambda x: x > 3).map(lambda x: x * 2)

A 2

B 4

C 3

D 1

💡 Hint

All transformations are narrow dependencies.

Since all transformations are narrow (map and filter), Spark can combine them into a single stage.

🔧 Debug

advanced

Why does this Spark job run slowly?

This Spark code runs slowly. What is the main reason?

Apache Spark

rdd = sc.textFile('data.txt')
words = rdd.flatMap(lambda line: line.split())
words_filtered = words.filter(lambda w: len(w) > 3)
words_filtered.cache()
count = words_filtered.count()
print(count)

A The cache() is called but no action triggers caching before count()

B The count() action is missing, so no computation happens

C The RDD is too small to benefit from caching

D The filter transformation is executed before flatMap causing errors

💡 Hint

Caching happens only after an action triggers computation.

Calling cache() marks the RDD to be cached but caching happens only when an action like count() runs. Here, count() triggers caching, but if multiple actions run later, caching helps. If only one action runs, caching adds overhead.

🚀 Application

expert

Optimizing a Spark pipeline with wide and narrow dependencies

You have this Spark pipeline. Which option correctly describes how Spark will execute it?

Apache Spark

rdd = sc.parallelize(range(10))
step1 = rdd.map(lambda x: x + 1)
step2 = step1.groupBy(lambda x: x % 3)
step3 = step2.mapValues(sum)
result = step3.collect()

A Spark will execute all transformations immediately without stages

B Spark will create one stage combining map and groupBy

C Spark will create two stages: one for map (narrow) and one for groupBy (wide) with shuffle

D Spark will create three stages: map, groupBy, and mapValues separately

💡 Hint

Wide dependencies like groupBy cause shuffle and new stages.

The map is a narrow transformation and runs in one stage. The groupBy causes a shuffle (wide dependency), so Spark creates a new stage for it. mapValues is a narrow transformation and runs in the same stage as groupBy output processing.