Why Spark replaced MapReduce for big data in Apache Spark - Challenge Your Understanding

Challenge - 5 Problems

🧠 Conceptual

intermediate

Why is Spark faster than MapReduce?

Which of the following reasons best explains why Apache Spark is generally faster than MapReduce for big data processing?

A Spark uses a single-threaded approach, unlike MapReduce which is multi-threaded.

B Spark uses more disk space than MapReduce, making it faster.

C MapReduce processes data in memory, but Spark writes all data to disk.

D Spark uses in-memory computing which reduces disk I/O, while MapReduce writes intermediate results to disk.

💡 Hint

Think about how data is stored and accessed during processing.

Spark keeps data in memory between operations, which avoids slow disk reads and writes that MapReduce does for every step. This makes Spark much faster.

🧠 Conceptual

intermediate

How does Spark handle iterative algorithms better than MapReduce?

Why is Apache Spark better suited for iterative algorithms like machine learning compared to MapReduce?

A Spark caches data in memory across iterations, while MapReduce reloads data from disk each time.

B MapReduce caches data in memory, but Spark reloads data from disk every iteration.

C Spark does not support iterative algorithms, unlike MapReduce.

D MapReduce uses in-memory caching, making it faster for iterative tasks.

💡 Hint

Consider how data is reused during multiple passes over the same dataset.

Spark can keep datasets in memory during multiple iterations, avoiding repeated disk reads. MapReduce must read and write data to disk every iteration, slowing it down.

❓ data_output

advanced

Identify the output of Spark RDD transformation

Given the following Spark code snippet, what is the output collected in the driver?

Apache Spark

rdd = sc.parallelize([1, 2, 3, 4, 5])
result = rdd.filter(lambda x: x % 2 == 0).map(lambda x: x * x).collect()
print(result)

A [1, 9, 25]

B [4, 16]

C [2, 4]

D [1, 4, 9, 16, 25]

💡 Hint

Filter keeps even numbers, then map squares them.

The filter keeps only even numbers [2,4]. Then map squares them to [4,16].

🧠 Conceptual

advanced

Which feature of Spark improves fault tolerance compared to MapReduce?

What feature of Apache Spark helps it recover from failures efficiently without writing intermediate data to disk like MapReduce?

A RDD lineage graph that tracks transformations to recompute lost data.

B Writing all intermediate data to HDFS for recovery.

C Using a single master node to store all data backups.

D Spark does not support fault tolerance.

💡 Hint

Think about how Spark knows how to rebuild lost data partitions.

Spark uses RDD lineage, a record of transformations, to recompute lost data partitions on failure without needing to write all intermediate data to disk.

🚀 Application

expert

Choosing Spark over MapReduce for a real-world task

You have a big dataset and need to run complex machine learning algorithms that require multiple passes over the data. Which reason best justifies choosing Apache Spark over MapReduce?

A Spark cannot handle iterative algorithms, so MapReduce is preferred.

B MapReduce is better because it writes intermediate results to disk, making it faster for iterative tasks.

C Spark's in-memory caching reduces repeated disk reads, speeding up iterative machine learning tasks.

D MapReduce uses in-memory caching which is faster for machine learning.

💡 Hint

Consider the nature of iterative algorithms and data access speed.

Iterative algorithms need to reuse data many times. Spark's in-memory caching avoids slow disk I/O, making it much faster than MapReduce for these tasks.