Why Spark replaced MapReduce for big data in Apache Spark - Quick Recap

Recall & Review

beginner

What is the main limitation of MapReduce that Spark addresses?

MapReduce writes data to disk after each step, which slows down processing. Spark keeps data in memory, making it much faster.

beginner

How does Spark improve speed compared to MapReduce?

Spark uses in-memory computing, which means it stores data in RAM during processing instead of writing to disk after every step.

intermediate

What kind of tasks is Spark better suited for compared to MapReduce?

Spark is better for iterative tasks like machine learning and interactive data analysis because it can reuse data in memory.

beginner

Why is Spark considered more user-friendly than MapReduce?

Spark provides easy-to-use APIs in multiple languages like Python, Java, and Scala, while MapReduce requires writing complex Java code.

intermediate

What is a key architectural difference between Spark and MapReduce?

MapReduce follows a strict two-stage process (map and reduce) with disk writes in between, while Spark uses a directed acyclic graph (DAG) to optimize execution and reduce disk I/O.

What does Spark use to speed up data processing compared to MapReduce?

A In-memory computing

B More disk writes

C Slower network

D Single-threaded processing

Spark speeds up processing by keeping data in memory, avoiding slow disk writes after each step.

Which programming languages does Spark support with easy APIs?

A HTML and CSS

B Only Java

C C++ and Ruby

D Python, Java, Scala

Spark offers APIs in Python, Java, and Scala, making it accessible to many developers.

Why is Spark better for machine learning tasks than MapReduce?

A It writes data to disk more often

B It can reuse data in memory for multiple iterations

C It uses only one CPU core

D It does not support iterative tasks

Machine learning often requires repeating calculations; Spark keeps data in memory to speed this up.

What is a major drawback of MapReduce compared to Spark?

A It uses a DAG for execution

B It processes data only in memory

C It writes intermediate data to disk after each step

D It supports too many programming languages

MapReduce writes data to disk after each map and reduce step, which slows down processing.

What execution model does Spark use to optimize tasks?

A Directed Acyclic Graph (DAG)

B Linear pipeline

C Single map-reduce step

D Random execution

Spark uses a DAG to plan and optimize task execution, reducing unnecessary disk I/O.

Explain why Spark replaced MapReduce for big data processing.

Think about speed, usability, and how data is handled during processing.

Describe the architectural differences between Spark and MapReduce.

Focus on how tasks are planned and how data moves during processing.

Practice

1. Why is Apache Spark generally faster than MapReduce for big data processing?

easy

A. Because Spark processes data in memory instead of writing to disk after each step

B. Because Spark uses more disk space for intermediate data

C. Because Spark only supports batch processing

D. Because Spark requires less memory than MapReduce

Solution

1. Step 1: Understand MapReduce data processing

   MapReduce writes intermediate results to disk after each map and reduce step, which slows down processing.

2. Step 2: Understand Spark's in-memory processing

   Spark keeps data in memory during processing, reducing slow disk I/O and speeding up tasks.

3. Final Answer:

   Because Spark processes data in memory instead of writing to disk after each step -> Option A

4. Quick Check:

   In-memory processing = Faster Spark [OK]

Hint: Remember Spark keeps data in RAM, MapReduce uses disk [OK]

Common Mistakes:

• Thinking Spark uses more disk space than MapReduce
• Believing Spark only supports batch processing
• Assuming Spark requires less memory always

2. Which of the following is a correct reason why Spark is easier to program than MapReduce?

easy

A. Spark supports multiple programming languages like Python, Java, and Scala

B. Spark requires writing complex XML configuration files

C. Spark only supports Java programming

D. Spark does not support any APIs

Solution

1. Step 1: Identify Spark's language support

   Spark supports multiple languages such as Python, Java, Scala, and R, making it accessible to many developers.

2. Step 2: Compare with MapReduce programming

   MapReduce programming is mostly Java-based and requires more boilerplate code, making it harder to write.

3. Final Answer:

   Spark supports multiple programming languages like Python, Java, and Scala -> Option A

4. Quick Check:

   Multi-language support = Easier Spark programming [OK]

Hint: Spark supports Python and Scala, MapReduce mostly Java [OK]

Common Mistakes:

• Thinking Spark requires XML configs
• Believing Spark only supports Java
• Assuming Spark has no APIs

3. Consider this Spark code snippet:

val data = spark.sparkContext.parallelize(1 to 5)
val result = data.map(x => x * 2).collect()

What will be the output of result?

medium

A. An error because of syntax

B. [1, 2, 3, 4, 5]

C. [0, 2, 4, 6, 8]

D. [2, 4, 6, 8, 10]

Solution

1. Step 1: Understand the input data

   The data RDD contains numbers from 1 to 5 inclusive.

2. Step 2: Apply the map transformation

   Each element is multiplied by 2, so 1*2=2, 2*2=4, ..., 5*2=10.

3. Final Answer:

   [2, 4, 6, 8, 10] -> Option D

4. Quick Check:

   Multiply each number by 2 = [2,4,6,8,10] [OK]

Hint: Map doubles each number, so output doubles input list [OK]

Common Mistakes:

• Confusing input and output values
• Thinking syntax causes error
• Assuming map adds instead of multiplies

4. This Spark code throws an error:

val data = spark.sparkContext.parallelize(1 to 3)
val result = data.reduce((a, b) => a + b + c)

What is the cause of the error?

medium

A. Reduce function requires three parameters

B. Variable 'c' is not defined in the reduce function

C. parallelize method is incorrect

D. SparkContext is not initialized

Solution

1. Step 1: Analyze the reduce function parameters

   The reduce function takes two parameters, here named 'a' and 'b'.

2. Step 2: Identify the error in the function body

   The function tries to use 'c' which is not defined anywhere, causing a compilation error.

3. Final Answer:

   Variable 'c' is not defined in the reduce function -> Option B

4. Quick Check:

   Undefined variable 'c' causes error [OK]

Hint: Check all variables used inside functions are defined [OK]

Common Mistakes:

• Thinking reduce needs three parameters
• Assuming parallelize is wrong
• Believing SparkContext is uninitialized

5. You want to perform real-time data processing and machine learning on big data. Why would Spark be a better choice than MapReduce?

hard

A. Spark cannot handle machine learning tasks

B. MapReduce supports real-time streaming better than Spark

C. Spark supports in-memory streaming and ML libraries, enabling faster real-time analytics

D. MapReduce processes data only in memory

Solution

1. Step 1: Understand Spark's capabilities

   Spark has built-in modules like Spark Streaming for real-time data and MLlib for machine learning.

2. Step 2: Compare with MapReduce limitations

   MapReduce is batch-oriented and does not support real-time streaming or integrated ML libraries efficiently.

3. Final Answer:

   Spark supports in-memory streaming and ML libraries, enabling faster real-time analytics -> Option C

4. Quick Check:

   Streaming + ML support = Spark advantage [OK]

Hint: Spark = streaming + ML; MapReduce = batch only [OK]

Common Mistakes:

• Believing MapReduce supports real-time streaming
• Thinking Spark cannot do machine learning
• Assuming MapReduce processes data in memory