Spark vs Hadoop MapReduce in Apache Spark - Visual Side-by-Side Comparison

Concept Flow - Spark vs Hadoop MapReduce

Input Data

↓

Map Phase

↓

Shuffle & Sort

↓

Reduce Phase

↓

Write Output

↓

Final Result

Shows the flow of data processing in Hadoop MapReduce versus Apache Spark, highlighting disk-based steps versus in-memory operations.

Execution Sample

Apache Spark

data = sc.textFile('data.txt')
words = data.flatMap(lambda line: line.split())
wordCounts = words.map(lambda w: (w,1)).reduceByKey(lambda a,b: a+b)
wordCounts.collect()

Counts words in a text file using Spark's in-memory transformations and actions.

Execution Table

Step	Operation	Data Location	Action	Result
1	Read data from 'data.txt'	Disk	Load text file into RDD	RDD with lines of text
2	flatMap split lines	Memory	Split each line into words	RDD with words
3	map to (word,1)	Memory	Create pairs for counting	RDD of (word,1) tuples
4	reduceByKey sum counts	Memory	Sum counts for each word	RDD of (word, total_count)
5	collect results	Memory to Driver	Bring results to driver program	List of (word, count) pairs

💡 All transformations are lazy; action 'collect' triggers execution and returns final counts.

Variable Tracker

Variable	Start	After Step 1	After Step 2	After Step 3	After Step 4	Final
data	empty	RDD(lines of text)	RDD(lines of text)	RDD(lines of text)	RDD(lines of text)	RDD(lines of text)
words	undefined	undefined	RDD(words)	RDD(words)	RDD(words)	RDD(words)
wordCounts	undefined	undefined	undefined	RDD((word,1))	RDD((word, total_count))	RDD((word, total_count))
result	undefined	undefined	undefined	undefined	undefined	List((word, count))

Key Moments - 3 Insights

Why does Spark run faster than Hadoop MapReduce?

Spark keeps data in memory during processing (see execution_table steps 2-4), avoiding slow disk reads/writes after each step, unlike Hadoop MapReduce which writes intermediate results to disk.

What triggers the actual computation in Spark?

Transformations like flatMap and map are lazy and do not run immediately. The action 'collect' (step 5 in execution_table) triggers all previous steps to execute.

Why does Hadoop MapReduce have shuffle and sort between map and reduce?

Because Hadoop writes intermediate data to disk and needs to organize it for reducers, causing extra disk I/O and latency, unlike Spark which keeps data in memory.

Visual Quiz - 3 Questions

Look at the execution_table, at which step does Spark actually perform the word count aggregation?

A Step 2: flatMap split lines

B Step 4: reduceByKey sum counts

C Step 3: map to (word,1)

D Step 5: collect results

💡 Hint

Check the 'Action' column in execution_table for where counts are summed.

According to variable_tracker, what is the state of 'words' after Step 3?

A RDD((word,1))

B RDD(lines of text)

C RDD(words)

D List((word, count))

💡 Hint

Look at the 'words' row and the 'After Step 3' column in variable_tracker.

If we remove the 'collect' action, what happens to the execution according to the exit_note?

A No computation runs, transformations are lazy

B Only map runs, reduce does not

C All transformations run immediately

D Data is written to disk

💡 Hint

Refer to the exit_note about lazy transformations and action triggering.

Concept Snapshot

Spark vs Hadoop MapReduce:
- Hadoop MapReduce reads/writes intermediate data to disk.
- Spark keeps data in memory for faster processing.
- Spark uses lazy transformations; actions trigger execution.
- MapReduce has map, shuffle/sort, reduce phases.
- Spark uses RDD/DataFrame transformations and actions.
- Spark is generally faster for iterative and interactive tasks.

Full Transcript

This visual execution compares Apache Spark and Hadoop MapReduce data processing flows. Hadoop MapReduce reads input from disk, maps data, shuffles and sorts intermediate results on disk, then reduces and writes output. Spark reads data into memory as RDDs, applies transformations like flatMap and map lazily, then performs reduceByKey in memory. The action 'collect' triggers execution and returns results. Variables like 'data', 'words', and 'wordCounts' change state step-by-step in memory. Key points include Spark's speed advantage due to in-memory processing and lazy evaluation. The quiz tests understanding of when aggregation happens, variable states, and the role of actions in Spark.

Practice

1. Which of the following is a key advantage of Apache Spark compared to Hadoop MapReduce?

easy

A. It cannot handle big data workloads.

B. It always uses disk storage for processing data.

C. It is slower but more stable than Spark.

D. It processes data faster by using memory instead of disk.

Solution

1. Step 1: Understand Spark's processing method

   Spark uses in-memory computation which speeds up data processing compared to disk-based methods.

2. Step 2: Compare with Hadoop MapReduce

   Hadoop MapReduce writes intermediate data to disk, making it slower but stable for large batch jobs.

3. Final Answer:

   It processes data faster by using memory instead of disk. -> Option D

4. Quick Check:

   Spark uses memory = Faster processing [OK]

Hint: Spark uses memory for speed; MapReduce uses disk [OK]

Common Mistakes:

• Thinking MapReduce is faster than Spark
• Assuming Spark always uses disk
• Believing Spark cannot handle big data

2. Which syntax correctly initializes a SparkContext in Python?

easy

A. sc = SparkContext.new()

B. sc = SparkContext()

C. sc = SparkContext.create()

D. sc = new SparkContext()

Solution

1. Step 1: Recall SparkContext initialization in PySpark

   The correct way is to call SparkContext() directly to create a new context.

2. Step 2: Check each option

   Options B and C use non-existent methods; D uses Java-like syntax not valid in Python.

3. Final Answer:

   sc = SparkContext() -> Option B

4. Quick Check:

   Correct Python syntax = sc = SparkContext() [OK]

Hint: Use SparkContext() directly in Python [OK]

Common Mistakes:

• Using Java syntax in Python
• Calling non-existent methods like new() or create()
• Forgetting to import SparkContext before use

3. Given the following code snippet in Spark, what will be the output?

rdd = sc.parallelize([1, 2, 3, 4])
result = rdd.map(lambda x: x * 2).collect()

medium

A. [1, 2, 3, 4]

B. [4, 8, 12, 16]

C. [2, 4, 6, 8]

D. [0, 2, 4, 6]

Solution

1. Step 1: Understand the map transformation

   The map applies the function x * 2 to each element in the RDD: 1*2=2, 2*2=4, 3*2=6, 4*2=8.

2. Step 2: Collect the results

   collect() gathers all transformed elements into a list: [2, 4, 6, 8].

3. Final Answer:

   [2, 4, 6, 8] -> Option C

4. Quick Check:

   Each element doubled = [2,4,6,8] [OK]

Hint: map applies function to each element [OK]

Common Mistakes:

• Confusing map with filter
• Forgetting to call collect() to get results
• Multiplying elements incorrectly

4. Identify the error in this Hadoop MapReduce pseudo-code snippet:

map(key, value):
  emit(value, key)

reduce(key, values):
  sum = 0
  for v in values:
    sum += v
  emit(key, sum)

medium

A. The reduce function sums values but values are keys from map output.

B. The map function emits key and value in wrong order.

C. The reduce function tries to sum keys instead of values.

D. There is no error; code is correct.

Solution

1. Step 1: Analyze map output

   Map emits (value, key), so keys in reduce are original values, and values are original keys.

2. Step 2: Check reduce summation

   Reduce sums 'values', but these are original keys, likely non-numeric or not intended for summation.

3. Final Answer:

   The reduce function sums values but values are keys from map output. -> Option A

4. Quick Check:

   Reduce sums wrong data = error [OK]

Hint: Check what map emits vs what reduce receives [OK]

Common Mistakes:

• Assuming map output keys and values are unchanged
• Summing keys instead of values
• Ignoring data types of reduce inputs

5. You have a large dataset to process repeatedly with iterative algorithms. Which framework is better and why?

hard

A. Apache Spark, because it caches data in memory for faster iterative processing.

B. Apache Spark, because it uses disk storage to handle iterative jobs.

C. Hadoop MapReduce, because it is easier to program for iterative tasks.

D. Hadoop MapReduce, because it writes intermediate results to disk for reliability.

Solution

1. Step 1: Understand iterative algorithm needs

   Iterative algorithms reuse data multiple times, so fast access is important.

2. Step 2: Compare Spark and MapReduce for iteration

   Spark caches data in memory, speeding up repeated access; MapReduce writes to disk each time, slowing down iteration.

3. Final Answer:

   Apache Spark, because it caches data in memory for faster iterative processing. -> Option A

4. Quick Check:

   Memory caching = faster iteration [OK]

Hint: Use Spark for fast iterative jobs with memory caching [OK]

Common Mistakes:

• Choosing MapReduce for iterative speed
• Confusing disk use with speed
• Thinking Spark uses disk for iteration