Spark vs Hadoop MapReduce in Apache Spark - Practice Questions

Challenge - 5 Problems

🧠 Conceptual

intermediate

Key Difference in Data Processing Models

Which statement best describes the main difference between Apache Spark and Hadoop MapReduce in how they process data?

A Spark processes data in-memory for faster computation, while Hadoop MapReduce writes intermediate results to disk.

B Hadoop MapReduce processes data in-memory, while Spark writes intermediate results to disk.

C Both Spark and Hadoop MapReduce process data only on disk without using memory.

D Spark and Hadoop MapReduce both use in-memory processing exclusively.

💡 Hint

Think about how each framework handles intermediate data during processing.

Spark keeps data in memory between steps, making it faster. Hadoop MapReduce writes data to disk after each step, which slows it down.

❓ Predict Output

intermediate

Output of Spark RDD Transformation

What will be the output of this Spark code snippet?

Apache Spark

rdd = sc.parallelize([1, 2, 3, 4])
result = rdd.map(lambda x: x * 2).filter(lambda x: x > 4).collect()
print(result)

A [4, 6, 8]

B [2, 4]

C [6, 8]

D [8]

💡 Hint

First multiply each number by 2, then keep only those greater than 4.

Mapping doubles each number: [2,4,6,8]. Filtering keeps numbers >4: [6,8].

❓ data_output

advanced

Memory Usage Comparison

Given a large dataset, which framework will typically use less disk I/O during iterative machine learning tasks?

A Apache Spark, because it caches data in memory across iterations.

B Hadoop MapReduce, because it writes intermediate results to disk after each iteration.

C Both use the same amount of disk I/O regardless of the task.

D Neither uses disk I/O during iterative tasks.

💡 Hint

Consider how iterative tasks benefit from caching.

Spark caches data in memory, reducing disk reads and writes during iterations. Hadoop MapReduce writes to disk each time, increasing disk I/O.

🔧 Debug

advanced

Identifying Cause of Slow Job

A Spark job is running slower than expected compared to a similar Hadoop MapReduce job. Which of the following is the most likely cause?

A The Spark job is using too much disk space for intermediate data.

B The Spark job is not caching data and repeatedly recomputes RDDs.

C Spark always runs slower than Hadoop MapReduce for all jobs.

D The Hadoop job is using in-memory caching, causing overhead.

💡 Hint

Think about how Spark handles repeated computations without caching.

If Spark does not cache data, it recomputes RDDs each time they are used, causing slowdowns. Hadoop MapReduce writes to disk but does not recompute.

🚀 Application

expert

Choosing Framework for Real-Time Data Processing

You need to process streaming data with low latency for real-time analytics. Which framework is best suited for this task?

A Hadoop MapReduce, because it processes batch data efficiently.

B Hadoop MapReduce, because it supports real-time stream processing natively.

C Neither, because both are designed only for batch processing.

D Apache Spark, because it supports in-memory stream processing with Spark Streaming.

💡 Hint

Consider which framework supports streaming and low latency.

Spark Streaming processes data in near real-time using in-memory computations. Hadoop MapReduce is designed for batch jobs and is not suitable for real-time streaming.

Practice

1. Which of the following is a key advantage of Apache Spark compared to Hadoop MapReduce?

easy

A. It cannot handle big data workloads.

B. It always uses disk storage for processing data.

C. It is slower but more stable than Spark.

D. It processes data faster by using memory instead of disk.

Solution

1. Step 1: Understand Spark's processing method

   Spark uses in-memory computation which speeds up data processing compared to disk-based methods.

2. Step 2: Compare with Hadoop MapReduce

   Hadoop MapReduce writes intermediate data to disk, making it slower but stable for large batch jobs.

3. Final Answer:

   It processes data faster by using memory instead of disk. -> Option D

4. Quick Check:

   Spark uses memory = Faster processing [OK]

Hint: Spark uses memory for speed; MapReduce uses disk [OK]

Common Mistakes:

• Thinking MapReduce is faster than Spark
• Assuming Spark always uses disk
• Believing Spark cannot handle big data

2. Which syntax correctly initializes a SparkContext in Python?

easy

A. sc = SparkContext.new()

B. sc = SparkContext()

C. sc = SparkContext.create()

D. sc = new SparkContext()

Solution

1. Step 1: Recall SparkContext initialization in PySpark

   The correct way is to call SparkContext() directly to create a new context.

2. Step 2: Check each option

   Options B and C use non-existent methods; D uses Java-like syntax not valid in Python.

3. Final Answer:

   sc = SparkContext() -> Option B

4. Quick Check:

   Correct Python syntax = sc = SparkContext() [OK]

Hint: Use SparkContext() directly in Python [OK]

Common Mistakes:

• Using Java syntax in Python
• Calling non-existent methods like new() or create()
• Forgetting to import SparkContext before use

3. Given the following code snippet in Spark, what will be the output?

rdd = sc.parallelize([1, 2, 3, 4])
result = rdd.map(lambda x: x * 2).collect()

medium

A. [1, 2, 3, 4]

B. [4, 8, 12, 16]

C. [2, 4, 6, 8]

D. [0, 2, 4, 6]

Solution

1. Step 1: Understand the map transformation

   The map applies the function x * 2 to each element in the RDD: 1*2=2, 2*2=4, 3*2=6, 4*2=8.

2. Step 2: Collect the results

   collect() gathers all transformed elements into a list: [2, 4, 6, 8].

3. Final Answer:

   [2, 4, 6, 8] -> Option C

4. Quick Check:

   Each element doubled = [2,4,6,8] [OK]

Hint: map applies function to each element [OK]

Common Mistakes:

• Confusing map with filter
• Forgetting to call collect() to get results
• Multiplying elements incorrectly

4. Identify the error in this Hadoop MapReduce pseudo-code snippet:

map(key, value):
  emit(value, key)

reduce(key, values):
  sum = 0
  for v in values:
    sum += v
  emit(key, sum)

medium

A. The reduce function sums values but values are keys from map output.

B. The map function emits key and value in wrong order.

C. The reduce function tries to sum keys instead of values.

D. There is no error; code is correct.

Solution

1. Step 1: Analyze map output

   Map emits (value, key), so keys in reduce are original values, and values are original keys.

2. Step 2: Check reduce summation

   Reduce sums 'values', but these are original keys, likely non-numeric or not intended for summation.

3. Final Answer:

   The reduce function sums values but values are keys from map output. -> Option A

4. Quick Check:

   Reduce sums wrong data = error [OK]

Hint: Check what map emits vs what reduce receives [OK]

Common Mistakes:

• Assuming map output keys and values are unchanged
• Summing keys instead of values
• Ignoring data types of reduce inputs

5. You have a large dataset to process repeatedly with iterative algorithms. Which framework is better and why?

hard

A. Apache Spark, because it caches data in memory for faster iterative processing.

B. Apache Spark, because it uses disk storage to handle iterative jobs.

C. Hadoop MapReduce, because it is easier to program for iterative tasks.

D. Hadoop MapReduce, because it writes intermediate results to disk for reliability.

Solution

1. Step 1: Understand iterative algorithm needs

   Iterative algorithms reuse data multiple times, so fast access is important.

2. Step 2: Compare Spark and MapReduce for iteration

   Spark caches data in memory, speeding up repeated access; MapReduce writes to disk each time, slowing down iteration.

3. Final Answer:

   Apache Spark, because it caches data in memory for faster iterative processing. -> Option A

4. Quick Check:

   Memory caching = faster iteration [OK]

Hint: Use Spark for fast iterative jobs with memory caching [OK]

Common Mistakes:

• Choosing MapReduce for iterative speed
• Confusing disk use with speed
• Thinking Spark uses disk for iteration