Spark vs Hadoop MapReduce in Apache Spark - Hands-On Comparison

Comparing Spark and Hadoop MapReduce Performance

📖 Scenario: You work as a data analyst at a company that processes large amounts of sales data. Your manager wants to understand how Apache Spark and Hadoop MapReduce handle data processing differently by comparing their performance on a simple task.

🎯 Goal: You will create a small dataset of sales, configure a threshold for filtering, apply both Spark and Hadoop MapReduce style filtering, and then output the filtered results to compare how Spark simplifies the process.

📋 What You'll Learn

Create a list of sales records with product names and sales amounts

Set a sales threshold to filter products with sales above this value

Use Apache Spark to filter the sales data based on the threshold

Print the filtered sales records

💡 Why This Matters

🌍 Real World

Filtering large sales datasets quickly to find products with high sales is common in retail analytics.

💼 Career

Data scientists and analysts use Spark to process big data efficiently, making this skill valuable for roles in data engineering and analytics.

1

Create the sales data list

Create a list called sales_data with these exact tuples: ("apple", 50), ("banana", 30), ("orange", 70), ("grape", 20), ("mango", 90).

Apache Spark

# Create the sales_data list with product and sales amount tuples
# Your code here

Hint

Use a list of tuples with product names as strings and sales amounts as integers.

2

Set the sales threshold

Create a variable called threshold and set it to 40.

Apache Spark

sales_data = [("apple", 50), ("banana", 30), ("orange", 70), ("grape", 20), ("mango", 90)]
# Set the threshold variable to 40
# Your code here

Hint

Just assign the number 40 to the variable named threshold.

3

Filter sales data using Spark

Use Apache Spark to create a SparkSession called spark. Then create a DataFrame called df from sales_data with columns product and sales. Filter df to keep rows where sales is greater than threshold and save it as filtered_df.

Apache Spark

sales_data = [("apple", 50), ("banana", 30), ("orange", 70), ("grape", 20), ("mango", 90)]
threshold = 40
# Import SparkSession and create spark
# Create DataFrame df from sales_data with columns 'product' and 'sales'
# Filter df where sales > threshold and save as filtered_df
# Your code here

Hint

Use SparkSession.builder.master("local").appName("SalesFilter").getOrCreate() to create spark. Use createDataFrame with columns ["product", "sales"]. Use filter with condition df.sales > threshold.

4

Show the filtered sales results

Use filtered_df.show() to display the filtered sales records.

Apache Spark

from pyspark.sql import SparkSession

sales_data = [("apple", 50), ("banana", 30), ("orange", 70), ("grape", 20), ("mango", 90)]
threshold = 40

spark = SparkSession.builder.master("local").appName("SalesFilter").getOrCreate()
df = spark.createDataFrame(sales_data, ["product", "sales"])
filtered_df = df.filter(df.sales > threshold)
# Show the filtered DataFrame
# Your code here

Hint

Call show() on filtered_df to print the filtered rows.

Practice

1. Which of the following is a key advantage of Apache Spark compared to Hadoop MapReduce?

easy

A. It cannot handle big data workloads.

B. It always uses disk storage for processing data.

C. It is slower but more stable than Spark.

D. It processes data faster by using memory instead of disk.

Solution

1. Step 1: Understand Spark's processing method

   Spark uses in-memory computation which speeds up data processing compared to disk-based methods.

2. Step 2: Compare with Hadoop MapReduce

   Hadoop MapReduce writes intermediate data to disk, making it slower but stable for large batch jobs.

3. Final Answer:

   It processes data faster by using memory instead of disk. -> Option D

4. Quick Check:

   Spark uses memory = Faster processing [OK]

Hint: Spark uses memory for speed; MapReduce uses disk [OK]

Common Mistakes:

• Thinking MapReduce is faster than Spark
• Assuming Spark always uses disk
• Believing Spark cannot handle big data

2. Which syntax correctly initializes a SparkContext in Python?

easy

A. sc = SparkContext.new()

B. sc = SparkContext()

C. sc = SparkContext.create()

D. sc = new SparkContext()

Solution

1. Step 1: Recall SparkContext initialization in PySpark

   The correct way is to call SparkContext() directly to create a new context.

2. Step 2: Check each option

   Options B and C use non-existent methods; D uses Java-like syntax not valid in Python.

3. Final Answer:

   sc = SparkContext() -> Option B

4. Quick Check:

   Correct Python syntax = sc = SparkContext() [OK]

Hint: Use SparkContext() directly in Python [OK]

Common Mistakes:

• Using Java syntax in Python
• Calling non-existent methods like new() or create()
• Forgetting to import SparkContext before use

3. Given the following code snippet in Spark, what will be the output?

rdd = sc.parallelize([1, 2, 3, 4])
result = rdd.map(lambda x: x * 2).collect()

medium

A. [1, 2, 3, 4]

B. [4, 8, 12, 16]

C. [2, 4, 6, 8]

D. [0, 2, 4, 6]

Solution

1. Step 1: Understand the map transformation

   The map applies the function x * 2 to each element in the RDD: 1*2=2, 2*2=4, 3*2=6, 4*2=8.

2. Step 2: Collect the results

   collect() gathers all transformed elements into a list: [2, 4, 6, 8].

3. Final Answer:

   [2, 4, 6, 8] -> Option C

4. Quick Check:

   Each element doubled = [2,4,6,8] [OK]

Hint: map applies function to each element [OK]

Common Mistakes:

• Confusing map with filter
• Forgetting to call collect() to get results
• Multiplying elements incorrectly

4. Identify the error in this Hadoop MapReduce pseudo-code snippet:

map(key, value):
  emit(value, key)

reduce(key, values):
  sum = 0
  for v in values:
    sum += v
  emit(key, sum)

medium

A. The reduce function sums values but values are keys from map output.

B. The map function emits key and value in wrong order.

C. The reduce function tries to sum keys instead of values.

D. There is no error; code is correct.

Solution

1. Step 1: Analyze map output

   Map emits (value, key), so keys in reduce are original values, and values are original keys.

2. Step 2: Check reduce summation

   Reduce sums 'values', but these are original keys, likely non-numeric or not intended for summation.

3. Final Answer:

   The reduce function sums values but values are keys from map output. -> Option A

4. Quick Check:

   Reduce sums wrong data = error [OK]

Hint: Check what map emits vs what reduce receives [OK]

Common Mistakes:

• Assuming map output keys and values are unchanged
• Summing keys instead of values
• Ignoring data types of reduce inputs

5. You have a large dataset to process repeatedly with iterative algorithms. Which framework is better and why?

hard

A. Apache Spark, because it caches data in memory for faster iterative processing.

B. Apache Spark, because it uses disk storage to handle iterative jobs.

C. Hadoop MapReduce, because it is easier to program for iterative tasks.

D. Hadoop MapReduce, because it writes intermediate results to disk for reliability.

Solution

1. Step 1: Understand iterative algorithm needs

   Iterative algorithms reuse data multiple times, so fast access is important.

2. Step 2: Compare Spark and MapReduce for iteration

   Spark caches data in memory, speeding up repeated access; MapReduce writes to disk each time, slowing down iteration.

3. Final Answer:

   Apache Spark, because it caches data in memory for faster iterative processing. -> Option A

4. Quick Check:

   Memory caching = faster iteration [OK]

Hint: Use Spark for fast iterative jobs with memory caching [OK]

Common Mistakes:

• Choosing MapReduce for iterative speed
• Confusing disk use with speed
• Thinking Spark uses disk for iteration