Why Spark replaced MapReduce for big data in Apache Spark - See It in Action

Why Spark replaced MapReduce for big data

📖 Scenario: Imagine you work at a company that processes huge amounts of data every day. You used to use a tool called MapReduce, but now your team wants to switch to a newer tool called Apache Spark. You want to understand why Spark is better for big data tasks.

🎯 Goal: Build a simple example to compare how MapReduce and Spark handle data processing, and see why Spark is faster and easier to use.

📋 What You'll Learn

Create a small dataset as a list of numbers

Set a threshold value to filter numbers

Use Spark's filter function to process the data

Print the final filtered list to see the result

💡 Why This Matters

🌍 Real World

Companies use Spark to quickly analyze large data sets like user logs, sales data, or sensor data to make fast decisions.

💼 Career

Knowing why Spark replaced MapReduce helps you understand modern big data tools used in data engineering and data science jobs.

1

Create the initial data list

Create a list called data with these exact numbers: [10, 20, 30, 40, 50]

Apache Spark

# Create a list called data with the numbers 10, 20, 30, 40, 50
# Your code here

Need a hint?

Use square brackets to create a list with the numbers given.

2

Set a threshold value

Create a variable called threshold and set it to 25

Apache Spark

data = [10, 20, 30, 40, 50]
# Create a variable called threshold and set it to 25
# Your code here

Need a hint?

Just assign the number 25 to the variable threshold.

3

Use Spark to filter data

Use Spark's filter function to create a new RDD from data called filtered_data that only keeps numbers greater than threshold. Use sc.parallelize(data) to create the RDD.

Apache Spark

data = [10, 20, 30, 40, 50]
threshold = 25
# Use Spark to create filtered_data with numbers greater than threshold
# Your code here

Need a hint?

Use sc.parallelize(data) to create an RDD, then use filter with a lambda function to keep numbers greater than threshold.

4

Print the filtered data

Print the list of numbers in filtered_data by collecting the RDD and printing it.

Apache Spark

data = [10, 20, 30, 40, 50]
threshold = 25
filtered_data = sc.parallelize(data).filter(lambda x: x > threshold)
# Print the collected filtered_data
# Your code here

Need a hint?

Use filtered_data.collect() to get the list and print it.