What is an RDD (Resilient Distributed Dataset) in Apache Spark - Hands-On Activity

Understanding RDDs (Resilient Distributed Datasets) in Apache Spark

📖 Scenario: Imagine you work at a company that processes large amounts of data spread across many computers. You want to learn how Apache Spark helps handle this data efficiently using a special data structure called an RDD.

🎯 Goal: You will create a simple RDD, configure a filter condition, apply the filter to keep only certain data, and then display the filtered results. This will help you understand what an RDD is and how it works.

📋 What You'll Learn

Create an RDD from a list of numbers

Create a filter threshold variable

Use the filter transformation on the RDD

Collect and print the filtered results

💡 Why This Matters

🌍 Real World

Companies use RDDs in Apache Spark to process big data quickly and reliably across many computers, such as analyzing logs, user data, or sensor information.

💼 Career

Understanding RDDs is essential for data engineers and data scientists working with big data tools like Apache Spark to build scalable data pipelines and analytics.

1

Create an RDD from a list of numbers

Create a SparkContext called sc and then create an RDD called numbers_rdd from the list [1, 2, 3, 4, 5, 6] using sc.parallelize().

Apache Spark

from pyspark import SparkContext
sc = SparkContext.getOrCreate()
# Create an RDD called numbers_rdd from the list [1, 2, 3, 4, 5, 6]
# Your code here

Need a hint?

Use sc.parallelize() to create an RDD from a Python list.

2

Set a filter threshold

Create a variable called threshold and set it to 3. This will be used to filter numbers greater than this value.

Apache Spark

from pyspark import SparkContext
sc = SparkContext.getOrCreate()
numbers_rdd = sc.parallelize([1, 2, 3, 4, 5, 6])
# Create a variable called threshold and set it to 3
# Your code here

Need a hint?

Just assign the number 3 to the variable threshold.

3

Filter the RDD using the threshold

Create a new RDD called filtered_rdd by applying the filter() transformation on numbers_rdd. Keep only numbers greater than threshold.

Apache Spark

from pyspark import SparkContext
sc = SparkContext.getOrCreate()
numbers_rdd = sc.parallelize([1, 2, 3, 4, 5, 6])
threshold = 3
# Create filtered_rdd by filtering numbers_rdd to keep numbers greater than threshold
# Your code here

Need a hint?

Use filter() with a lambda function that returns True for numbers greater than threshold.

4

Collect and print the filtered results

Use filtered_rdd.collect() to get the filtered numbers as a list and print the result.

Apache Spark

from pyspark import SparkContext
sc = SparkContext.getOrCreate()
numbers_rdd = sc.parallelize([1, 2, 3, 4, 5, 6])
threshold = 3
filtered_rdd = numbers_rdd.filter(lambda x: x > threshold)
# Collect and print the filtered results
# Your code here

Need a hint?

Use print(filtered_rdd.collect()) to display the filtered list.