What is an RDD (Resilient Distributed Dataset) in Apache Spark - Hands-On Activity

Understanding RDDs (Resilient Distributed Datasets) in Apache Spark

📖 Scenario: Imagine you work at a company that processes large amounts of data spread across many computers. You want to learn how Apache Spark helps handle this data efficiently using a special data structure called an RDD.

🎯 Goal: You will create a simple RDD, configure a filter condition, apply the filter to keep only certain data, and then display the filtered results. This will help you understand what an RDD is and how it works.

📋 What You'll Learn

Create an RDD from a list of numbers

Create a filter threshold variable

Use the filter transformation on the RDD

Collect and print the filtered results

💡 Why This Matters

🌍 Real World

Companies use RDDs in Apache Spark to process big data quickly and reliably across many computers, such as analyzing logs, user data, or sensor information.

💼 Career

Understanding RDDs is essential for data engineers and data scientists working with big data tools like Apache Spark to build scalable data pipelines and analytics.

1

Create an RDD from a list of numbers

Create a SparkContext called sc and then create an RDD called numbers_rdd from the list [1, 2, 3, 4, 5, 6] using sc.parallelize().

Apache Spark

from pyspark import SparkContext
sc = SparkContext.getOrCreate()
# Create an RDD called numbers_rdd from the list [1, 2, 3, 4, 5, 6]
# Your code here

Hint

Use sc.parallelize() to create an RDD from a Python list.

2

Set a filter threshold

Create a variable called threshold and set it to 3. This will be used to filter numbers greater than this value.

Apache Spark

from pyspark import SparkContext
sc = SparkContext.getOrCreate()
numbers_rdd = sc.parallelize([1, 2, 3, 4, 5, 6])
# Create a variable called threshold and set it to 3
# Your code here

Hint

Just assign the number 3 to the variable threshold.

3

Filter the RDD using the threshold

Create a new RDD called filtered_rdd by applying the filter() transformation on numbers_rdd. Keep only numbers greater than threshold.

Apache Spark

from pyspark import SparkContext
sc = SparkContext.getOrCreate()
numbers_rdd = sc.parallelize([1, 2, 3, 4, 5, 6])
threshold = 3
# Create filtered_rdd by filtering numbers_rdd to keep numbers greater than threshold
# Your code here

Hint

Use filter() with a lambda function that returns True for numbers greater than threshold.

4

Collect and print the filtered results

Use filtered_rdd.collect() to get the filtered numbers as a list and print the result.

Apache Spark

from pyspark import SparkContext
sc = SparkContext.getOrCreate()
numbers_rdd = sc.parallelize([1, 2, 3, 4, 5, 6])
threshold = 3
filtered_rdd = numbers_rdd.filter(lambda x: x > threshold)
# Collect and print the filtered results
# Your code here

Hint

Use print(filtered_rdd.collect()) to display the filtered list.

Practice

1. What is an RDD in Apache Spark?

easy

A. A distributed collection of data that can be processed in parallel

B. A type of database used for storing structured data

C. A tool for visualizing data in charts and graphs

D. A programming language for big data analysis

Solution

1. Step 1: Understand the definition of RDD

   An RDD is a core data structure in Apache Spark that represents a distributed collection of objects.

2. Step 2: Identify the correct description

   Among the options, only A distributed collection of data that can be processed in parallel correctly describes RDD as a distributed collection processed in parallel.

3. Final Answer:

   A distributed collection of data that can be processed in parallel -> Option A

4. Quick Check:

   RDD = distributed data collection [OK]

Hint: RDDs are distributed data collections for parallel processing [OK]

Common Mistakes:

• Confusing RDD with databases
• Thinking RDD is a visualization tool
• Assuming RDD is a programming language

2. Which of the following is the correct way to create an RDD from a list in PySpark?

easy

A. rdd = sc.createRDD([1, 2, 3, 4])

B. rdd = SparkContext.create([1, 2, 3, 4])

C. rdd = sc.parallelize([1, 2, 3, 4])

D. rdd = spark.parallelize([1, 2, 3, 4])

Solution

1. Step 1: Recall the method to create RDD from a list

   In PySpark, the SparkContext object (usually named sc) has a method called parallelize to create an RDD from a Python list.

2. Step 2: Check each option for correctness

   rdd = sc.parallelize([1, 2, 3, 4]) uses sc.parallelize correctly. Options B and C use incorrect method names. rdd = spark.parallelize([1, 2, 3, 4]) uses spark, which is a SparkSession, not SparkContext, so parallelize is not directly available there.

3. Final Answer:

   rdd = sc.parallelize([1, 2, 3, 4]) -> Option C

4. Quick Check:

   Use sc.parallelize to create RDD from list [OK]

Hint: Use sc.parallelize(list) to create an RDD [OK]

Common Mistakes:

• Using SparkSession instead of SparkContext
• Wrong method names like create or createRDD
• Confusing RDD creation with DataFrame creation

3. What will be the output of the following PySpark code?

rdd = sc.parallelize([1, 2, 3, 4])
result = rdd.map(lambda x: x * 2).collect()

medium

A. [1, 2, 3, 4]

B. [2, 4, 6, 8]

C. [0, 2, 4, 6]

D. Error: map function not found

Solution

1. Step 1: Understand the map transformation on RDD

   The map function applies the given lambda function to each element of the RDD. Here, each element is multiplied by 2.

2. Step 2: Calculate the result of map and collect

   Input list is [1, 2, 3, 4]. Multiplying each by 2 gives [2, 4, 6, 8]. collect() gathers the results into a list.

3. Final Answer:

   [2, 4, 6, 8] -> Option B

4. Quick Check:

   map doubles values, collect returns list [OK]

Hint: map applies function to each element, collect returns list [OK]

Common Mistakes:

• Thinking map returns original list
• Confusing map with filter
• Expecting map to modify RDD in place

4. Identify the error in the following PySpark code snippet:

rdd = sc.parallelize([1, 2, 3, 4])
result = rdd.filter(lambda x: x > 2).map(x * 2).collect()

medium

A. The map function syntax is incorrect

B. The filter function is used incorrectly

C. The collect function cannot be used here

D. No error, code runs fine

Solution

1. Step 1: Check the filter function usage

   The filter function is correctly used with a lambda to keep elements greater than 2.

2. Step 2: Check the map function syntax

   map expects a function as argument. Here, x * 2 is an expression, not a function. It should be lambda x: x * 2.

3. Final Answer:

   The map function syntax is incorrect -> Option A

4. Quick Check:

   map needs a function, not an expression [OK]

Hint: map requires a function like lambda x: x * 2 [OK]

Common Mistakes:

• Passing expression instead of function to map
• Misusing filter syntax
• Thinking collect causes error

5. You have an RDD of numbers and want to create a new RDD containing only the squares of even numbers. Which PySpark code correctly achieves this?

hard

A. rdd.map(lambda x: x % 2 == 0).filter(lambda x: x ** 2)

B. rdd.map(lambda x: x ** 2).filter(lambda x: x % 2 != 0)

C. rdd.filter(lambda x: x ** 2 % 2 != 0).map(lambda x: x ** 2)

D. rdd.filter(lambda x: x % 2 == 0).map(lambda x: x ** 2)

Solution

1. Step 1: Filter even numbers from the RDD

   Use filter with lambda x: x % 2 == 0 to keep only even numbers.

2. Step 2: Map the filtered RDD to squares

   Apply map with lambda x: x ** 2 to square each even number.

3. Final Answer:

   rdd.filter(lambda x: x % 2 == 0).map(lambda x: x ** 2) -> Option D

4. Quick Check:

   Filter evens first, then square [OK]

Hint: Filter evens first, then map to square [OK]

Common Mistakes:

• Mapping before filtering changes logic
• Filtering on squared values instead of original
• Using incorrect lambda expressions