Creating RDDs from collections and files in Apache Spark - Try It Yourself

Creating RDDs from collections and files

📖 Scenario: You are working with Apache Spark to process data. You want to learn how to create RDDs (Resilient Distributed Datasets) from simple collections and from files stored on your computer. This is a basic skill to start working with Spark data processing.

🎯 Goal: Build a Spark program that creates RDDs from a Python list and from a text file, so you can later process data in Spark.

📋 What You'll Learn

Create an RDD from a Python list using sc.parallelize()

Create an RDD from a text file using sc.textFile()

Use SparkContext variable named sc

Print the contents of both RDDs

💡 Why This Matters

🌍 Real World

Creating RDDs from collections and files is the first step in processing big data with Apache Spark. It lets you load data from memory or disk to start analysis.

💼 Career

Data engineers and data scientists use these skills to prepare data for distributed processing and analysis in Spark environments.

1

Create an RDD from a Python list

Create a Python list called data_list with these exact values: "apple", "banana", "cherry". Then create an RDD called rdd_from_list by using sc.parallelize(data_list).

Apache Spark

# Create data_list with 'apple', 'banana', 'cherry'
# Create rdd_from_list using sc.parallelize(data_list)

Hint

Use sc.parallelize() to convert a Python list into an RDD.

2

Create an RDD from a text file

Create a variable called file_path and set it to the string "data/sample.txt". Then create an RDD called rdd_from_file by using sc.textFile(file_path).

Apache Spark

data_list = ["apple", "banana", "cherry"]
rdd_from_list = sc.parallelize(data_list)
# Set file_path to 'data/sample.txt'
# Create rdd_from_file using sc.textFile(file_path)

Hint

Use sc.textFile() to read a text file into an RDD.

3

Collect data from both RDDs

Create two variables: list_data and file_data. Assign rdd_from_list.collect() to list_data and rdd_from_file.collect() to file_data.

Apache Spark

data_list = ["apple", "banana", "cherry"]
rdd_from_list = sc.parallelize(data_list)
file_path = "data/sample.txt"
rdd_from_file = sc.textFile(file_path)
# Collect data from rdd_from_list into list_data
# Collect data from rdd_from_file into file_data

Hint

Use the collect() method to get all elements from an RDD as a list.

4

Print the collected data

Print the variables list_data and file_data each on its own line using two separate print() statements.

Apache Spark

data_list = ["apple", "banana", "cherry"]
rdd_from_list = sc.parallelize(data_list)
file_path = "data/sample.txt"
rdd_from_file = sc.textFile(file_path)
list_data = rdd_from_list.collect()
file_data = rdd_from_file.collect()
# Print list_data
# Print file_data

Hint

Use print() to display the collected lists.

Practice

1. What does the parallelize() method do in Apache Spark?

easy

A. Creates an RDD from a local collection like a list

B. Reads data from a text file into an RDD

C. Saves an RDD to a file

D. Filters elements in an RDD

Solution

1. Step 1: Understand the purpose of parallelize()

   This method takes a local collection such as a list and creates an RDD from it, enabling parallel processing.

2. Step 2: Differentiate from other methods

   textFile() reads from files, saveAsTextFile() saves data, and filter() filters data. These are different from parallelize().

3. Final Answer:

   Creates an RDD from a local collection like a list -> Option A

4. Quick Check:

   parallelize() = create RDD from list [OK]

Hint: parallelize() = local list to RDD [OK]

Common Mistakes:

• Confusing parallelize() with textFile()
• Thinking parallelize() reads files
• Mixing up saving and creating RDDs

2. Which of the following is the correct syntax to create an RDD from a list [1, 2, 3] using SparkContext sc?

easy

A. sc.load([1, 2, 3])

B. sc.parallelize([1, 2, 3])

C. sc.createRDD([1, 2, 3])

D. sc.textFile([1, 2, 3])

Solution

1. Step 1: Recall the method to create RDD from a list

   The correct method is parallelize() called on SparkContext with the list as argument.

2. Step 2: Check syntax correctness

   sc.parallelize([1, 2, 3]) is valid syntax. Other options are invalid or do not exist.

3. Final Answer:

   sc.parallelize([1, 2, 3]) -> Option B

4. Quick Check:

   Use parallelize() with list [OK]

Hint: Use sc.parallelize(list) to create RDD [OK]

Common Mistakes:

• Using textFile() with a list argument
• Trying to use non-existent methods like createRDD()
• Passing list to load() which is for files

3. Given the following code snippet:

rdd = sc.parallelize(['apple', 'banana', 'cherry'])
result = rdd.collect()

What will be the value of result?

medium

A. A list of line numbers

B. An RDD object

C. ['apple', 'banana', 'cherry']

D. A text file path

Solution

1. Step 1: Understand what parallelize() does

   It creates an RDD from the list ['apple', 'banana', 'cherry'].

2. Step 2: Understand what collect() does

   collect() returns all elements of the RDD as a list to the driver program.

3. Final Answer:

   ['apple', 'banana', 'cherry'] -> Option C

4. Quick Check:

   collect() returns list of RDD elements [OK]

Hint: collect() returns all RDD elements as list [OK]

Common Mistakes:

• Thinking collect() returns an RDD
• Confusing collect() with saving to file
• Expecting line numbers instead of data

4. Identify the error in the following code snippet:

rdd = sc.textFile([1, 2, 3])
print(rdd.collect())

medium

A. textFile() expects a string path, not a list

B. collect() cannot be called on textFile RDD

C. sc.textFile() requires two arguments

D. print() cannot display RDD contents

Solution

1. Step 1: Check the argument type for textFile()

   textFile() expects a string path to a file, not a list of strings.

2. Step 2: Understand the error cause

   Passing a list causes a type error because Spark cannot interpret it as a file path.

3. Final Answer:

   textFile() expects a string path, not a list -> Option A

4. Quick Check:

   textFile() needs string path [OK]

Hint: Pass file path as string to textFile() [OK]

Common Mistakes:

• Passing list instead of string to textFile()
• Thinking collect() is invalid on textFile RDD
• Assuming textFile needs multiple arguments

5. You want to create an RDD from a text file named data.txt and then filter lines containing the word "error". Which code snippet correctly creates the RDD and applies the filter?

hard

A. rdd = sc.load('data.txt')
filtered = rdd.filter(lambda line: 'error' in line)

B. rdd = sc.parallelize('data.txt')
filtered = rdd.filter(lambda line: 'error' in line)

C. rdd = sc.textFile([1, 2, 3])
filtered = rdd.filter(lambda line: 'error' in line)

D. rdd = sc.textFile('data.txt')
filtered = rdd.filter(lambda line: 'error' in line)

Solution

1. Step 1: Choose correct method to create RDD from file

   sc.textFile('data.txt') reads the file line by line into an RDD. Passing a string path is required.

2. Step 2: Apply filter to select lines containing 'error'

   The filter() method with a lambda checks each line for the substring 'error'. This is the correct way to filter.

3. Final Answer:

   rdd = sc.textFile('data.txt') filtered = rdd.filter(lambda line: 'error' in line) -> Option D

4. Quick Check:

   Use textFile() with string path + filter() [OK]

Hint: Use textFile('file') then filter() with lambda [OK]

Common Mistakes:

• Using parallelize() to read files
• Passing list instead of string to textFile()
• Using non-existent sc.load() method