Creating RDDs from collections and files in Apache Spark - Practice Exercises

Challenge - 5 Problems

❓ Predict Output

intermediate

Output of RDD count after filtering

What is the output of the following Spark code snippet?

Apache Spark

val sc = spark.sparkContext
val data = List(1, 2, 3, 4, 5, 6)
val rdd = sc.parallelize(data)
val filteredRDD = rdd.filter(x => x % 2 == 0)
val count = filteredRDD.count()
println(count)

A 4

B 6

C 2

D 3

💡 Hint

Count how many numbers in the list are even.

The list has 6 numbers. Even numbers are 2, 4, and 6, so count is 3.

❓ data_output

intermediate

Result of reading a text file into an RDD

Given a text file with 4 lines, what will be the number of elements in the RDD after reading it with sc.textFile?

Apache Spark

val rdd = sc.textFile("/path/to/file.txt")
rdd.count()

A 1

B 4

C 0

D Depends on file size

💡 Hint

Each line in the file becomes one element in the RDD.

sc.textFile reads the file line by line, so the RDD has one element per line.

🔧 Debug

advanced

Identify the error in RDD creation from a collection

What error will this Spark code produce?

Apache Spark

val data = 1 to 5
val rdd = sc.parallelize(data.toString())
rdd.collect().foreach(println)

A The RDD contains characters of the string 'Range(1,2,3,4,5)'

B Compilation error: toString cannot be used here

C Runtime error: Unsupported operation on Range

D The RDD contains numbers 1 to 5 as expected

💡 Hint

Check what data.toString returns and what parallelize expects.

data.toString converts the range to a string, so parallelize treats it as a sequence of characters.

🧠 Conceptual

advanced

Difference between sc.parallelize and sc.textFile

Which statement correctly describes the difference between sc.parallelize and sc.textFile?

A Both create RDDs from collections but sc.textFile supports filtering

B sc.parallelize reads data from a file, sc.textFile creates an RDD from a collection

C sc.parallelize creates an RDD from an existing collection in memory, sc.textFile reads data from a file into an RDD

D Both create RDDs from files but sc.parallelize is faster

💡 Hint

Think about the source of data for each method.

sc.parallelize is used to create RDDs from in-memory collections, while sc.textFile reads data from files.

❓ visualization

expert

Visualizing partitions of an RDD created from a file

You create an RDD from a text file with 100 lines using sc.textFile with 4 partitions. Which visualization best represents the distribution of lines across partitions?

A A bar chart with 4 bars each showing approximately 25 lines

B A pie chart with 100 equal slices representing each line

C A line chart showing increasing number of lines per partition from 1 to 4

D A scatter plot with 100 points randomly scattered

💡 Hint

Partitions try to split data evenly.

sc.textFile splits the file lines evenly across the specified partitions, so each partition has about 25 lines.

Practice

1. What does the parallelize() method do in Apache Spark?

easy

A. Creates an RDD from a local collection like a list

B. Reads data from a text file into an RDD

C. Saves an RDD to a file

D. Filters elements in an RDD

Solution

1. Step 1: Understand the purpose of parallelize()

   This method takes a local collection such as a list and creates an RDD from it, enabling parallel processing.

2. Step 2: Differentiate from other methods

   textFile() reads from files, saveAsTextFile() saves data, and filter() filters data. These are different from parallelize().

3. Final Answer:

   Creates an RDD from a local collection like a list -> Option A

4. Quick Check:

   parallelize() = create RDD from list [OK]

Hint: parallelize() = local list to RDD [OK]

Common Mistakes:

• Confusing parallelize() with textFile()
• Thinking parallelize() reads files
• Mixing up saving and creating RDDs

2. Which of the following is the correct syntax to create an RDD from a list [1, 2, 3] using SparkContext sc?

easy

A. sc.load([1, 2, 3])

B. sc.parallelize([1, 2, 3])

C. sc.createRDD([1, 2, 3])

D. sc.textFile([1, 2, 3])

Solution

1. Step 1: Recall the method to create RDD from a list

   The correct method is parallelize() called on SparkContext with the list as argument.

2. Step 2: Check syntax correctness

   sc.parallelize([1, 2, 3]) is valid syntax. Other options are invalid or do not exist.

3. Final Answer:

   sc.parallelize([1, 2, 3]) -> Option B

4. Quick Check:

   Use parallelize() with list [OK]

Hint: Use sc.parallelize(list) to create RDD [OK]

Common Mistakes:

• Using textFile() with a list argument
• Trying to use non-existent methods like createRDD()
• Passing list to load() which is for files

3. Given the following code snippet:

rdd = sc.parallelize(['apple', 'banana', 'cherry'])
result = rdd.collect()

What will be the value of result?

medium

A. A list of line numbers

B. An RDD object

C. ['apple', 'banana', 'cherry']

D. A text file path

Solution

1. Step 1: Understand what parallelize() does

   It creates an RDD from the list ['apple', 'banana', 'cherry'].

2. Step 2: Understand what collect() does

   collect() returns all elements of the RDD as a list to the driver program.

3. Final Answer:

   ['apple', 'banana', 'cherry'] -> Option C

4. Quick Check:

   collect() returns list of RDD elements [OK]

Hint: collect() returns all RDD elements as list [OK]

Common Mistakes:

• Thinking collect() returns an RDD
• Confusing collect() with saving to file
• Expecting line numbers instead of data

4. Identify the error in the following code snippet:

rdd = sc.textFile([1, 2, 3])
print(rdd.collect())

medium

A. textFile() expects a string path, not a list

B. collect() cannot be called on textFile RDD

C. sc.textFile() requires two arguments

D. print() cannot display RDD contents

Solution

1. Step 1: Check the argument type for textFile()

   textFile() expects a string path to a file, not a list of strings.

2. Step 2: Understand the error cause

   Passing a list causes a type error because Spark cannot interpret it as a file path.

3. Final Answer:

   textFile() expects a string path, not a list -> Option A

4. Quick Check:

   textFile() needs string path [OK]

Hint: Pass file path as string to textFile() [OK]

Common Mistakes:

• Passing list instead of string to textFile()
• Thinking collect() is invalid on textFile RDD
• Assuming textFile needs multiple arguments

5. You want to create an RDD from a text file named data.txt and then filter lines containing the word "error". Which code snippet correctly creates the RDD and applies the filter?

hard

A. rdd = sc.load('data.txt')
filtered = rdd.filter(lambda line: 'error' in line)

B. rdd = sc.parallelize('data.txt')
filtered = rdd.filter(lambda line: 'error' in line)

C. rdd = sc.textFile([1, 2, 3])
filtered = rdd.filter(lambda line: 'error' in line)

D. rdd = sc.textFile('data.txt')
filtered = rdd.filter(lambda line: 'error' in line)

Solution

1. Step 1: Choose correct method to create RDD from file

   sc.textFile('data.txt') reads the file line by line into an RDD. Passing a string path is required.

2. Step 2: Apply filter to select lines containing 'error'

   The filter() method with a lambda checks each line for the substring 'error'. This is the correct way to filter.

3. Final Answer:

   rdd = sc.textFile('data.txt') filtered = rdd.filter(lambda line: 'error' in line) -> Option D

4. Quick Check:

   Use textFile() with string path + filter() [OK]

Hint: Use textFile('file') then filter() with lambda [OK]

Common Mistakes:

• Using parallelize() to read files
• Passing list instead of string to textFile()
• Using non-existent sc.load() method