Creating RDDs from collections and files in Apache Spark - Practice Exercises

Challenge - 5 Problems

❓ Predict Output

intermediate

Output of RDD count after filtering

What is the output of the following Spark code snippet?

Apache Spark

val sc = spark.sparkContext
val data = List(1, 2, 3, 4, 5, 6)
val rdd = sc.parallelize(data)
val filteredRDD = rdd.filter(x => x % 2 == 0)
val count = filteredRDD.count()
println(count)

A 4

B 6

C 2

D 3

💡 Hint

Count how many numbers in the list are even.

The list has 6 numbers. Even numbers are 2, 4, and 6, so count is 3.

❓ data_output

intermediate

Result of reading a text file into an RDD

Given a text file with 4 lines, what will be the number of elements in the RDD after reading it with sc.textFile?

Apache Spark

val rdd = sc.textFile("/path/to/file.txt")
rdd.count()

A 1

B 4

C 0

D Depends on file size

💡 Hint

Each line in the file becomes one element in the RDD.

sc.textFile reads the file line by line, so the RDD has one element per line.

🔧 Debug

advanced

Identify the error in RDD creation from a collection

What error will this Spark code produce?

Apache Spark

val data = 1 to 5
val rdd = sc.parallelize(data.toString())
rdd.collect().foreach(println)

A The RDD contains characters of the string 'Range(1,2,3,4,5)'

B Compilation error: toString cannot be used here

C Runtime error: Unsupported operation on Range

D The RDD contains numbers 1 to 5 as expected

💡 Hint

Check what data.toString returns and what parallelize expects.

data.toString converts the range to a string, so parallelize treats it as a sequence of characters.

🧠 Conceptual

advanced

Difference between sc.parallelize and sc.textFile

Which statement correctly describes the difference between sc.parallelize and sc.textFile?

A Both create RDDs from collections but sc.textFile supports filtering

B sc.parallelize reads data from a file, sc.textFile creates an RDD from a collection

C sc.parallelize creates an RDD from an existing collection in memory, sc.textFile reads data from a file into an RDD

D Both create RDDs from files but sc.parallelize is faster

💡 Hint

Think about the source of data for each method.

sc.parallelize is used to create RDDs from in-memory collections, while sc.textFile reads data from files.

❓ visualization

expert

Visualizing partitions of an RDD created from a file

You create an RDD from a text file with 100 lines using sc.textFile with 4 partitions. Which visualization best represents the distribution of lines across partitions?

A A bar chart with 4 bars each showing approximately 25 lines

B A pie chart with 100 equal slices representing each line

C A line chart showing increasing number of lines per partition from 1 to 4

D A scatter plot with 100 points randomly scattered

💡 Hint

Partitions try to split data evenly.

sc.textFile splits the file lines evenly across the specified partitions, so each partition has about 25 lines.