Creating RDDs from collections and files in Apache Spark - Quick Revision & Summary

Recall & Review

beginner

What is an RDD in Apache Spark?

RDD stands for Resilient Distributed Dataset. It is a basic data structure in Spark that represents a distributed collection of objects that can be processed in parallel.

beginner

How do you create an RDD from a collection in Spark?

You use the SparkContext's parallelize() method to create an RDD from an existing collection like a list or array.

beginner

How do you create an RDD from a file in Spark?

You use the SparkContext's textFile() method and provide the file path. This reads the file and creates an RDD where each element is a line from the file.

intermediate

What is the difference between parallelize() and textFile() when creating RDDs?

parallelize() creates an RDD from an existing collection in memory, while textFile() creates an RDD by reading data from a file stored on disk or distributed storage.

beginner

Why is creating RDDs from files useful in real life?

Because data often comes from files like logs, CSVs, or text files, creating RDDs from files lets you process large datasets stored on disk or cloud storage easily and in parallel.

Which Spark method creates an RDD from a list in memory?

A textFile()

B parallelize()

C read()

D collect()

parallelize() is used to create an RDD from an existing collection like a list.

What does textFile() return when called with a file path?

A A DataFrame

B A list of file names

C An RDD where each element is a line from the file

D A single string with the whole file content

textFile() reads the file and creates an RDD with each line as an element.

Which of these is NOT a way to create an RDD?

A fromCSV()

B textFile()

C parallelize()

D wholeTextFiles()

fromCSV() is not a Spark method to create RDDs; CSV files are usually read into DataFrames.

If you want to process data stored in a text file on disk, which method should you use?

A parallelize()

B map()

C collect()

D textFile()

textFile() reads data from a file and creates an RDD.

What type of data does parallelize() accept to create an RDD?

A A collection like a list or array

B A file path

C A database connection

D A SparkSession

parallelize() takes a collection in memory to create an RDD.

Explain how to create an RDD from a collection and from a file in Apache Spark.

Think about where your data lives: memory or disk.

Why would you choose to create an RDD from a file instead of a collection?

Consider data size and source.