Why DataFrames are preferred over RDDs in Apache Spark - Challenge Your Understanding

Challenge - 5 Problems

🧠 Conceptual

intermediate

Advantages of DataFrames over RDDs

Which of the following is a key reason why DataFrames are preferred over RDDs in Apache Spark?

A DataFrames do not support SQL queries, unlike RDDs which do.

B DataFrames provide schema and optimize queries using Catalyst optimizer, improving performance.

C RDDs support schema and automatic query optimization, making them faster than DataFrames.

D RDDs automatically cache data by default, while DataFrames do not.

💡 Hint

Think about how Spark optimizes query execution for structured data.

DataFrames have a schema and use Spark's Catalyst optimizer to plan and optimize queries, which makes them faster and more efficient than RDDs that lack schema and optimization.

❓ Predict Output

intermediate

Output of DataFrame vs RDD count

What will be the output of the following Spark code snippet?

Apache Spark

rdd = spark.sparkContext.parallelize([1, 2, 3, 4])
df = rdd.toDF(['numbers'])
count_rdd = rdd.count()
count_df = df.count()
print(count_rdd, count_df)

A 4 4

B 4 0

C Error because RDD and DataFrame counts differ

D 0 4

💡 Hint

Both RDD and DataFrame represent the same data here.

Both RDD and DataFrame contain the same 4 elements, so their counts are both 4.

❓ data_output

advanced

Schema inference in DataFrames

Given the following code, what is the schema of the DataFrame?

Apache Spark

data = [(1, 'Alice', 29), (2, 'Bob', 31)]
df = spark.createDataFrame(data, schema=['id', 'name', 'age'])
df.printSchema()

A

root
 |-- id: long (nullable = true)
 |-- name: string (nullable = true)
 |-- age: long (nullable = true)

B

root
 |-- id: string (nullable = true)
 |-- name: string (nullable = true)
 |-- age: string (nullable = true)

C

root
 |-- id: integer (nullable = false)
 |-- name: string (nullable = false)
 |-- age: integer (nullable = false)

D

root
 |-- id: double (nullable = true)
 |-- name: string (nullable = true)
 |-- age: double (nullable = true)

💡 Hint

Check the default data types inferred by Spark for integers and strings.

Spark infers integer numbers as long type by default and strings as string type with nullable true.

🔧 Debug

advanced

Error when applying SQL functions on RDD

What error will occur when running this code?

Apache Spark

rdd = spark.sparkContext.parallelize([(1, 'a'), (2, 'b')])
rdd.select('1').show()

A No error, outputs the first column

B TypeError: select() missing 1 required positional argument

C SyntaxError: invalid syntax

D AttributeError: 'RDD' object has no attribute 'select'

💡 Hint

Consider which Spark objects support the select() method.

RDDs do not have the select() method; only DataFrames do. Trying to call select on an RDD causes AttributeError.

🚀 Application

expert

Performance difference in aggregation

You have a large dataset and want to calculate the average age grouped by city. Which approach will be faster and why?

A Using RDD filter and collect because filtering reduces data size before aggregation.

B Using RDD map and reduceByKey because RDDs are lower level and faster for all operations.

C Using DataFrame groupBy and avg functions because Spark optimizes these operations with Catalyst and Tungsten.

D Using DataFrame collect and then Python aggregation because local processing is faster.

💡 Hint

Think about how Spark optimizes structured data operations.

DataFrames use Spark's Catalyst optimizer and Tungsten execution engine to optimize aggregation operations, making them faster than manual RDD operations.