SparkSession and SparkContext in Apache Spark - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

Output of SparkSession and SparkContext version check

What will be the output of the following Spark code snippet?

Apache Spark

from pyspark.sql import SparkSession
spark = SparkSession.builder.appName('TestApp').getOrCreate()
sc = spark.sparkContext
print(sc.version)

A "2.4.0"

B SyntaxError

C None

D "3.4.0"

💡 Hint

Check the version of SparkContext obtained from SparkSession.

The SparkContext version reflects the Spark version installed. The code prints the version string, e.g., "3.4.0".

❓ data_output

intermediate

Number of partitions in RDD created from SparkContext

Given the following code, what is the number of partitions in the RDD?

Apache Spark

from pyspark.sql import SparkSession
spark = SparkSession.builder.appName('PartitionTest').getOrCreate()
sc = spark.sparkContext
rdd = sc.parallelize(range(10), 4)
print(rdd.getNumPartitions())

A 1

B 10

C 4

D 0

💡 Hint

Look at the second argument of parallelize which sets partitions.

The parallelize method's second argument sets the number of partitions. Here it is 4.

🔧 Debug

advanced

Identify the error in SparkSession creation

What error will the following code produce?

Apache Spark

from pyspark.sql import SparkSession
spark = SparkSession.builder.appName('App').getOrCreate
print(spark.sparkContext.appName)

A AttributeError: 'function' object has no attribute 'sparkContext'

B TypeError: getOrCreate() missing 1 required positional argument

C NameError: name 'SparkSession' is not defined

D No error, prints 'App'

💡 Hint

Check if getOrCreate is called correctly.

The code misses parentheses after getOrCreate, so spark is a function, not a SparkSession object. Accessing spark.sparkContext causes AttributeError.

🧠 Conceptual

advanced

Difference between SparkSession and SparkContext

Which statement correctly describes the difference between SparkSession and SparkContext?

A <code>SparkSession</code> is the entry point for DataFrame API, while <code>SparkContext</code> is the entry point for RDD API.

B <code>SparkContext</code> manages SQL queries, <code>SparkSession</code> manages cluster resources.

C <code>SparkSession</code> is deprecated and replaced by <code>SparkContext</code>.

D <code>SparkContext</code> is used only for streaming, <code>SparkSession</code> only for batch jobs.

💡 Hint

Think about the APIs each object supports.

SparkSession provides unified entry for DataFrame and SQL APIs. SparkContext is the older entry point for RDDs.

🚀 Application

expert

Result of caching DataFrame and counting partitions

Consider the following code. What will be the output?

Apache Spark

from pyspark.sql import SparkSession
spark = SparkSession.builder.appName('CacheTest').getOrCreate()
df = spark.createDataFrame([(1, 'a'), (2, 'b'), (3, 'c'), (4, 'd')], ['id', 'value'])
df_cached = df.cache()
count = df_cached.count()
partitions = df_cached.rdd.getNumPartitions()
print(count, partitions)

A 0 1

B 4 1

C SyntaxError

D 4 4

💡 Hint

Check default number of partitions for DataFrame created from local data.

DataFrame created from local list has 1 partition by default. Counting rows returns 4. Caching does not change partitions.