SparkSession and SparkContext in Apache Spark - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

Output of SparkSession and SparkContext version check

What will be the output of the following Spark code snippet?

Apache Spark

from pyspark.sql import SparkSession
spark = SparkSession.builder.appName('TestApp').getOrCreate()
sc = spark.sparkContext
print(sc.version)

A "2.4.0"

B SyntaxError

C None

D "3.4.0"

💡 Hint

Check the version of SparkContext obtained from SparkSession.

The SparkContext version reflects the Spark version installed. The code prints the version string, e.g., "3.4.0".

❓ data_output

intermediate

Number of partitions in RDD created from SparkContext

Given the following code, what is the number of partitions in the RDD?

Apache Spark

from pyspark.sql import SparkSession
spark = SparkSession.builder.appName('PartitionTest').getOrCreate()
sc = spark.sparkContext
rdd = sc.parallelize(range(10), 4)
print(rdd.getNumPartitions())

A 1

B 10

C 4

D 0

💡 Hint

Look at the second argument of parallelize which sets partitions.

The parallelize method's second argument sets the number of partitions. Here it is 4.

🔧 Debug

advanced

Identify the error in SparkSession creation

What error will the following code produce?

Apache Spark

from pyspark.sql import SparkSession
spark = SparkSession.builder.appName('App').getOrCreate
print(spark.sparkContext.appName)

A AttributeError: 'function' object has no attribute 'sparkContext'

B TypeError: getOrCreate() missing 1 required positional argument

C NameError: name 'SparkSession' is not defined

D No error, prints 'App'

💡 Hint

Check if getOrCreate is called correctly.

The code misses parentheses after getOrCreate, so spark is a function, not a SparkSession object. Accessing spark.sparkContext causes AttributeError.

🧠 Conceptual

advanced

Difference between SparkSession and SparkContext

Which statement correctly describes the difference between SparkSession and SparkContext?

A <code>SparkSession</code> is the entry point for DataFrame API, while <code>SparkContext</code> is the entry point for RDD API.

B <code>SparkContext</code> manages SQL queries, <code>SparkSession</code> manages cluster resources.

C <code>SparkSession</code> is deprecated and replaced by <code>SparkContext</code>.

D <code>SparkContext</code> is used only for streaming, <code>SparkSession</code> only for batch jobs.

💡 Hint

Think about the APIs each object supports.

SparkSession provides unified entry for DataFrame and SQL APIs. SparkContext is the older entry point for RDDs.

🚀 Application

expert

Result of caching DataFrame and counting partitions

Consider the following code. What will be the output?

Apache Spark

from pyspark.sql import SparkSession
spark = SparkSession.builder.appName('CacheTest').getOrCreate()
df = spark.createDataFrame([(1, 'a'), (2, 'b'), (3, 'c'), (4, 'd')], ['id', 'value'])
df_cached = df.cache()
count = df_cached.count()
partitions = df_cached.rdd.getNumPartitions()
print(count, partitions)

A 0 1

B 4 1

C SyntaxError

D 4 4

💡 Hint

Check default number of partitions for DataFrame created from local data.

DataFrame created from local list has 1 partition by default. Counting rows returns 4. Caching does not change partitions.

Practice

1. What is the primary role of SparkSession in Apache Spark?

easy

A. It stores data permanently in Spark.

B. It manages the cluster resources directly.

C. It is the main entry point to work with Spark SQL and DataFrames.

D. It replaces the need for any SparkContext.

Solution

1. Step 1: Understand SparkSession's purpose

   SparkSession is designed as the main interface for working with structured data using Spark SQL and DataFrames.

2. Step 2: Differentiate from other components

   SparkContext manages cluster connection but SparkSession provides higher-level APIs for SQL and DataFrames.

3. Final Answer:

   It is the main entry point to work with Spark SQL and DataFrames. -> Option C

4. Quick Check:

   SparkSession = main SQL/DataFrame entry [OK]

Hint: SparkSession is your main tool for SQL and DataFrames [OK]

Common Mistakes:

• Confusing SparkSession with SparkContext's cluster management
• Thinking SparkSession stores data permanently
• Believing SparkSession removes need for SparkContext

2. Which of the following is the correct way to create a SparkSession in PySpark?

easy

A. spark = SparkContext.builder.appName('MyApp').getOrCreate()

B. spark = SparkSession.builder.appName('MyApp').getOrCreate()

C. spark = SparkSession.create('MyApp')

D. spark = SparkContext.create('MyApp')

Solution

1. Step 1: Recall SparkSession creation syntax

   The correct way is to use SparkSession.builder, set app name, then call getOrCreate() to create or get existing session.

2. Step 2: Identify incorrect options

   SparkContext does not have builder or getOrCreate methods; SparkSession.create is not valid syntax.

3. Final Answer:

   spark = SparkSession.builder.appName('MyApp').getOrCreate() -> Option B

4. Quick Check:

   Use SparkSession.builder.appName().getOrCreate() [OK]

Hint: Use SparkSession.builder.appName().getOrCreate() to start [OK]

Common Mistakes:

• Using SparkContext instead of SparkSession to create session
• Calling non-existent create() method
• Missing getOrCreate() at the end

3. Given the following PySpark code, what will be the output of print(sc.appName)?

from pyspark.sql import SparkSession
spark = SparkSession.builder.appName('TestApp').master('local').getOrCreate()
sc = spark.sparkContext
print(sc.appName)

medium

A. None

B. local

C. SparkContext

D. TestApp

Solution

1. Step 1: Understand appName assignment

   The appName 'TestApp' is set when building SparkSession and is passed to SparkContext.

2. Step 2: Access SparkContext appName property

   sc.appName returns the application name set, which is 'TestApp'.

3. Final Answer:

   TestApp -> Option D

4. Quick Check:

   sc.appName = 'TestApp' [OK]

Hint: sc.appName matches SparkSession appName [OK]

Common Mistakes:

• Confusing master URL with appName
• Expecting sc.appName to be 'SparkContext'
• Assuming sc.appName is None by default

4. Identify the error in the following PySpark code snippet:

from pyspark.sql import SparkSession
spark = SparkSession.builder.appName('App').getOrCreate()
sc = spark.sparkContext
sc = SparkContext('local', 'App')
print(sc.appName)

medium

A. Reassigning sc with SparkContext directly causes error.

B. Missing import for SparkContext.

C. SparkSession.builder.appName() is incorrect syntax.

D. print(sc.appName) is invalid method call.

Solution

1. Step 1: Analyze sc reassignment

   After creating SparkSession and getting sc, the code tries to create a new SparkContext directly, which is not allowed if one already exists.

2. Step 2: Understand SparkContext singleton rule

   Only one SparkContext can exist per JVM; creating another causes an error.

3. Final Answer:

   Reassigning sc with SparkContext directly causes error. -> Option A

4. Quick Check:

   Only one SparkContext allowed; reassigning causes error [OK]

Hint: Do not create SparkContext if SparkSession exists [OK]

Common Mistakes:

• Trying to create multiple SparkContext instances
• Ignoring SparkContext singleton restriction
• Assuming print(sc.appName) is invalid

5. You want to create a SparkSession with the app name 'DataApp' and run it locally using 4 cores. Which code snippet correctly achieves this and also retrieves the SparkContext?

hard

A. spark = SparkSession.builder.appName('DataApp').master('local[4]').getOrCreate()
sc = spark.sparkContext

B. sc = SparkContext('local[4]', 'DataApp')
spark = SparkSession(sc)

C. spark = SparkSession.create('DataApp', master='local[4]')
sc = spark.getContext()

D. spark = SparkSession.builder.master('4').appName('DataApp').getOrCreate()
sc = spark.context

Solution

1. Step 1: Use SparkSession.builder with appName and master

   The correct way is to use builder, set appName to 'DataApp' and master to 'local[4]' for 4 cores, then call getOrCreate().

2. Step 2: Retrieve SparkContext from SparkSession

   Access SparkContext via spark.sparkContext property after creating SparkSession.

3. Final Answer:

   spark = SparkSession.builder.appName('DataApp').master('local[4]').getOrCreate() sc = spark.sparkContext -> Option A

4. Quick Check:

   Use builder.appName().master('local[4]').getOrCreate() + spark.sparkContext [OK]

Hint: Set master='local[4]' and get spark.sparkContext [OK]

Common Mistakes:

• Using SparkContext constructor directly instead of SparkSession.builder
• Wrong master string like '4' instead of 'local[4]'
• Incorrect method names like create() or getContext()