SparkSession and SparkContext in Apache Spark - Cheat Sheet & Quick Revision

Recall & Review

beginner

What is SparkContext in Apache Spark?

SparkContext is the entry point to Spark functionality. It connects your program to the Spark cluster and allows you to create RDDs (Resilient Distributed Datasets). Think of it as the main controller for Spark jobs.

beginner

What is SparkSession and why is it important?

SparkSession is the new entry point introduced in Spark 2.0. It combines SQLContext, HiveContext, and SparkContext into one object. It lets you work with DataFrames, SQL, and streaming easily in one place.

intermediate

How do SparkSession and SparkContext relate to each other?

SparkSession internally manages a SparkContext. When you create a SparkSession, it creates or uses an existing SparkContext. You usually use SparkSession now because it handles more features and is simpler.

beginner

How to create a SparkSession in PySpark?

You create a SparkSession using the builder pattern like this:<br><pre>from pyspark.sql import SparkSession
spark = SparkSession.builder.appName('MyApp').getOrCreate()</pre><br>This starts Spark and lets you use DataFrames and SQL.

intermediate

Why should you prefer SparkSession over SparkContext in modern Spark applications?

SparkSession provides a unified interface for all Spark features like SQL, streaming, and machine learning. It simplifies code and manages SparkContext internally, so you don't have to handle multiple contexts separately.

What is the main role of SparkContext in Apache Spark?

A Connect your program to the Spark cluster and create RDDs

B Manage SQL queries only

C Visualize data in Spark

D Store data permanently

SparkContext connects your program to the Spark cluster and lets you create RDDs for distributed computing.

Which object should you use to work with DataFrames and SQL in Spark 2.0 and later?

A SparkContext

B SQLContext

C SparkSession

D HiveContext

SparkSession is the unified entry point for DataFrames, SQL, and other Spark features since Spark 2.0.

How do you create a SparkSession in PySpark?

A spark = SparkContext()

B spark = SparkSession.builder.appName('App').getOrCreate()

C spark = SQLContext()

D spark = HiveContext()

You create a SparkSession using the builder pattern with appName and getOrCreate() methods.

What does SparkSession internally manage?

A Only SQLContext

B No other context

C Only HiveContext

D SparkContext

SparkSession internally manages SparkContext to handle cluster connection and job execution.

Why is SparkSession preferred over SparkContext in modern Spark applications?

A It provides a unified interface for multiple Spark features

B It requires less memory

C It is slower but more reliable

D It only supports RDDs

SparkSession unifies access to SQL, streaming, machine learning, and RDDs, simplifying Spark programming.

Explain the difference between SparkContext and SparkSession and when to use each.

Think about how Spark evolved from SparkContext to SparkSession.

Describe how to create a SparkSession in PySpark and why it is important.

Focus on the code and the benefits of SparkSession.