SparkSession and SparkContext in Apache Spark - Mini Project: Build & Apply

SparkSession and SparkContext Basics

📖 Scenario: You are working with Apache Spark to analyze data. To start, you need to create the main entry points to Spark: SparkSession and SparkContext. These allow you to work with data in a distributed way.

🎯 Goal: Create a SparkSession named spark and get its SparkContext as sc. Then print the Spark application name.

📋 What You'll Learn

Create a SparkSession named spark with app name 'MySparkApp'

Get the SparkContext from spark and assign it to sc

Print the Spark application name using sc.appName

💡 Why This Matters

🌍 Real World

SparkSession and SparkContext are the starting points for any Apache Spark application. They allow you to connect to the Spark cluster and work with data in a distributed way.

💼 Career

Understanding how to create and use SparkSession and SparkContext is essential for data engineers and data scientists working with big data and distributed computing.

1

Create a SparkSession named spark

Write code to create a SparkSession named spark with the application name set to 'MySparkApp'. Use SparkSession.builder.appName('MySparkApp').getOrCreate().

Apache Spark

# Your code here to create SparkSession named spark

Need a hint?

Use SparkSession.builder.appName('MySparkApp').getOrCreate() to create the SparkSession.

2

Get the SparkContext from spark

Write code to get the SparkContext from the SparkSession spark and assign it to a variable named sc. Use sc = spark.sparkContext.

Apache Spark

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName('MySparkApp').getOrCreate()
# Your code here to get SparkContext as sc

Need a hint?

Access the SparkContext using spark.sparkContext.

3

Print the Spark application name

Write code to print the Spark application name using the variable sc and its appName property. Use print(sc.appName).

Apache Spark

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName('MySparkApp').getOrCreate()
sc = spark.sparkContext
# Your code here to print the Spark application name

Need a hint?

Use print(sc.appName) to display the application name.

4

Run the program and see the output

Run the complete program to see the Spark application name printed as output.

Apache Spark

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName('MySparkApp').getOrCreate()
sc = spark.sparkContext
print(sc.appName)  # Run this line to see output

Need a hint?

The output should be the string MySparkApp.