Why cloud simplifies Spark operations in Apache Spark - See It in Action

Why Cloud Simplifies Spark Operations

📖 Scenario: You work as a data analyst in a company that processes large amounts of data using Apache Spark. You want to understand how using cloud services can make your Spark tasks easier and faster.

🎯 Goal: Build a simple Spark program that reads data, applies a filter, and counts results. Then add a configuration variable to simulate cloud resource settings. Finally, print the count to see the output.

📋 What You'll Learn

Create a Spark DataFrame with sample data

Add a configuration variable to simulate cloud resource allocation

Filter the DataFrame based on a condition

Print the count of filtered rows

💡 Why This Matters

🌍 Real World

Companies use cloud platforms to run Spark jobs without managing hardware. This makes data processing faster and easier.

💼 Career

Data engineers and analysts often use cloud Spark services to handle big data efficiently and scale resources as needed.

1

Create a Spark DataFrame with sample data

Create a Spark DataFrame called df with these exact rows: (1, 'apple'), (2, 'banana'), (3, 'cherry'). Use columns named id and fruit.

Apache Spark

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName('CloudSpark').getOrCreate()
# Create DataFrame df with rows (1, 'apple'), (2, 'banana'), (3, 'cherry') and columns 'id', 'fruit'
# Your code here

Need a hint?

Use spark.createDataFrame() with a list of tuples and column names.

2

Add a cloud resource configuration variable

Create a variable called cloud_memory_gb and set it to 8 to simulate 8 GB of cloud memory allocation.

Apache Spark

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName('CloudSpark').getOrCreate()
data = [(1, 'apple'), (2, 'banana'), (3, 'cherry')]
df = spark.createDataFrame(data, ['id', 'fruit'])
# Create variable cloud_memory_gb and set it to 8
# Your code here

Need a hint?

Just create a variable named cloud_memory_gb and assign the number 8.

3

Filter the DataFrame for fruits starting with 'b'

Use df.filter() with a condition to keep only rows where the fruit column starts with the letter 'b'. Save the result in a variable called filtered_df.

Apache Spark

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName('CloudSpark').getOrCreate()
data = [(1, 'apple'), (2, 'banana'), (3, 'cherry')]
df = spark.createDataFrame(data, ['id', 'fruit'])
cloud_memory_gb = 8
# Filter df for fruits starting with 'b' and save as filtered_df
# Your code here

Need a hint?

Use df.filter(df.fruit.startswith('b')) to filter rows.

4

Print the count of filtered rows

Print the number of rows in filtered_df using filtered_df.count().

Apache Spark

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName('CloudSpark').getOrCreate()
data = [(1, 'apple'), (2, 'banana'), (3, 'cherry')]
df = spark.createDataFrame(data, ['id', 'fruit'])
cloud_memory_gb = 8
filtered_df = df.filter(df.fruit.startswith('b'))
# Print the count of filtered_df rows
# Your code here

Need a hint?

Use print(filtered_df.count()) to show the number of filtered rows.