Spot instances for cost savings in Apache Spark - Mini Project: Build & Apply

Spot instances for cost savings

📖 Scenario: You work for a cloud services company that wants to analyze the cost savings from using spot instances instead of on-demand instances. Spot instances are cheaper but can be interrupted. You have data about instance types, their on-demand prices, and spot prices.Your task is to calculate the percentage cost savings for each instance type when using spot instances.

🎯 Goal: Build a Spark DataFrame with instance pricing data, add a configuration for minimum savings threshold, filter instance types that meet or exceed this threshold, and display the results.

📋 What You'll Learn

Create a Spark DataFrame with instance types and their on-demand and spot prices.

Add a configuration variable for minimum savings percentage.

Calculate the percentage savings for each instance type using spot instances.

Filter the DataFrame to only include instance types with savings greater than or equal to the threshold.

Display the filtered DataFrame.

💡 Why This Matters

🌍 Real World

Cloud engineers and data analysts use this kind of analysis to optimize cloud costs by choosing cheaper spot instances when possible.

💼 Career

Understanding how to manipulate Spark DataFrames and perform cost analysis is valuable for roles in cloud cost management, data engineering, and data science.

1

Create the initial DataFrame with instance pricing

Create a Spark DataFrame called instances_df with the following data: instance_type as ['m5.large', 'c5.large', 'r5.large'], on_demand_price as [0.096, 0.085, 0.126], and spot_price as [0.029, 0.025, 0.038].

Apache Spark

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName('SpotInstances').getOrCreate()
# Create the DataFrame instances_df with the given data
# Your code here

Hint

Use spark.createDataFrame with a list of tuples and specify the column names.

2

Add a minimum savings threshold configuration

Create a variable called min_savings and set it to 60 to represent the minimum percentage savings required to consider an instance type.

Apache Spark

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName('SpotInstances').getOrCreate()

instances_data = [
    ('m5.large', 0.096, 0.029),
    ('c5.large', 0.085, 0.025),
    ('r5.large', 0.126, 0.038)
]

instances_df = spark.createDataFrame(instances_data, ['instance_type', 'on_demand_price', 'spot_price'])

# Create min_savings variable and set it to 60
# Your code here

Hint

Just create a variable min_savings and assign the value 60.

3

Calculate percentage savings and filter by threshold

Add a new column called percent_savings to instances_df that calculates the percentage savings as ((on_demand_price - spot_price) / on_demand_price) * 100. Then filter instances_df to only include rows where percent_savings is greater than or equal to min_savings. Save the filtered DataFrame as filtered_df.

Apache Spark

from pyspark.sql import SparkSession
from pyspark.sql.functions import col

spark = SparkSession.builder.appName('SpotInstances').getOrCreate()

instances_data = [
    ('m5.large', 0.096, 0.029),
    ('c5.large', 0.085, 0.025),
    ('r5.large', 0.126, 0.038)
]

instances_df = spark.createDataFrame(instances_data, ['instance_type', 'on_demand_price', 'spot_price'])

min_savings = 60

# Calculate percent_savings and filter instances_df
# Your code here

Hint

Use withColumn to add percent_savings and filter with col to filter rows.

4

Display the filtered DataFrame

Use filtered_df.show() to display the instance types with percentage savings greater than or equal to min_savings.

Apache Spark

from pyspark.sql import SparkSession
from pyspark.sql.functions import col

spark = SparkSession.builder.appName('SpotInstances').getOrCreate()

instances_data = [
    ('m5.large', 0.096, 0.029),
    ('c5.large', 0.085, 0.025),
    ('r5.large', 0.126, 0.038)
]

instances_df = spark.createDataFrame(instances_data, ['instance_type', 'on_demand_price', 'spot_price'])

min_savings = 60

instances_df = instances_df.withColumn('percent_savings', ((col('on_demand_price') - col('spot_price')) / col('on_demand_price')) * 100)

filtered_df = instances_df.filter(col('percent_savings') >= min_savings)

# Display the filtered DataFrame
# Your code here

Hint

Use filtered_df.show() to print the DataFrame.

Practice

1.

What is the main benefit of using spot instances for Apache Spark jobs?

easy

A. They provide more memory for Spark executors.

B. They increase the speed of Spark jobs.

C. They guarantee zero interruptions during job execution.

D. They reduce cloud computing costs significantly.

Solution

1. Step 1: Understand spot instances purpose

   Spot instances are cheaper cloud resources that can be interrupted but cost less.

2. Step 2: Identify main benefit for Spark jobs

   Using spot instances lowers the cost of running Spark jobs but may cause interruptions.

3. Final Answer:

   They reduce cloud computing costs significantly. -> Option D

4. Quick Check:

   Spot instances = cost savings [OK]

Hint: Spot instances save money but may interrupt jobs [OK]

Common Mistakes:

• Thinking spot instances speed up jobs
• Assuming spot instances never interrupt
• Believing spot instances add more memory

2.

Which of the following is the correct way to specify spot instances in a Spark cluster configuration on a cloud platform?

spark-submit --conf spark.executor.instances=5 --conf spark.executor.spot=true

easy

A. You must use --spot-instance flag in spark-submit.

B. This is correct syntax to request spot instances.

C. Spot instances are configured outside Spark code, not with spark-submit options.

D. Spot instances require setting spark.executor.memory.spot=true.

Solution

1. Step 1: Understand where spot instance setup happens

   Spot instance configuration is done in the cloud platform, not inside Spark submit commands.

2. Step 2: Analyze the given spark-submit command

   The command tries to set spot instances via Spark configs, which is incorrect.

3. Final Answer:

   Spot instances are configured outside Spark code, not with spark-submit options. -> Option C

4. Quick Check:

   Spot config = cloud platform setup [OK]

Hint: Spot instance setup is outside Spark code [OK]

Common Mistakes:

• Trying to set spot instances via spark-submit options
• Using incorrect flags like --spot-instance
• Confusing memory settings with spot instance flags

3.

Consider this Spark job running on a cluster with spot instances enabled. The job is flexible and can handle interruptions. What is the expected effect on the job's cost and runtime?

Cost = ?; Runtime = ?

medium

A. Cost decreases; runtime may increase due to interruptions.

B. Cost increases; runtime decreases due to faster nodes.

C. Cost and runtime both decrease.

D. Cost stays the same; runtime increases.

Solution

1. Step 1: Understand spot instance behavior

   Spot instances are cheaper but can be interrupted, causing retries or delays.

2. Step 2: Predict cost and runtime impact

   Cost goes down because spot instances are cheaper; runtime may increase due to interruptions and restarts.

3. Final Answer:

   Cost decreases; runtime may increase due to interruptions. -> Option A

4. Quick Check:

   Spot = cheaper but possibly slower [OK]

Hint: Spot instances cut cost but may slow jobs [OK]

Common Mistakes:

• Assuming spot instances speed up jobs
• Thinking cost increases with spot instances
• Ignoring possible interruptions

4.

You configured your Spark cluster to use spot instances, but your job keeps failing unexpectedly. Which of the following is the most likely cause?

medium

A. You did not allocate enough memory in Spark configs.

B. Spot instances were interrupted by the cloud provider.

C. Spark code has syntax errors.

D. The Spark version is incompatible with spot instances.

Solution

1. Step 1: Identify common failure reasons with spot instances

   Spot instances can be interrupted anytime by the cloud provider, causing job failures.

2. Step 2: Evaluate other options

   Syntax errors or memory issues cause different errors; incompatibility is rare and unrelated to spot instances.

3. Final Answer:

   Spot instances were interrupted by the cloud provider. -> Option B

4. Quick Check:

   Spot interruptions cause failures [OK]

Hint: Spot interruptions cause unexpected failures [OK]

Common Mistakes:

• Blaming Spark code syntax for spot instance failures
• Ignoring cloud provider interruptions
• Assuming memory or version issues cause spot failures

5.

You want to run a Spark job that processes large data but can tolerate interruptions. You have a limited budget. How should you configure your cluster to save costs using spot instances?

Choose the best approach.

hard

A. Use spot instances for all executors and implement checkpointing to recover from interruptions.

B. Use only on-demand instances to avoid interruptions and save costs.

C. Mix spot and on-demand instances without any fault tolerance in your job.

D. Use spot instances but disable retries to save time.

Solution

1. Step 1: Understand workload tolerance and cost constraints

   The job can handle interruptions and budget is limited, so cost saving is priority.

2. Step 2: Choose cluster setup for cost and fault tolerance

   Using spot instances saves money; checkpointing helps recover from interruptions safely.

3. Step 3: Evaluate other options

   Only on-demand is costly; mixing without fault tolerance risks failures; disabling retries risks data loss.

4. Final Answer:

   Use spot instances for all executors and implement checkpointing to recover from interruptions. -> Option A

5. Quick Check:

   Spot + checkpointing = cost saving + reliability [OK]

Hint: Combine spot instances with checkpointing for savings and safety [OK]

Common Mistakes:

• Avoiding spot instances despite budget limits
• Not implementing fault tolerance with spot instances
• Disabling retries risking job failure