Spot instances for cost savings in Apache Spark - Step-by-Step Execution

Concept Flow - Spot instances for cost savings

Request Spot Instance

↓

Instance Launched if Available

↓

Run Spark Job

↓

Spot Instance May be Interrupted

↓

Job Interrupted

↓

Handle Interruption: Retry or Save State

↓

Cost Savings Achieved

This flow shows how spot instances are requested, run Spark jobs, may get interrupted, and how handling interruptions leads to cost savings.

Execution Sample

Apache Spark

spark.conf.set("spark.dynamicAllocation.enabled", "true")
spark.conf.set("spark.executor.instances", "2")
spark.conf.set("spark.executor.spot", "true")

rdd = spark.sparkContext.parallelize(range(10))
print(rdd.collect())

This code configures Spark to use spot instances and runs a simple job collecting numbers 0 to 9.

Execution Table

Step	Action	Spot Instance Status	Job Status	Output
1	Request spot instances	Requested	Pending	No output yet
2	Spot instances launched	Active	Starting job	No output yet
3	Run Spark job tasks	Active	Running	Partial results processed
4	Spot instance interruption check	Active	Running	Partial results processed
5	Spot instance interrupted	Interrupted	Job failed	Job interrupted error
6	Handle interruption: retry job	Requested	Retrying	No output yet
7	Spot instances relaunched	Active	Running	Partial results processed
8	Job completes successfully	Active	Completed	[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
9	Calculate cost savings	N/A	N/A	Cost reduced by using spot instances

💡 Job completes successfully or is interrupted and retried until completion, achieving cost savings.

Variable Tracker

Variable	Start	After Step 2	After Step 5	After Step 7	Final
spot_instance_status	None	Active	Interrupted	Active	Active
job_status	Pending	Starting job	Job failed	Running	Completed
output	None	None	Job interrupted error	Partial results	[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]

Key Moments - 3 Insights

Why does the job fail at step 5 even though the spot instance was active before?

At step 5, the spot instance is interrupted by the cloud provider, causing the job to fail because the instance is no longer available to run tasks, as shown in the execution_table row 5.

How does Spark handle the interruption to still complete the job?

Spark retries the job by requesting new spot instances and rerunning tasks, as seen in steps 6 and 7 of the execution_table, allowing the job to eventually complete.

What is the main benefit of using spot instances despite interruptions?

The main benefit is cost savings, because spot instances are cheaper than regular instances, and handling interruptions with retries still results in lower overall cost, as summarized in step 9.

Visual Quiz - 3 Questions

Look at the execution table, what is the job status at step 4?

A Pending

B Running

C Completed

D Job failed

💡 Hint

Refer to execution_table row 4 under 'Job Status' column.

At which step does the spot instance get interrupted?

A Step 7

B Step 3

C Step 5

D Step 9

💡 Hint

Check execution_table row 5 under 'Spot Instance Status'.

If the job never got interrupted, which step would be skipped?

A Step 6 (Handle interruption: retry job)

B Step 2 (Spot instances launched)

C Step 8 (Job completes successfully)

D Step 1 (Request spot instances)

💡 Hint

Look at execution_table rows 5 and 6 to see what happens after interruption.

Concept Snapshot

Spot instances are cheaper cloud servers that can be interrupted.
Spark can run jobs on spot instances to save costs.
Jobs may fail if spot instances are interrupted.
Spark retries jobs automatically to handle interruptions.
This approach reduces cost but requires handling possible job restarts.

Full Transcript

Spot instances are cloud servers offered at lower prices but can be taken away anytime. When running Spark jobs on spot instances, the job starts by requesting these instances. If the instances are available, the job runs. However, spot instances can be interrupted, causing the job to fail. Spark handles this by retrying the job on new spot instances until it completes. This retry mechanism allows cost savings while ensuring job completion. The execution table shows each step from requesting spot instances, running the job, handling interruptions, and finally completing the job with cost savings.