Spot instances for cost savings in Apache Spark - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

Output of Spark job with spot instance interruption handling

Consider a Spark job running on spot instances that can be interrupted. What will be the output count of processed records if the job is configured to retry 2 times on failure and the spot instance is interrupted once during processing of 100 records?

Apache Spark

val data = spark.sparkContext.parallelize(1 to 100)
val processed = data.map(x => {
  if (x == 50) throw new RuntimeException("Spot instance interrupted")
  else x
}).count()

A 0

B 50

C 100

D 150

💡 Hint

Think about how retries affect the total processed count.

The job retries the failed task twice, so all 100 records are eventually processed successfully.

🧠 Conceptual

intermediate

Understanding cost savings with spot instances

Which of the following best explains why spot instances reduce costs in cloud-based Spark clusters?

A Spot instances are cheaper because they use idle cloud capacity that can be reclaimed anytime.

B Spot instances are cheaper because they use older hardware with lower performance.

C Spot instances are cheaper because they have fewer security features.

D Spot instances are cheaper because they run on dedicated physical servers.

💡 Hint

Think about how cloud providers price unused resources.

Spot instances are offered at lower prices because they use spare capacity that can be taken back by the provider at any time.

🔧 Debug

advanced

Identify the error in Spark spot instance handling code

What error will this Spark code produce when running on spot instances with possible interruptions? val rdd = spark.sparkContext.parallelize(1 to 10) val result = rdd.map(x => if (x == 5) throw new Exception("Instance lost") else x).collect() println(result.mkString(","))

Apache Spark

val rdd = spark.sparkContext.parallelize(1 to 10)
val result = rdd.map(x => if (x == 5) throw new Exception("Instance lost") else x).collect()
println(result.mkString(","))

A Exception: Instance lost

B Array(1, 2, 3, 4, 6, 7, 8, 9, 10)

C NullPointerException

D Job completes with output 1 to 10

💡 Hint

What happens when an exception is thrown inside a Spark transformation?

The exception thrown inside the map transformation causes the job to fail with that exception.

❓ data_output

advanced

Data output after handling spot instance interruptions with checkpointing

Given a Spark job that uses checkpointing to handle spot instance interruptions, what will be the output count after processing 200 records with an interruption at record 150?

Apache Spark

spark.sparkContext.setCheckpointDir("/tmp/checkpoint")
val data = spark.sparkContext.parallelize(1 to 200)
val processed = data.map(x => {
  if (x == 150) throw new RuntimeException("Spot instance interrupted")
  else x
}).checkpoint().count()

A 0

B 150

C 100

D 200

💡 Hint

Checkpointing helps recover from failures by saving progress.

Checkpointing saves the RDD state so that after interruption, Spark can recover and process all 200 records.

🚀 Application

expert

Optimizing Spark cluster cost with spot instances and autoscaling

You have a Spark cluster running on spot instances with autoscaling enabled. Which strategy will best minimize cost while maintaining job reliability?

A Use only on-demand instances with autoscaling for maximum reliability.

B Use a mix of spot and on-demand instances with autoscaling to handle interruptions gracefully.

C Use only spot instances without autoscaling to avoid interruptions.

D Disable autoscaling and use spot instances to save cost.

💡 Hint

Consider balancing cost savings and job reliability.

Mixing spot and on-demand instances with autoscaling allows cost savings while maintaining reliability by replacing interrupted spot instances.