Spot instances for cost savings in Apache Spark - Interactive Code Practice

Practice - 5 Tasks

1fill in blank

easy

Complete the code to create a Spark session that can use spot instances.

Apache Spark

from pyspark.sql import SparkSession
spark = SparkSession.builder.appName('SpotInstanceApp').config('spark.[1]', 'true').getOrCreate()

Drag options to blanks, or click blank then click option'

Aexecutor.instances

Bspark.executor.memory

CdynamicAllocation.enabled

Dspot.instances.enabled

Attempts:

3 left

💡 Hint

Common Mistakes

Choosing a configuration unrelated to dynamic allocation.

Using memory or instance count settings instead of enabling dynamic allocation.

The configuration spark.dynamicAllocation.enabled allows Spark to dynamically allocate executors, which is useful when using spot instances to save costs.

2fill in blank

medium

Complete the code to set the maximum number of executors for spot instances.

Apache Spark

spark.conf.set('spark.dynamicAllocation.maxExecutors', [1])

Drag options to blanks, or click blank then click option'

A'10'

B'5'

C'20'

D'0'

Attempts:

3 left

💡 Hint

Common Mistakes

Setting the value to zero disables executors.

Using a number too low to allow scaling.

Setting spark.dynamicAllocation.maxExecutors to '10' limits the maximum executors Spark can allocate, which helps control costs when using spot instances.

3fill in blank

hard

Fix the error in the code to request spot instances with a maximum bid price.

Apache Spark

spark.conf.set('spark.executor.instances', '5')
spark.conf.set('spark.executor.spotBidPrice', [1])

Drag options to blanks, or click blank then click option'

A0.5

B'0.5$'

C'$0.5'

D'0.5'

Attempts:

3 left

💡 Hint

Common Mistakes

Including currency symbols like '$' in the string.

Using a numeric type instead of a string.

The bid price should be a string representing a numeric value without currency symbols. So '0.5' is correct.

4fill in blank

hard

Fill both blanks to create a dictionary that maps instance types to their spot prices.

Apache Spark

spot_prices = { [1]: [2] for [1] in ['m4.large', 'm4.xlarge'] }

Drag options to blanks, or click blank then click option'

A'm4.large'

B0.05

C'm4.xlarge'

D0.10

Attempts:

3 left

💡 Hint

Common Mistakes

Using the wrong variable as key or value.

Mixing up instance types and prices.

The dictionary comprehension uses instance types as keys and their spot prices as values. Here, 'm4.large' is the key and 0.05 is the price.

5fill in blank

hard

Fill all three blanks to filter a DataFrame for spot instances with price less than 0.1 and select instance type and price.

Apache Spark

filtered_df = df.filter(df.price [1] [2]).select([3], 'price')

Drag options to blanks, or click blank then click option'

A<

B0.1

C'instance_type'

D'price'

Attempts:

3 left

💡 Hint

Common Mistakes

Using wrong comparison operators like '>' or '='.

Using incorrect column names or missing quotes.

The filter uses '<' to select rows where price is less than 0.1. Then it selects the 'instance_type' and 'price' columns.