Cluster sizing and auto-scaling in Apache Spark - Interactive Code Practice

Practice - 5 Tasks

1fill in blank

easy

Complete the code to create a Spark session with dynamic allocation enabled.

Apache Spark

from pyspark.sql import SparkSession
spark = SparkSession.builder.appName('App').config('spark.dynamicAllocation.enabled', [1]).getOrCreate()

Drag options to blanks, or click blank then click option'

ATrue

BFalse

C'yes'

DNone

Attempts:

3 left

💡 Hint

Common Mistakes

Using string values like 'yes' instead of boolean True

Setting dynamic allocation to False disables auto-scaling

Dynamic allocation must be set to True to enable auto-scaling of executors in Spark.

2fill in blank

medium

Complete the code to set the minimum number of executors to 2 in Spark configuration.

Apache Spark

spark.conf.set('spark.dynamicAllocation.minExecutors', [1])

Drag options to blanks, or click blank then click option'

ANone

B2

CTrue

D'2'

Attempts:

3 left

💡 Hint

Common Mistakes

Using integer 2 without quotes

Using boolean values instead of string numbers

The value for spark.dynamicAllocation.minExecutors must be a string representing the number of executors, e.g., '2'.

3fill in blank

hard

Fix the error in the code to correctly set the maximum number of executors to 10.

Apache Spark

spark.conf.set('spark.dynamicAllocation.maxExecutors', [1])

Drag options to blanks, or click blank then click option'

A10

BTrue

C'10'

DNone

Attempts:

3 left

💡 Hint

Common Mistakes

Using integer 10 without quotes

Using boolean or None instead of string

The maximum executors setting requires a string value, so use '10' with quotes.

4fill in blank

hard

Fill both blanks to create a dictionary that maps executor IDs to their memory sizes in GB.

Apache Spark

executor_memory = { [1]: [2] for [1] in ['exec1', 'exec2', 'exec3'] }

Drag options to blanks, or click blank then click option'

Ak

B8

C'exec2'

D16

Attempts:

3 left

💡 Hint

Common Mistakes

Using integers as keys

Using strings for memory sizes

The dictionary comprehension uses executor IDs as keys (strings like 'exec1') and memory sizes as values (integers like 8).

5fill in blank

hard

Fill all three blanks to create a filtered dictionary of executors with memory greater than 10 GB.

Apache Spark

filtered_memory = { [1]: [2] for [1] , [2] in executor_memory.items() if [2] > 10 }

Drag options to blanks, or click blank then click option'

Aexecutor

Bmem

Attempts:

3 left

💡 Hint

Common Mistakes

Mixing variable names

Using the wrong variable in the condition

The comprehension iterates over executor_memory.items() with variables executor and mem. It filters for memory values greater than 10.