Cluster sizing and auto-scaling in Apache Spark - Interactive Code Practice

Practice - 5 Tasks

1fill in blank

easy

Complete the code to create a Spark session with dynamic allocation enabled.

Apache Spark

from pyspark.sql import SparkSession
spark = SparkSession.builder.appName('App').config('spark.dynamicAllocation.enabled', [1]).getOrCreate()

Drag options to blanks, or click blank then click option'

ATrue

BFalse

C'yes'

DNone

Attempts:

3 left

💡 Hint

Common Mistakes

Using string values like 'yes' instead of boolean True

Setting dynamic allocation to False disables auto-scaling

Dynamic allocation must be set to True to enable auto-scaling of executors in Spark.

2fill in blank

medium

Complete the code to set the minimum number of executors to 2 in Spark configuration.

Apache Spark

spark.conf.set('spark.dynamicAllocation.minExecutors', [1])

Drag options to blanks, or click blank then click option'

ANone

B2

CTrue

D'2'

Attempts:

3 left

💡 Hint

Common Mistakes

Using integer 2 without quotes

Using boolean values instead of string numbers

The value for spark.dynamicAllocation.minExecutors must be a string representing the number of executors, e.g., '2'.

3fill in blank

hard

Fix the error in the code to correctly set the maximum number of executors to 10.

Apache Spark

spark.conf.set('spark.dynamicAllocation.maxExecutors', [1])

Drag options to blanks, or click blank then click option'

A10

BTrue

C'10'

DNone

Attempts:

3 left

💡 Hint

Common Mistakes

Using integer 10 without quotes

Using boolean or None instead of string

The maximum executors setting requires a string value, so use '10' with quotes.

4fill in blank

hard

Fill both blanks to create a dictionary that maps executor IDs to their memory sizes in GB.

Apache Spark

executor_memory = { [1]: [2] for [1] in ['exec1', 'exec2', 'exec3'] }

Drag options to blanks, or click blank then click option'

Ak

B8

C'exec2'

D16

Attempts:

3 left

💡 Hint

Common Mistakes

Using integers as keys

Using strings for memory sizes

The dictionary comprehension uses executor IDs as keys (strings like 'exec1') and memory sizes as values (integers like 8).

5fill in blank

hard

Fill all three blanks to create a filtered dictionary of executors with memory greater than 10 GB.

Apache Spark

filtered_memory = { [1]: [2] for [1] , [2] in executor_memory.items() if [2] > 10 }

Drag options to blanks, or click blank then click option'

Aexecutor

Bmem

Attempts:

3 left

💡 Hint

Common Mistakes

Mixing variable names

Using the wrong variable in the condition

The comprehension iterates over executor_memory.items() with variables executor and mem. It filters for memory values greater than 10.

Practice

1. What does auto-scaling in Apache Spark do?

easy

A. Prevents any changes to cluster size during job execution

B. Manually sets the number of executors before running the job

C. Limits the job to run on a single executor only

D. Automatically adjusts the number of executors based on workload

Solution

1. Step 1: Understand auto-scaling concept

   Auto-scaling means Spark can add or remove executors automatically depending on the workload.

2. Step 2: Compare options with definition

   Only Automatically adjusts the number of executors based on workload describes automatic adjustment of executors based on workload, matching auto-scaling.

3. Final Answer:

   Automatically adjusts the number of executors based on workload -> Option D

4. Quick Check:

   Auto-scaling = automatic executor adjustment [OK]

Hint: Auto-scaling means Spark changes executors automatically [OK]

Common Mistakes:

• Confusing manual setting with auto-scaling
• Thinking auto-scaling fixes cluster size
• Assuming auto-scaling limits executors to one

2. Which of the following is the correct way to set minimum and maximum executors for auto-scaling in Spark configuration?

easy

A. spark.dynamicAllocation.minExecutors and spark.dynamicAllocation.maxExecutors

B. spark.executor.min and spark.executor.max

C. spark.autoScale.min and spark.autoScale.max

D. spark.executor.dynamic.min and spark.executor.dynamic.max

Solution

1. Step 1: Recall Spark config keys for auto-scaling

   Spark uses spark.dynamicAllocation.minExecutors and spark.dynamicAllocation.maxExecutors to set min and max executors.

2. Step 2: Match options with correct keys

   Only spark.dynamicAllocation.minExecutors and spark.dynamicAllocation.maxExecutors matches the exact Spark configuration keys for min and max executors.

3. Final Answer:

   spark.dynamicAllocation.minExecutors and spark.dynamicAllocation.maxExecutors -> Option A

4. Quick Check:

   Correct config keys = spark.dynamicAllocation.minExecutors and spark.dynamicAllocation.maxExecutors [OK]

Hint: Remember dynamicAllocation prefix for min/max executors [OK]

Common Mistakes:

• Using incorrect config prefixes
• Mixing executor and auto-scaling terms
• Assuming autoScale is a valid config prefix

3. Given this Spark configuration snippet:

spark.dynamicAllocation.enabled=true
spark.dynamicAllocation.minExecutors=2
spark.dynamicAllocation.maxExecutors=5
spark.dynamicAllocation.initialExecutors=3

What is the initial number of executors when the job starts?

medium

A. 3

B. 2

C. 5

D. Undefined, depends on workload

Solution

1. Step 1: Identify initialExecutors setting

   The config spark.dynamicAllocation.initialExecutors=3 sets the starting number of executors to 3.

2. Step 2: Confirm initialExecutors is between min and max

   3 is between minExecutors=2 and maxExecutors=5, so it is valid and used at start.

3. Final Answer:

   3 -> Option A

4. Quick Check:

   Initial executors = 3 [OK]

Hint: Initial executors come from initialExecutors setting [OK]

Common Mistakes:

• Confusing minExecutors with initialExecutors
• Assuming maxExecutors is the start
• Thinking initialExecutors depends on workload

4. You set spark.dynamicAllocation.minExecutors=4 and spark.dynamicAllocation.maxExecutors=3. What will happen when you run your Spark job?

medium

A. Spark will start with 4 executors ignoring maxExecutors

B. Spark will ignore minExecutors and use maxExecutors=3

C. Spark will throw a configuration error and not start

D. Spark will auto-scale between 3 and 4 executors normally

Solution

1. Step 1: Analyze min and max executor values

   Minimum executors cannot be greater than maximum executors; here min=4 > max=3, which is invalid.

2. Step 2: Understand Spark behavior on invalid config

   Spark detects this invalid config and throws an error instead of running the job.

3. Final Answer:

   Spark will throw a configuration error and not start -> Option C

4. Quick Check:

   minExecutors > maxExecutors causes error [OK]

Hint: Min executors must be ≤ max executors to avoid errors [OK]

Common Mistakes:

• Assuming Spark silently fixes invalid min/max
• Thinking Spark ignores minExecutors
• Believing auto-scaling works with reversed limits

5. You want your Spark job to start with 5 executors, but never go below 3 or above 10 executors during auto-scaling. Which configuration settings should you use?

hard

A. spark.dynamicAllocation.initialExecutors=10, spark.dynamicAllocation.minExecutors=5, spark.dynamicAllocation.maxExecutors=3

B. spark.dynamicAllocation.initialExecutors=5, spark.dynamicAllocation.minExecutors=3, spark.dynamicAllocation.maxExecutors=10

C. spark.dynamicAllocation.maxExecutors=5, spark.dynamicAllocation.minExecutors=3, spark.dynamicAllocation.initialExecutors=10

D. spark.dynamicAllocation.minExecutors=5, spark.dynamicAllocation.initialExecutors=3, spark.dynamicAllocation.maxExecutors=10

Solution

1. Step 1: Identify correct roles of each setting

   initialExecutors sets starting executors, minExecutors sets minimum allowed, and maxExecutors sets maximum allowed.

2. Step 2: Match values to requirements

   Start with 5 executors means initialExecutors=5, minimum 3 means minExecutors=3, maximum 10 means maxExecutors=10.

3. Final Answer:

   spark.dynamicAllocation.initialExecutors=5, spark.dynamicAllocation.minExecutors=3, spark.dynamicAllocation.maxExecutors=10 -> Option B

4. Quick Check:

   Initial=5, Min=3, Max=10 correct config [OK]

Hint: Initial executors = start, min ≤ initial ≤ max [OK]

Common Mistakes:

• Mixing up min and initial values
• Setting max lower than min
• Assigning initial outside min-max range