Dynamic resource allocation in Apache Spark - Interactive Code Practice

Practice - 5 Tasks

1fill in blank

easy

Complete the code to enable dynamic allocation in Spark configuration.

Apache Spark

spark.conf.set("spark.dynamicAllocation.enabled", [1])

Drag options to blanks, or click blank then click option'

A"true"

B"false"

C"auto"

D"enabled"

Attempts:

3 left

💡 Hint

Common Mistakes

Using "false" disables dynamic allocation.

Using "auto" or "enabled" are not valid values for this setting.

Setting spark.dynamicAllocation.enabled to "true" activates dynamic resource allocation in Spark.

2fill in blank

medium

Complete the code to set the minimum number of executors for dynamic allocation.

Apache Spark

spark.conf.set("spark.dynamicAllocation.minExecutors", [1])

Drag options to blanks, or click blank then click option'

A"0"

B"5"

C"1"

D"10"

Attempts:

3 left

💡 Hint

Common Mistakes

Setting minimum executors to "0" may cause no executors to run.

Setting too high a minimum may waste resources.

The minimum number of executors should be set to "1" or more to ensure at least one executor is allocated.

3fill in blank

hard

Fix the error in the code to set the initial number of executors for dynamic allocation.

Apache Spark

spark.conf.set("spark.dynamicAllocation.initialExecutors", [1])

Drag options to blanks, or click blank then click option'

A"5"

B5

Cinitial

D"five"

Attempts:

3 left

💡 Hint

Common Mistakes

Using a string like "five" instead of a numeric string.

Not putting quotes around the number makes it an int, causing TypeError.

The value must be a string representing an integer, not a bare integer. So use "5" with quotes.

4fill in blank

hard

Fill both blanks to configure Spark to enable dynamic allocation and set the shuffle tracking mode.

Apache Spark

spark.conf.set("spark.dynamicAllocation.enabled", [1])
spark.conf.set("spark.dynamicAllocation.shuffleTracking.enabled", [2])

Drag options to blanks, or click blank then click option'

A"true"

B"false"

Attempts:

3 left

💡 Hint

Common Mistakes

Setting shuffle tracking to "false" disables an important feature.

Using inconsistent values for these related settings.

Both settings should be set to "true" to enable dynamic allocation and shuffle tracking.

5fill in blank

hard

Fill all three blanks to create a Spark session with dynamic allocation enabled and set min and max executors.

Apache Spark

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("DynamicAllocationApp")
    .config("spark.dynamicAllocation.enabled", [1])
    .config("spark.dynamicAllocation.minExecutors", [2])
    .config("spark.dynamicAllocation.maxExecutors", [3])
    .getOrCreate()

Drag options to blanks, or click blank then click option'

A"true"

B"1"

C"10"

D"false"

Attempts:

3 left

💡 Hint

Common Mistakes

Disabling dynamic allocation by setting it to "false".

Setting min executors higher than max executors.

Using numbers without quotes in config strings.

Enable dynamic allocation with "true", set minimum executors to "1", and maximum to "10" for balanced resource use.

Practice

1. What is the main purpose of enabling spark.dynamicAllocation.enabled in Apache Spark?

easy

A. To disable Spark's ability to allocate resources dynamically

B. To allow Spark to automatically adjust the number of executors based on workload

C. To fix the number of executors throughout the job execution

D. To manually assign executors before job starts

Solution

1. Step 1: Understand dynamic allocation concept

   Dynamic allocation lets Spark add or remove executors automatically depending on workload changes.

2. Step 2: Identify the effect of enabling the setting

   Setting spark.dynamicAllocation.enabled to true activates this automatic adjustment feature.

3. Final Answer:

   To allow Spark to automatically adjust the number of executors based on workload -> Option B

4. Quick Check:

   Dynamic allocation enabled = automatic executor adjustment [OK]

Hint: Dynamic allocation means Spark changes executors automatically [OK]

Common Mistakes:

• Thinking dynamic allocation fixes executor count
• Confusing manual and automatic resource management
• Assuming dynamic allocation disables executor changes

2. Which of the following is the correct way to enable dynamic resource allocation in Spark configuration?

easy

A. spark.dynamicAllocation.enabled = true

B. spark.dynamicallocation.enabled = true

C. spark.dynamicAllocation.enable = true

D. spark.dynamicAllocation = true

Solution

1. Step 1: Recall the exact configuration key

   The correct key is spark.dynamicAllocation.enabled with exact casing.

2. Step 2: Match the correct syntax

   spark.dynamicAllocation.enabled = true uses the exact key and sets it to true, which is the correct syntax.

3. Final Answer:

   spark.dynamicAllocation.enabled = true -> Option A

4. Quick Check:

   Correct config key spelling = spark.dynamicAllocation.enabled = true [OK]

Hint: Remember exact config key spelling and casing [OK]

Common Mistakes:

• Misspelling 'enabled' or 'dynamicAllocation'
• Using wrong casing in config keys
• Omitting '.enabled' suffix

3. Given the Spark configuration below, what is the maximum number of executors Spark can allocate dynamically?

spark.dynamicAllocation.enabled = true
spark.dynamicAllocation.minExecutors = 2
spark.dynamicAllocation.maxExecutors = 10
spark.dynamicAllocation.initialExecutors = 3

medium

A. 10

B. 2

C. 3

D. 15

Solution

1. Step 1: Identify maxExecutors setting

   The spark.dynamicAllocation.maxExecutors sets the upper limit of executors Spark can allocate.

2. Step 2: Read the maxExecutors value

   It is set to 10, so Spark will not allocate more than 10 executors dynamically.

3. Final Answer:

   10 -> Option A

4. Quick Check:

   Max executors = 10 [OK]

Hint: Max executors is set by spark.dynamicAllocation.maxExecutors [OK]

Common Mistakes:

• Confusing minExecutors with maxExecutors
• Choosing initialExecutors as max
• Assuming maxExecutors is unlimited

4. You have enabled dynamic allocation but notice Spark never scales beyond 1 executor. Which configuration is most likely missing or incorrect?

medium

A. spark.dynamicAllocation.minExecutors is set to 1

B. spark.dynamicAllocation.enabled is set to false

C. spark.dynamicAllocation.maxExecutors is set to 1

D. spark.dynamicAllocation.initialExecutors is set to 5

Solution

1. Step 1: Analyze why scaling is limited

   If Spark never scales beyond 1 executor, the maxExecutors setting might be limiting it.

2. Step 2: Check maxExecutors value

   If spark.dynamicAllocation.maxExecutors is set to 1, Spark cannot allocate more than 1 executor.

3. Final Answer:

   spark.dynamicAllocation.maxExecutors is set to 1 -> Option C

4. Quick Check:

   Max executors limit = 1 causes no scaling [OK]

Hint: Check maxExecutors if scaling is stuck at 1 executor [OK]

Common Mistakes:

• Assuming minExecutors limits max scaling
• Forgetting to enable dynamic allocation
• Confusing initialExecutors with maxExecutors

5. You want Spark to start with 4 executors, allow scaling down to 2, and up to 8 executors dynamically. Which configuration snippet correctly sets this up?

hard

A. spark.dynamicAllocation.enabled = true
spark.dynamicAllocation.initialExecutors = 8
spark.dynamicAllocation.minExecutors = 4
spark.dynamicAllocation.maxExecutors = 2

B. spark.dynamicAllocation.enabled = true
spark.dynamicAllocation.minExecutors = 4
spark.dynamicAllocation.initialExecutors = 2
spark.dynamicAllocation.maxExecutors = 8

C. spark.dynamicAllocation.enabled = false
spark.dynamicAllocation.initialExecutors = 4
spark.dynamicAllocation.minExecutors = 2
spark.dynamicAllocation.maxExecutors = 8

D. spark.dynamicAllocation.enabled = true
spark.dynamicAllocation.initialExecutors = 4
spark.dynamicAllocation.minExecutors = 2
spark.dynamicAllocation.maxExecutors = 8

Solution

1. Step 1: Enable dynamic allocation

   The setting spark.dynamicAllocation.enabled must be true to activate dynamic resource allocation.

2. Step 2: Set initial, min, and max executors correctly

   Initial executors should be 4, minExecutors 2, and maxExecutors 8 to match requirements.

3. Step 3: Verify no conflicting settings

   spark.dynamicAllocation.enabled = true
   spark.dynamicAllocation.initialExecutors = 4
   spark.dynamicAllocation.minExecutors = 2
   spark.dynamicAllocation.maxExecutors = 8

   matches all requirements correctly; others have wrong values or disable dynamic allocation.

4. Final Answer:

   Correct configuration with enabled dynamic allocation and proper min, max, initial executors -> Option D

5. Quick Check:

   Enabled + initial=4 + min=2 + max=8 =

   spark.dynamicAllocation.enabled = true
   spark.dynamicAllocation.initialExecutors = 4
   spark.dynamicAllocation.minExecutors = 2
   spark.dynamicAllocation.maxExecutors = 8

   [OK]

Hint: Enable dynamic allocation and set min, max, initial executors properly [OK]

Common Mistakes:

• Disabling dynamic allocation by mistake
• Setting maxExecutors less than minExecutors
• Swapping initialExecutors and minExecutors values