Cluster sizing and auto-scaling in Apache Spark - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

Understanding Cluster Sizing Impact

What is the primary effect of increasing the number of worker nodes in a Spark cluster on job execution?

A It has no effect on execution time as Spark jobs run sequentially regardless of cluster size.

B It increases the execution time because more nodes cause more network overhead.

C It decreases the total execution time by allowing more tasks to run in parallel.

D It causes Spark to use less memory per node, slowing down the job.

💡 Hint

Think about how parallel processing works in Spark.

Adding more worker nodes allows Spark to distribute tasks across more machines, reducing total execution time by parallelizing work.

❓ Predict Output

intermediate

Auto-scaling Behavior in Spark

Given the following Spark configuration snippet, what will happen when the workload decreases significantly?

spark.conf.set("spark.dynamicAllocation.enabled", "true")
spark.conf.set("spark.dynamicAllocation.minExecutors", "2")
spark.conf.set("spark.dynamicAllocation.maxExecutors", "10")
spark.conf.set("spark.dynamicAllocation.initialExecutors", "5")

A The number of executors will scale up to 10 when workload decreases.

B The number of executors will scale down but never go below 2.

C The number of executors will remain fixed at 5 regardless of workload.

D The number of executors will immediately drop to 0 when workload decreases.

💡 Hint

Consider the minimum executors setting in dynamic allocation.

Dynamic allocation scales executors between the minimum and maximum limits based on workload. It will not go below the minimum of 2 executors.

❓ data_output

advanced

Analyzing Cluster Utilization Metrics

Given this Spark cluster utilization data collected over 5 minutes:

Minute: 1, CPU Usage: 85%, Memory Usage: 70%
Minute: 2, CPU Usage: 90%, Memory Usage: 75%
Minute: 3, CPU Usage: 95%, Memory Usage: 80%
Minute: 4, CPU Usage: 92%, Memory Usage: 78%
Minute: 5, CPU Usage: 88%, Memory Usage: 74%

What is the best interpretation of this data regarding cluster sizing?

A The cluster is mostly idle and can reduce nodes to save costs.

B The cluster memory is over-utilized but CPU is under-utilized.

C The cluster has balanced load and no changes are needed.

D The cluster is under heavy load and may benefit from adding more nodes.

💡 Hint

Look at CPU and memory usage percentages to assess load.

CPU usage consistently above 85% indicates high load, suggesting the cluster may need more nodes to handle workload efficiently.

🔧 Debug

advanced

Identifying Auto-scaling Misconfiguration

Review this Spark configuration snippet and identify the issue that prevents auto-scaling from working properly:

spark.conf.set("spark.dynamicAllocation.enabled", "true")
spark.conf.set("spark.dynamicAllocation.minExecutors", "5")
spark.conf.set("spark.dynamicAllocation.maxExecutors", "3")
spark.conf.set("spark.dynamicAllocation.initialExecutors", "4")

A minExecutors is set higher than maxExecutors, causing a configuration conflict.

B dynamicAllocation.enabled is set to true, which disables auto-scaling.

C initialExecutors is less than minExecutors, causing Spark to ignore it.

D maxExecutors is less than initialExecutors, which is allowed and causes no issues.

💡 Hint

Check the relationship between minExecutors and maxExecutors values.

minExecutors must be less than or equal to maxExecutors. Here minExecutors=5 is greater than maxExecutors=3, causing a conflict that prevents proper auto-scaling.

🚀 Application

expert

Optimizing Cluster Size for Variable Workloads

You manage a Spark cluster with dynamic workloads that spike unpredictably. Which strategy best balances cost and performance using auto-scaling?

A Set a low minExecutors to save costs and a high maxExecutors to handle spikes, enabling fast scaling.

B Set minExecutors equal to maxExecutors to keep cluster size fixed and avoid scaling delays.

C Disable dynamic allocation and manually add nodes during spikes to control costs.

D Set minExecutors high and maxExecutors low to prevent scaling and maintain stability.

💡 Hint

Consider how dynamic allocation adapts to workload changes.

Setting a low minimum keeps costs down during low usage, while a high maximum allows the cluster to scale up quickly during spikes, balancing cost and performance.

Practice

1. What does auto-scaling in Apache Spark do?

easy

A. Prevents any changes to cluster size during job execution

B. Manually sets the number of executors before running the job

C. Limits the job to run on a single executor only

D. Automatically adjusts the number of executors based on workload

Solution

1. Step 1: Understand auto-scaling concept

   Auto-scaling means Spark can add or remove executors automatically depending on the workload.

2. Step 2: Compare options with definition

   Only Automatically adjusts the number of executors based on workload describes automatic adjustment of executors based on workload, matching auto-scaling.

3. Final Answer:

   Automatically adjusts the number of executors based on workload -> Option D

4. Quick Check:

   Auto-scaling = automatic executor adjustment [OK]

Hint: Auto-scaling means Spark changes executors automatically [OK]

Common Mistakes:

• Confusing manual setting with auto-scaling
• Thinking auto-scaling fixes cluster size
• Assuming auto-scaling limits executors to one

2. Which of the following is the correct way to set minimum and maximum executors for auto-scaling in Spark configuration?

easy

A. spark.dynamicAllocation.minExecutors and spark.dynamicAllocation.maxExecutors

B. spark.executor.min and spark.executor.max

C. spark.autoScale.min and spark.autoScale.max

D. spark.executor.dynamic.min and spark.executor.dynamic.max

Solution

1. Step 1: Recall Spark config keys for auto-scaling

   Spark uses spark.dynamicAllocation.minExecutors and spark.dynamicAllocation.maxExecutors to set min and max executors.

2. Step 2: Match options with correct keys

   Only spark.dynamicAllocation.minExecutors and spark.dynamicAllocation.maxExecutors matches the exact Spark configuration keys for min and max executors.

3. Final Answer:

   spark.dynamicAllocation.minExecutors and spark.dynamicAllocation.maxExecutors -> Option A

4. Quick Check:

   Correct config keys = spark.dynamicAllocation.minExecutors and spark.dynamicAllocation.maxExecutors [OK]

Hint: Remember dynamicAllocation prefix for min/max executors [OK]

Common Mistakes:

• Using incorrect config prefixes
• Mixing executor and auto-scaling terms
• Assuming autoScale is a valid config prefix

3. Given this Spark configuration snippet:

spark.dynamicAllocation.enabled=true
spark.dynamicAllocation.minExecutors=2
spark.dynamicAllocation.maxExecutors=5
spark.dynamicAllocation.initialExecutors=3

What is the initial number of executors when the job starts?

medium

A. 3

B. 2

C. 5

D. Undefined, depends on workload

Solution

1. Step 1: Identify initialExecutors setting

   The config spark.dynamicAllocation.initialExecutors=3 sets the starting number of executors to 3.

2. Step 2: Confirm initialExecutors is between min and max

   3 is between minExecutors=2 and maxExecutors=5, so it is valid and used at start.

3. Final Answer:

   3 -> Option A

4. Quick Check:

   Initial executors = 3 [OK]

Hint: Initial executors come from initialExecutors setting [OK]

Common Mistakes:

• Confusing minExecutors with initialExecutors
• Assuming maxExecutors is the start
• Thinking initialExecutors depends on workload

4. You set spark.dynamicAllocation.minExecutors=4 and spark.dynamicAllocation.maxExecutors=3. What will happen when you run your Spark job?

medium

A. Spark will start with 4 executors ignoring maxExecutors

B. Spark will ignore minExecutors and use maxExecutors=3

C. Spark will throw a configuration error and not start

D. Spark will auto-scale between 3 and 4 executors normally

Solution

1. Step 1: Analyze min and max executor values

   Minimum executors cannot be greater than maximum executors; here min=4 > max=3, which is invalid.

2. Step 2: Understand Spark behavior on invalid config

   Spark detects this invalid config and throws an error instead of running the job.

3. Final Answer:

   Spark will throw a configuration error and not start -> Option C

4. Quick Check:

   minExecutors > maxExecutors causes error [OK]

Hint: Min executors must be ≤ max executors to avoid errors [OK]

Common Mistakes:

• Assuming Spark silently fixes invalid min/max
• Thinking Spark ignores minExecutors
• Believing auto-scaling works with reversed limits

5. You want your Spark job to start with 5 executors, but never go below 3 or above 10 executors during auto-scaling. Which configuration settings should you use?

hard

A. spark.dynamicAllocation.initialExecutors=10, spark.dynamicAllocation.minExecutors=5, spark.dynamicAllocation.maxExecutors=3

B. spark.dynamicAllocation.initialExecutors=5, spark.dynamicAllocation.minExecutors=3, spark.dynamicAllocation.maxExecutors=10

C. spark.dynamicAllocation.maxExecutors=5, spark.dynamicAllocation.minExecutors=3, spark.dynamicAllocation.initialExecutors=10

D. spark.dynamicAllocation.minExecutors=5, spark.dynamicAllocation.initialExecutors=3, spark.dynamicAllocation.maxExecutors=10

Solution

1. Step 1: Identify correct roles of each setting

   initialExecutors sets starting executors, minExecutors sets minimum allowed, and maxExecutors sets maximum allowed.

2. Step 2: Match values to requirements

   Start with 5 executors means initialExecutors=5, minimum 3 means minExecutors=3, maximum 10 means maxExecutors=10.

3. Final Answer:

   spark.dynamicAllocation.initialExecutors=5, spark.dynamicAllocation.minExecutors=3, spark.dynamicAllocation.maxExecutors=10 -> Option B

4. Quick Check:

   Initial=5, Min=3, Max=10 correct config [OK]

Hint: Initial executors = start, min ≤ initial ≤ max [OK]

Common Mistakes:

• Mixing up min and initial values
• Setting max lower than min
• Assigning initial outside min-max range