Cluster sizing and auto-scaling in Apache Spark - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

Understanding Cluster Sizing Impact

What is the primary effect of increasing the number of worker nodes in a Spark cluster on job execution?

A It has no effect on execution time as Spark jobs run sequentially regardless of cluster size.

B It increases the execution time because more nodes cause more network overhead.

C It decreases the total execution time by allowing more tasks to run in parallel.

D It causes Spark to use less memory per node, slowing down the job.

💡 Hint

Think about how parallel processing works in Spark.

Adding more worker nodes allows Spark to distribute tasks across more machines, reducing total execution time by parallelizing work.

❓ Predict Output

intermediate

Auto-scaling Behavior in Spark

Given the following Spark configuration snippet, what will happen when the workload decreases significantly?

spark.conf.set("spark.dynamicAllocation.enabled", "true")
spark.conf.set("spark.dynamicAllocation.minExecutors", "2")
spark.conf.set("spark.dynamicAllocation.maxExecutors", "10")
spark.conf.set("spark.dynamicAllocation.initialExecutors", "5")

A The number of executors will scale up to 10 when workload decreases.

B The number of executors will scale down but never go below 2.

C The number of executors will remain fixed at 5 regardless of workload.

D The number of executors will immediately drop to 0 when workload decreases.

💡 Hint

Consider the minimum executors setting in dynamic allocation.

Dynamic allocation scales executors between the minimum and maximum limits based on workload. It will not go below the minimum of 2 executors.

❓ data_output

advanced

Analyzing Cluster Utilization Metrics

Given this Spark cluster utilization data collected over 5 minutes:

Minute: 1, CPU Usage: 85%, Memory Usage: 70%
Minute: 2, CPU Usage: 90%, Memory Usage: 75%
Minute: 3, CPU Usage: 95%, Memory Usage: 80%
Minute: 4, CPU Usage: 92%, Memory Usage: 78%
Minute: 5, CPU Usage: 88%, Memory Usage: 74%

What is the best interpretation of this data regarding cluster sizing?

A The cluster is mostly idle and can reduce nodes to save costs.

B The cluster memory is over-utilized but CPU is under-utilized.

C The cluster has balanced load and no changes are needed.

D The cluster is under heavy load and may benefit from adding more nodes.

💡 Hint

Look at CPU and memory usage percentages to assess load.

CPU usage consistently above 85% indicates high load, suggesting the cluster may need more nodes to handle workload efficiently.

🔧 Debug

advanced

Identifying Auto-scaling Misconfiguration

Review this Spark configuration snippet and identify the issue that prevents auto-scaling from working properly:

spark.conf.set("spark.dynamicAllocation.enabled", "true")
spark.conf.set("spark.dynamicAllocation.minExecutors", "5")
spark.conf.set("spark.dynamicAllocation.maxExecutors", "3")
spark.conf.set("spark.dynamicAllocation.initialExecutors", "4")

A minExecutors is set higher than maxExecutors, causing a configuration conflict.

B dynamicAllocation.enabled is set to true, which disables auto-scaling.

C initialExecutors is less than minExecutors, causing Spark to ignore it.

D maxExecutors is less than initialExecutors, which is allowed and causes no issues.

💡 Hint

Check the relationship between minExecutors and maxExecutors values.

minExecutors must be less than or equal to maxExecutors. Here minExecutors=5 is greater than maxExecutors=3, causing a conflict that prevents proper auto-scaling.

🚀 Application

expert

Optimizing Cluster Size for Variable Workloads

You manage a Spark cluster with dynamic workloads that spike unpredictably. Which strategy best balances cost and performance using auto-scaling?

A Set a low minExecutors to save costs and a high maxExecutors to handle spikes, enabling fast scaling.

B Set minExecutors equal to maxExecutors to keep cluster size fixed and avoid scaling delays.

C Disable dynamic allocation and manually add nodes during spikes to control costs.

D Set minExecutors high and maxExecutors low to prevent scaling and maintain stability.

💡 Hint

Consider how dynamic allocation adapts to workload changes.

Setting a low minimum keeps costs down during low usage, while a high maximum allows the cluster to scale up quickly during spikes, balancing cost and performance.