Why scaling requires different strategies in MLOps - Why It Works

Introduction

Scaling means making your system handle more work or users. Different parts of your system may need different ways to grow because they work in different ways and have different limits.

When your app gets more users and you need to keep it fast and reliable

When your database grows and simple copying is not enough

When your machine learning model needs more computing power to train faster

When you want to add more servers but keep everything working together smoothly

When you need to balance cost and performance as your system grows

Commands

This command increases the number of running copies (pods) of your machine learning model deployment to 3. It helps handle more requests by adding more instances.

Terminal

kubectl scale deployment my-ml-model --replicas=3

Expected Output

deployment.apps/my-ml-model scaled

→

--replicas - Sets the desired number of pod replicas

This command lists all pods with the label 'app=my-ml-model' to verify that scaling created the new pods.

Terminal

kubectl get pods -l app=my-ml-model

Expected Output

NAME                        READY   STATUS    RESTARTS   AGE

my-ml-model-5d8f7c7f7f-abcde   1/1     Running   0          2m

my-ml-model-5d8f7c7f7f-bcdef   1/1     Running   0          1m

my-ml-model-5d8f7c7f7f-cdefg   1/1     Running   0          30s

This command sets up automatic scaling for the deployment. It will keep at least 2 pods and can add up to 5 pods based on CPU usage going above 50%.

Terminal

kubectl autoscale deployment my-ml-model --min=2 --max=5 --cpu-percent=50

Expected Output

horizontalpodautoscaler.autoscaling/my-ml-model autoscaled

→

--min - Minimum number of pods to keep running

→

--max - Maximum number of pods allowed

→

--cpu-percent - CPU usage threshold to trigger scaling

This command checks the status of the horizontal pod autoscaler to see current scaling activity.

Terminal

kubectl get hpa my-ml-model

Expected Output

NAME          REFERENCE              TARGETS   MINPODS   MAXPODS   REPLICAS   AGE

my-ml-model   Deployment/my-ml-model   50%/50%   2         5         3          5m

Key Concept

If you remember nothing else from this pattern, remember: different parts of your system need different scaling methods because they have unique limits and workloads.

Common Mistakes

Trying to scale only by adding more copies without checking resource limits

This can cause resource exhaustion or poor performance if the system bottleneck is elsewhere

Analyze which part limits performance and choose the right scaling method like vertical scaling or autoscaling

Setting autoscaling thresholds too high or too low

Too high means slow response to load changes; too low causes frequent scaling and instability

Set thresholds based on real usage patterns and monitor to adjust

Summary

Scaling means adjusting system capacity to handle more work.

Different parts like compute, storage, and network need different scaling strategies.

Commands like kubectl scale and autoscale help manage scaling in Kubernetes.

Practice

1. Why do systems need different scaling strategies as they grow?

easy

A. Because all systems grow at the same speed

B. Because scaling always means adding more machines

C. Because different growth patterns require different resource management

D. Because vertical scaling is always better than horizontal scaling

Solution

1. Step 1: Understand system growth patterns

   Systems grow in different ways, such as more users or more data, which affects resource needs differently.

2. Step 2: Match scaling strategy to growth type

   Different growth types require different scaling approaches to manage resources efficiently and keep performance.

3. Final Answer:

   Because different growth patterns require different resource management -> Option C

4. Quick Check:

   Growth patterns = Different strategies [OK]

Hint: Match scaling to how system grows for best results [OK]

Common Mistakes:

• Assuming one scaling method fits all
• Thinking scaling always means adding machines
• Ignoring resource limits of single machines

2. Which of the following is the correct way to describe vertical scaling?

easy

A. Adding more machines to handle more load

B. Making a single machine more powerful by adding CPU or RAM

C. Splitting data across multiple databases

D. Reducing the number of users on the system

Solution

1. Step 1: Define vertical scaling

   Vertical scaling means improving one machine's capacity by adding resources like CPU or memory.

2. Step 2: Compare options

   Making a single machine more powerful by adding CPU or RAM matches this definition; others describe horizontal scaling or unrelated actions.

3. Final Answer:

   Making a single machine more powerful by adding CPU or RAM -> Option B

4. Quick Check:

   Vertical scaling = stronger single machine [OK]

Hint: Vertical scaling = upgrade one machine's power [OK]

Common Mistakes:

• Confusing vertical with horizontal scaling
• Thinking vertical scaling means adding machines
• Selecting unrelated options like reducing users

3. Consider a system that uses horizontal scaling by adding identical servers behind a load balancer. What is the main benefit of this approach?

medium

A. It allows the system to handle more users by distributing load

B. It simplifies the software by using only one server

C. It reduces the need for network connections

D. It increases the power of a single server

Solution

1. Step 1: Understand horizontal scaling

   Horizontal scaling adds more servers to share the workload, improving capacity.

2. Step 2: Identify benefit of load balancing

   Load balancers distribute user requests across servers, allowing more users to be served efficiently.

3. Final Answer:

   It allows the system to handle more users by distributing load -> Option A

4. Quick Check:

   Horizontal scaling = distribute load [OK]

Hint: More servers = more users handled [OK]

Common Mistakes:

• Thinking horizontal scaling powers one server
• Believing it reduces network needs
• Assuming it simplifies software to one server

4. A team tried to scale their ML model serving by only upgrading the CPU and RAM of one server, but the system still slowed down under heavy user load. What is the likely problem?

medium

A. They must have a bug in the model code

B. They needed to reduce the model size instead

C. They should have used a faster programming language

D. They should have added more servers instead of upgrading one

Solution

1. Step 1: Analyze the scaling approach

   Upgrading one server is vertical scaling, which has limits and may not handle very high loads.

2. Step 2: Identify better scaling strategy

   Adding more servers (horizontal scaling) distributes load and improves performance under heavy use.

3. Final Answer:

   They should have added more servers instead of upgrading one -> Option D

4. Quick Check:

   Heavy load needs horizontal scaling [OK]

Hint: Heavy load? Add servers, not just power [OK]

Common Mistakes:

• Blaming model size without checking scaling
• Assuming programming language causes slowdown
• Ignoring scaling limits of single server

5. You manage an ML system that processes large datasets and serves predictions to many users. Vertical scaling is costly and limited. Which combined strategy best balances cost, performance, and reliability?

hard

A. Use horizontal scaling with multiple servers and optimize model efficiency

B. Only upgrade the biggest server continuously

C. Reduce the number of users to fit one server

D. Switch to a simpler model without scaling

Solution

1. Step 1: Evaluate vertical scaling limits

   Vertical scaling is costly and hits hardware limits, so relying on it alone is not sustainable.

2. Step 2: Combine horizontal scaling and optimization

   Adding servers (horizontal scaling) spreads load, while optimizing the model reduces resource use, balancing cost and performance.

3. Step 3: Consider reliability

   Multiple servers improve fault tolerance, making the system more reliable than a single powerful server.

4. Final Answer:

   Use horizontal scaling with multiple servers and optimize model efficiency -> Option A

5. Quick Check:

   Combine horizontal scaling + optimization = best balance [OK]

Hint: Combine adding servers with model optimization [OK]

Common Mistakes:

• Relying only on vertical scaling
• Ignoring user demand growth
• Choosing to reduce users instead of scaling
• Dropping scaling for simpler models only