Auto-scaling inference endpoints in MLOps - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

Understanding Auto-scaling Triggers

Which metric is most commonly used to trigger auto-scaling of inference endpoints in a cloud environment?

A Time of day

B Disk space usage

C Number of active users on the website

D CPU utilization percentage

💡 Hint

Think about what resource usage directly affects the ability to handle inference requests.

CPU utilization is a direct indicator of how busy the inference endpoint is. When CPU usage is high, scaling out helps handle more requests.

💻 Command Output

intermediate

Interpreting Auto-scaling CLI Output

Given the following CLI output from an auto-scaling tool monitoring an inference endpoint, what is the current number of active instances?

Endpoint: model-v1
Instances: 3
CPU Utilization: 75%
Scaling Status: Stable

A 3

B 75

C 1

D 0

💡 Hint

Look for the line that indicates how many instances are running.

The line 'Instances: 3' shows that there are currently 3 active instances serving the endpoint.

❓ Configuration

advanced

Configuring Auto-scaling Policy

Which YAML snippet correctly configures an auto-scaling policy to scale out when CPU usage exceeds 70% and scale in when below 30%?

A

autoScaling:
  minInstances: 1
  maxInstances: 5
  targetCPUUtilizationPercentage: 50

B

autoScaling:
  minInstances: 1
  maxInstances: 5
  cpuUtilization:
    scaleOut: 70
    scaleIn: 30

C

autoScaling:
  minInstances: 1
  maxInstances: 5
  scaleOutCPUThreshold: 70
  scaleInCPUThreshold: 30

D

autoScaling:
  minInstances: 1
  maxInstances: 5
  targetCPUUtilizationPercentage: 70

💡 Hint

Look for explicit scale out and scale in thresholds.

Option C explicitly sets scaleOutCPUThreshold and scaleInCPUThreshold, which are required to define when to scale out and in.

❓ Troubleshoot

advanced

Diagnosing Auto-scaling Failure

An inference endpoint is not scaling out despite high CPU usage. Which of the following is the most likely cause?

A The endpoint has no traffic

B The maxInstances limit is set to 1

C The CPU usage is below the scale-out threshold

D The auto-scaling feature is disabled in the cloud provider

💡 Hint

Check if the scaling limits allow more instances to be created.

If maxInstances is set to 1, the system cannot add more instances even if CPU usage is high, preventing scale out.

🔀 Workflow

expert

Auto-scaling Workflow for Inference Endpoint

Arrange the steps in the correct order for an auto-scaling workflow of an inference endpoint.

A 1,2,3,4

B 2,1,3,4

C 1,3,2,4

D 1,2,4,3

💡 Hint

Think about monitoring first, then triggering, then adding instances, then routing traffic.

The workflow starts with monitoring metrics, then triggers scaling, adds instances, and finally distributes requests.