Cost optimization at scale in MLOps - Step-by-Step Execution

Process Flow - Cost optimization at scale

Identify high cost areas

↓

Analyze resource usage

↓

Apply cost-saving strategies

↓

Monitor cost impact

↓

Adjust and optimize continuously

↓

Repeat

This flow shows how to find expensive parts, analyze usage, apply savings, monitor results, and keep improving costs.

Execution Sample

MLOps

resources = {'GPU_hours': 1000, 'Storage_GB': 5000}

cost_per_unit = {'GPU_hours': 2, 'Storage_GB': 0.1}

initial_cost = sum(resources[r] * cost_per_unit[r] for r in resources)

resources['GPU_hours'] = 700  # optimize GPU usage

optimized_cost = sum(resources[r] * cost_per_unit[r] for r in resources)

Calculates initial cost, reduces GPU hours, then calculates optimized cost.

Process Table

Step	Resources	Cost Calculation	Cost Value	Action
1	{'GPU_hours': 1000, 'Storage_GB': 5000}	1000*2 + 5000*0.1	2000 + 500 = 2500	Calculate initial cost
2	{'GPU_hours': 700, 'Storage_GB': 5000}	700*2 + 5000*0.1	1400 + 500 = 1900	Reduced GPU hours to optimize cost
3	{'GPU_hours': 700, 'Storage_GB': 5000}	No change	1900	Final optimized cost

💡 Optimization applied by reducing GPU hours, lowering total cost from 2500 to 1900

Status Tracker

Variable	Start	After Step 2	Final
resources['GPU_hours']	1000	700	700
resources['Storage_GB']	5000	5000	5000
initial_cost	2500	2500	2500
optimized_cost	N/A	1900	1900

Key Moments - 2 Insights

Why does reducing GPU hours lower the total cost significantly?

Because GPU hours cost 2 units each, which is much higher than storage cost of 0.1 units per GB, so cutting GPU usage impacts cost more (see execution_table step 2).

Why does storage cost remain the same after optimization?

Storage amount was not changed, so its cost stays constant at 500 units (execution_table steps 1-3).

Visual Quiz - 3 Questions

Look at the execution table, what is the total cost at step 1?

A 1900

B 2500

C 2000

D 500

💡 Hint

Check the 'Cost Value' column at step 1 in the execution_table.

At which step does the GPU hours value change?

A Step 1

B Step 3

C Step 2

D No change

💡 Hint

Look at the 'Resources' column and variable_tracker for GPU_hours changes.

If storage was reduced to 4000 GB at step 2, what would happen to the optimized cost?

A It would decrease

B It would stay the same

C It would increase

D It would become zero

💡 Hint

Lower storage means lower cost since cost per GB is 0.1 (see cost calculation in execution_table).

Concept Snapshot

Cost optimization at scale:
- Identify costly resources
- Measure usage and cost per unit
- Apply reductions on expensive resources
- Recalculate costs to see savings
- Monitor and repeat for continuous improvement

Full Transcript

Cost optimization at scale means finding where your system spends the most money, like GPU hours or storage. You check how much each resource costs and how much you use. Then you reduce usage of the most expensive parts, like cutting GPU hours from 1000 to 700. This lowers your total cost from 2500 to 1900 units. Storage cost stays the same if you don't change storage size. You keep watching costs and usage to find more savings over time.

Practice

1. What is the main goal of cost optimization at scale in MLOps?

easy

A. To increase the number of servers regardless of workload

B. To avoid monitoring costs after deployment

C. To use only the most expensive cloud resources

D. To save money by matching resource use to workload needs

Solution

1. Step 1: Understand cost optimization purpose

   Cost optimization means using resources efficiently to reduce expenses.

2. Step 2: Match resources to workload needs

   Adjusting resources based on workload avoids waste and saves money.

3. Final Answer:

   To save money by matching resource use to workload needs -> Option D

4. Quick Check:

   Cost optimization = save money by matching resources [OK]

Hint: Cost optimization means using just enough resources [OK]

Common Mistakes:

• Thinking more servers always means better
• Ignoring cost monitoring after deployment
• Assuming expensive resources are always best

2. Which of the following is a correct way to specify a spot instance in a Kubernetes pod spec for cost savings?

easy

A. affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: "kubernetes.io/lifecycle"
operator: In
values:
- spot

B. tolerations:
- key: "spot-instance"
operator: Exists
effect: NoSchedule

C. nodeSelector:
kubernetes.io/instance-type: spot

D. resources:
requests:
cpu: "spot"
memory: "spot"

Solution

1. Step 1: Understand spot instance labeling in Kubernetes

   Spot instances are often labeled with lifecycle=spot to identify cheaper nodes.

2. Step 2: Check node affinity syntax

   affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/lifecycle" operator: In values: - spot correctly uses nodeAffinity with matchExpressions to select nodes labeled as spot.

3. Final Answer:

   affinity with nodeSelectorTerms matching lifecycle=spot label -> Option A

4. Quick Check:

   Spot instance selection uses nodeAffinity with lifecycle=spot label [OK]

Hint: Use nodeAffinity with lifecycle=spot label for spot nodes [OK]

Common Mistakes:

• Using nodeSelector with wrong label key
• Setting resource requests to 'spot' (invalid)
• Confusing tolerations with node affinity

3. Given this autoscaling configuration snippet for a Kubernetes deployment:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: ml-model-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: ml-model
  minReplicas: 2
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 50

What happens when CPU usage rises to 75%?

medium

A. The number of pods will increase up to a maximum of 10

B. The number of pods will decrease to 2

C. The deployment will restart pods

D. Nothing changes because CPU target is 50%

Solution

1. Step 1: Understand Horizontal Pod Autoscaler (HPA) behavior

   HPA increases pods when CPU usage exceeds target utilization to balance load.

2. Step 2: Analyze CPU usage vs target

   CPU is at 75%, above the 50% target, so HPA will scale up pods up to maxReplicas (10).

3. Final Answer:

   The number of pods will increase up to a maximum of 10 -> Option A

4. Quick Check:

   CPU > target utilization triggers pod scaling up [OK]

Hint: CPU above target utilization triggers scaling up [OK]

Common Mistakes:

• Thinking pods scale down when CPU rises
• Confusing pod restart with scaling
• Assuming no change if CPU exceeds target

4. You have a cloud cost alert system but it keeps sending false alarms about overspending. What is the most likely cause?

medium

A. The cloud provider is charging incorrectly

B. The alert thresholds are set too low or too sensitive

C. The system is not connected to the billing API

D. The cost data is updated only once a year

Solution

1. Step 1: Understand alert system sensitivity

   Alerts trigger when costs exceed set thresholds; too low thresholds cause false alarms.

2. Step 2: Evaluate other options

   Incorrect charges or missing billing data cause different issues, not false alarms.

3. Final Answer:

   The alert thresholds are set too low or too sensitive -> Option B

4. Quick Check:

   Low alert thresholds cause false alarms [OK]

Hint: Check alert thresholds if false alarms occur [OK]

Common Mistakes:

• Blaming cloud provider without proof
• Ignoring alert configuration
• Assuming billing API is always connected

5. You want to reduce costs for a large ML training job that runs daily on cloud GPUs. Which combined strategy best optimizes cost at scale?

hard

A. Run training on CPUs to avoid GPU costs without changing code

B. Use only on-demand GPU instances and disable autoscaling

C. Use spot GPU instances with checkpointing and autoscaling to handle interruptions

D. Schedule training during peak hours to use full capacity

Solution

1. Step 1: Identify cost-saving options for GPU jobs

   Spot instances are cheaper but can be interrupted; checkpointing saves progress.

2. Step 2: Combine autoscaling with spot instances and checkpointing

   Autoscaling adjusts resources; checkpointing prevents data loss on interruptions.

3. Step 3: Evaluate other options

   On-demand is costly; CPUs are slower; peak hours usually cost more.

4. Final Answer:

   Use spot GPU instances with checkpointing and autoscaling to handle interruptions -> Option C

5. Quick Check:

   Spot + checkpoint + autoscale = best cost optimization [OK]

Hint: Combine spot instances with checkpointing and autoscaling [OK]

Common Mistakes:

• Ignoring interruptions on spot instances
• Using expensive on-demand only
• Running on CPUs without code changes
• Scheduling during costly peak hours