Cost optimization at scale in MLOps - Commands & Configuration

Introduction

Running machine learning workloads in the cloud can become expensive quickly. Cost optimization at scale helps you reduce cloud spending by managing resources efficiently and automating cost-saving actions.

When you want to automatically stop idle or underused cloud compute instances to save money.

When you need to track and alert on unexpected spikes in cloud resource usage.

When you want to schedule training jobs during cheaper off-peak hours.

When you want to use cheaper spot instances for non-critical workloads.

When you want to monitor and optimize storage costs for large datasets.

Config File - cost_optimization_pipeline.py

cost_optimization_pipeline.py

import mlflow
import time

def check_idle_resources():
    # Simulate checking for idle resources
    print("Checking for idle resources...")
    return ["instance-123", "instance-456"]

def stop_resources(instances):
    for instance in instances:
        print(f"Stopping {instance} to save cost.")

def main():
    mlflow.start_run(run_name="cost_optimization")
    idle_instances = check_idle_resources()
    if idle_instances:
        stop_resources(idle_instances)
        mlflow.log_metric("stopped_instances", len(idle_instances))
    else:
        print("No idle resources found.")
        mlflow.log_metric("stopped_instances", 0)
    mlflow.end_run()

if __name__ == "__main__":
    main()

This Python script uses MLflow to track a cost optimization run.

The check_idle_resources function simulates finding idle cloud instances.

The stop_resources function simulates stopping those instances to save money.

Metrics about stopped instances are logged to MLflow for monitoring.

Commands

Run the cost optimization script to check for idle resources and stop them, logging metrics to MLflow.

Terminal

python cost_optimization_pipeline.py

Expected Output

Checking for idle resources...

Stopping instance-123 to save cost.

Stopping instance-456 to save cost.

Start the MLflow tracking UI to view logged metrics and runs for cost optimization.

Terminal

mlflow ui

Expected Output

2024/06/01 12:00:00 INFO mlflow.server: Starting MLflow tracking UI at http://127.0.0.1:5000

→

--host - Specify the network interface to listen on

→

--port - Specify the port for the UI

Key Concept

If you remember nothing else from this pattern, remember: automate detection and shutdown of idle resources to save cloud costs effectively.

Code Example

MLOps

import mlflow
import time

def check_idle_resources():
    print("Checking for idle resources...")
    return ["instance-123", "instance-456"]

def stop_resources(instances):
    for instance in instances:
        print(f"Stopping {instance} to save cost.")

def main():
    mlflow.start_run(run_name="cost_optimization")
    idle_instances = check_idle_resources()
    if idle_instances:
        stop_resources(idle_instances)
        mlflow.log_metric("stopped_instances", len(idle_instances))
    else:
        print("No idle resources found.")
        mlflow.log_metric("stopped_instances", 0)
    mlflow.end_run()

if __name__ == "__main__":
    main()

Common Mistakes

Not logging cost-saving metrics to MLflow.

Without metrics, you cannot track or prove cost savings over time.

Always log key metrics like number of stopped instances or hours saved.

Stopping critical resources by mistake.

This causes downtime and disrupts production workloads.

Implement safeguards to identify only truly idle or non-critical resources.

Running cost optimization scripts manually and irregularly.

Manual runs miss opportunities to save money continuously.

Schedule scripts to run automatically at regular intervals.

Summary

Run a script to detect and stop idle cloud resources to save costs.

Log cost-saving metrics to MLflow for tracking and analysis.

Use MLflow UI to monitor cost optimization runs and results.

Practice

1. What is the main goal of cost optimization at scale in MLOps?

easy

A. To increase the number of servers regardless of workload

B. To avoid monitoring costs after deployment

C. To use only the most expensive cloud resources

D. To save money by matching resource use to workload needs

Solution

1. Step 1: Understand cost optimization purpose

   Cost optimization means using resources efficiently to reduce expenses.

2. Step 2: Match resources to workload needs

   Adjusting resources based on workload avoids waste and saves money.

3. Final Answer:

   To save money by matching resource use to workload needs -> Option D

4. Quick Check:

   Cost optimization = save money by matching resources [OK]

Hint: Cost optimization means using just enough resources [OK]

Common Mistakes:

• Thinking more servers always means better
• Ignoring cost monitoring after deployment
• Assuming expensive resources are always best

2. Which of the following is a correct way to specify a spot instance in a Kubernetes pod spec for cost savings?

easy

A. affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: "kubernetes.io/lifecycle"
operator: In
values:
- spot

B. tolerations:
- key: "spot-instance"
operator: Exists
effect: NoSchedule

C. nodeSelector:
kubernetes.io/instance-type: spot

D. resources:
requests:
cpu: "spot"
memory: "spot"

Solution

1. Step 1: Understand spot instance labeling in Kubernetes

   Spot instances are often labeled with lifecycle=spot to identify cheaper nodes.

2. Step 2: Check node affinity syntax

   affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/lifecycle" operator: In values: - spot correctly uses nodeAffinity with matchExpressions to select nodes labeled as spot.

3. Final Answer:

   affinity with nodeSelectorTerms matching lifecycle=spot label -> Option A

4. Quick Check:

   Spot instance selection uses nodeAffinity with lifecycle=spot label [OK]

Hint: Use nodeAffinity with lifecycle=spot label for spot nodes [OK]

Common Mistakes:

• Using nodeSelector with wrong label key
• Setting resource requests to 'spot' (invalid)
• Confusing tolerations with node affinity

3. Given this autoscaling configuration snippet for a Kubernetes deployment:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: ml-model-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: ml-model
  minReplicas: 2
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 50

What happens when CPU usage rises to 75%?

medium

A. The number of pods will increase up to a maximum of 10

B. The number of pods will decrease to 2

C. The deployment will restart pods

D. Nothing changes because CPU target is 50%

Solution

1. Step 1: Understand Horizontal Pod Autoscaler (HPA) behavior

   HPA increases pods when CPU usage exceeds target utilization to balance load.

2. Step 2: Analyze CPU usage vs target

   CPU is at 75%, above the 50% target, so HPA will scale up pods up to maxReplicas (10).

3. Final Answer:

   The number of pods will increase up to a maximum of 10 -> Option A

4. Quick Check:

   CPU > target utilization triggers pod scaling up [OK]

Hint: CPU above target utilization triggers scaling up [OK]

Common Mistakes:

• Thinking pods scale down when CPU rises
• Confusing pod restart with scaling
• Assuming no change if CPU exceeds target

4. You have a cloud cost alert system but it keeps sending false alarms about overspending. What is the most likely cause?

medium

A. The cloud provider is charging incorrectly

B. The alert thresholds are set too low or too sensitive

C. The system is not connected to the billing API

D. The cost data is updated only once a year

Solution

1. Step 1: Understand alert system sensitivity

   Alerts trigger when costs exceed set thresholds; too low thresholds cause false alarms.

2. Step 2: Evaluate other options

   Incorrect charges or missing billing data cause different issues, not false alarms.

3. Final Answer:

   The alert thresholds are set too low or too sensitive -> Option B

4. Quick Check:

   Low alert thresholds cause false alarms [OK]

Hint: Check alert thresholds if false alarms occur [OK]

Common Mistakes:

• Blaming cloud provider without proof
• Ignoring alert configuration
• Assuming billing API is always connected

5. You want to reduce costs for a large ML training job that runs daily on cloud GPUs. Which combined strategy best optimizes cost at scale?

hard

A. Run training on CPUs to avoid GPU costs without changing code

B. Use only on-demand GPU instances and disable autoscaling

C. Use spot GPU instances with checkpointing and autoscaling to handle interruptions

D. Schedule training during peak hours to use full capacity

Solution

1. Step 1: Identify cost-saving options for GPU jobs

   Spot instances are cheaper but can be interrupted; checkpointing saves progress.

2. Step 2: Combine autoscaling with spot instances and checkpointing

   Autoscaling adjusts resources; checkpointing prevents data loss on interruptions.

3. Step 3: Evaluate other options

   On-demand is costly; CPUs are slower; peak hours usually cost more.

4. Final Answer:

   Use spot GPU instances with checkpointing and autoscaling to handle interruptions -> Option C

5. Quick Check:

   Spot + checkpoint + autoscale = best cost optimization [OK]

Hint: Combine spot instances with checkpointing and autoscaling [OK]

Common Mistakes:

• Ignoring interruptions on spot instances
• Using expensive on-demand only
• Running on CPUs without code changes
• Scheduling during costly peak hours