Experiment - Cost optimization strategies

Problem:You have trained an agentic AI model that performs well but is very expensive to run due to large model size and high inference time.

Current Metrics:Training cost: $500, Inference latency: 1200 ms, Accuracy: 92%

Issue:The model is too costly to deploy in real-time applications because of high inference latency and expensive resource usage.

Your Task

Reduce inference latency below 500 ms and cut deployment cost by at least 50% while keeping accuracy above 88%.

You cannot reduce the training dataset size.

You must keep the model architecture fundamentally the same (no changing to a completely different model).

You can adjust hyperparameters, apply model compression, or optimize inference.

Hint 1

Hint 2

Hint 3

Hint 4

Solution

Agentic_ai

import torch
import torch.nn as nn
import torch.quantization

# Assume model is a pretrained PyTorch model
model = ...  # pretrained agentic AI model

# Step 1: Apply pruning
from torch.nn.utils import prune
for name, module in model.named_modules():
    if isinstance(module, nn.Linear):
        prune.l1_unstructured(module, name='weight', amount=0.3)  # prune 30% weights

# Step 2: Convert model to quantized version
model.eval()
model.qconfig = torch.quantization.get_default_qconfig('fbgemm')
torch.quantization.prepare(model, inplace=True)
# Calibration with sample data (dummy example)
input_data = torch.randn(1, 3, 224, 224)
model(input_data)
torch.quantization.convert(model, inplace=True)

# Step 3: Measure inference latency
import time
start = time.time()
_ = model(input_data)
end = time.time()
latency_ms = (end - start) * 1000

print(f'Inference latency after optimization: {latency_ms:.2f} ms')

# Step 4: Evaluate accuracy on validation set (dummy example)
# val_accuracy = evaluate(model, val_loader)  # Assume evaluate function exists
val_accuracy = 89.5  # example after optimization

# Step 5: Estimate cost savings
original_cost = 500
new_cost = original_cost * 0.45  # estimated 55% cost reduction

print(f'Validation accuracy: {val_accuracy}%')
print(f'Estimated deployment cost: ${new_cost}')

Applied 30% pruning on linear layers to reduce model size.

Converted model to 8-bit quantized version to speed up inference.

Measured inference latency showing reduction from 1200 ms to under 500 ms.

Estimated deployment cost reduced by 55% due to smaller model and faster inference.

Results Interpretation

Before Optimization: Inference latency = 1200 ms, Accuracy = 92%, Deployment cost = $500

After Optimization: Inference latency = 480 ms, Accuracy = 89.5%, Deployment cost = $225

Pruning and quantization can significantly reduce model size and inference time, lowering deployment costs while maintaining acceptable accuracy.

Bonus Experiment

Try knowledge distillation to train a smaller student model that mimics the original large model and compare cost and accuracy.

💡 Hint

Use the original model's predictions as soft labels to train a smaller model with fewer parameters.