Experiment - TorchServe setup

Problem:You want to deploy a PyTorch model so it can serve predictions through a web API using TorchServe.

Current Metrics:No deployment yet, so no serving or prediction metrics available.

Issue:Model is trained but not yet deployed for real-time inference. You need to package and serve it properly.

Your Task

Set up TorchServe to deploy a PyTorch model and test that it returns correct predictions via API.

Use a simple pretrained model (e.g., a small CNN or a pretrained ResNet).

Create a model archive (.mar) file using TorchServe tools.

Serve the model locally using TorchServe.

Test the API with sample input and get prediction output.

Hint 1

Hint 2

Hint 3

Hint 4

Solution

PyTorch

import torch
import torchvision.models as models
import torchvision.transforms as transforms
from PIL import Image
import requests
import json
import os

# Step 1: Save a pretrained model
model = models.resnet18(pretrained=True)
model.eval()

model_path = 'resnet18.pth'
torch.save(model.state_dict(), model_path)

# Step 2: Create a model archive using torch-model-archiver (run in shell):
# torch-model-archiver --model-name resnet18 --version 1.0 --serialized-file resnet18.pth --handler image_classifier --export-path model_store --extra-files index_to_name.json

# Note: You need to provide index_to_name.json mapping file for the image_classifier handler.

# Step 3: Start TorchServe (run in shell):
# torchserve --start --model-store model_store --models resnet18=resnet18.mar

# Step 4: Prepare an image for prediction
image_url = 'https://pytorch.org/assets/images/deeplab1.png'
image_path = 'test_image.png'

if not os.path.exists(image_path):
    img_data = requests.get(image_url).content
    with open(image_path, 'wb') as f:
        f.write(img_data)

# Step 5: Send image to TorchServe API
with open(image_path, 'rb') as f:
    image_bytes = f.read()

response = requests.post('http://127.0.0.1:8080/predictions/resnet18', data=image_bytes)

print('Prediction response:', response.text)

# Step 6: Stop TorchServe (run in shell):
# torchserve --stop

Saved pretrained ResNet18 model state_dict instead of TorchScript for compatibility with default image_classifier handler.

Added note about providing index_to_name.json mapping file required by image_classifier handler.

Kept other steps unchanged.

Results Interpretation

Before: Model trained but no deployment or serving.

After: Model deployed with TorchServe, serving predictions via REST API successfully.

TorchServe simplifies deploying PyTorch models for real-time inference by packaging models and providing a REST API for predictions.

Bonus Experiment

Try creating a custom handler to preprocess input images differently or postprocess outputs before returning predictions.

💡 Hint

Write a Python handler class inheriting from BaseHandler and specify it during model archiving.