TorchServe setup in PyTorch

Introduction

TorchServe helps you easily turn your PyTorch models into web services. It makes sharing and using models simple without extra coding.

You want to share your trained PyTorch model with others via an API.

You need to deploy a model to a server for real-time predictions.

You want to manage multiple models and versions in one place.

You want to monitor model performance and usage easily.

You want to avoid writing custom server code for your model.

Syntax

PyTorch

torch-model-archiver --model-name <model_name> --version <version> --serialized-file <model_file> --handler <handler_file> --extra-files <extra_files>
torchserve --start --model-store <model_store_dir> --models <model_name>=<model_name>.mar

The torch-model-archiver command packages your model into a .mar file for TorchServe.

The torchserve command starts the server and loads your packaged model.

Examples

This example packages a ResNet18 model and starts TorchServe with it.

PyTorch

torch-model-archiver --model-name resnet18 --version 1.0 --serialized-file resnet18.pth --handler image_classifier --extra-files index_to_name.json
torchserve --start --model-store model_store --models resnet18=resnet18.mar

Here, a custom handler script is used to package and serve the model.

PyTorch

torch-model-archiver --model-name mymodel --version 2.1 --serialized-file model.pt --handler custom_handler.py
torchserve --start --model-store model_store --models mymodel=mymodel.mar

Sample Model

This code saves a pretrained ResNet18 model, shows how to package and start TorchServe (commands as comments), and runs a local prediction to demonstrate expected output.

PyTorch

import torch
from torchvision import models, transforms
from PIL import Image
import requests
import json

# Step 1: Download and save a pretrained model
model = models.resnet18(pretrained=True)
model.eval()

# Save the model
torch.save(model.state_dict(), 'resnet18.pth')

# Step 2: Create model archive (run in terminal, shown here as comment)
# torch-model-archiver --model-name resnet18 --version 1.0 --serialized-file resnet18.pth --handler image_classifier --extra-files index_to_name.json

# Step 3: Start TorchServe (run in terminal, shown here as comment)
# torchserve --start --model-store model_store --models resnet18=resnet18.mar

# Step 4: Send a test request to the server
# (Make sure TorchServe is running and model is loaded)

# Download a sample image
url = 'https://pytorch.org/assets/images/dog.jpg'
image = Image.open(requests.get(url, stream=True).raw)

# Prepare image for model
preprocess = transforms.Compose([
    transforms.Resize(256),
    transforms.CenterCrop(224),
    transforms.ToTensor(),
    transforms.Normalize(mean=[0.485, 0.456, 0.406],
                         std=[0.229, 0.224, 0.225])
])
input_tensor = preprocess(image)
input_batch = input_tensor.unsqueeze(0)  # create mini-batch

# Load class index to name mapping
with open('index_to_name.json') as f:
    idx_to_label = json.load(f)

# Run prediction locally (for demo, since server call needs network)
with torch.no_grad():
    output = model(input_batch)
probabilities = torch.nn.functional.softmax(output, dim=1)[0]

# Get top 1 prediction
top_prob, top_catid = torch.topk(probabilities, 1)

print(f"Predicted class: {idx_to_label[str(top_catid.item())]} with probability {top_prob.item():.4f}")

Important Notes

Make sure to install TorchServe and torch-model-archiver via pip before starting.

The handler defines how input data is processed and output is returned; use built-in handlers or write your own.

Model archive files (.mar) are stored in the model store directory for TorchServe to load.

Summary

TorchServe packages PyTorch models into .mar files for easy deployment.

Use torch-model-archiver to create the model archive and torchserve to start the server.

Once running, you can send data to the server to get predictions from your model.

Practice

1. What is the main purpose of TorchServe in PyTorch model deployment?

easy

A. To convert PyTorch models into TensorFlow format

B. To train PyTorch models faster on GPUs

C. To visualize PyTorch model architectures

D. To package and serve PyTorch models for easy deployment

Solution

1. Step 1: Understand TorchServe's role

   TorchServe is designed to help deploy PyTorch models by packaging them and serving predictions.

2. Step 2: Compare options

   Options B, C, and D describe unrelated tasks like training speed, visualization, or conversion, which TorchServe does not do.

3. Final Answer:

   To package and serve PyTorch models for easy deployment -> Option D

4. Quick Check:

   TorchServe = model packaging and serving [OK]

Hint: TorchServe is for deployment, not training or conversion [OK]

Common Mistakes:

• Confusing TorchServe with training tools
• Thinking TorchServe converts model formats
• Assuming TorchServe visualizes models

2. Which command is used to create a model archive (.mar) file for TorchServe?

easy

A. torchserve --start

B. torch-model-archiver

C. python setup.py build

D. torch.save(model, 'model.pth')

Solution

1. Step 1: Identify the command for packaging models

   The torch-model-archiver command creates the .mar file needed by TorchServe.

2. Step 2: Eliminate other commands

   torchserve --start starts the server, python setup.py build is unrelated, and torch.save saves a model but not as a .mar archive.

3. Final Answer:

   torch-model-archiver -> Option B

4. Quick Check:

   Model archive = torch-model-archiver [OK]

Hint: Use torch-model-archiver to package models [OK]

Common Mistakes:

• Using torchserve to create archives
• Confusing torch.save with archiving
• Running unrelated Python build commands

3. Given the following commands, what will be the output after starting TorchServe?

!torch-model-archiver --model-name mymodel --version 1.0 --serialized-file model.pth --handler handler.py --export-path model_store
!torchserve --start --model-store model_store --models mymodel=mymodel-1.0.mar

What happens when you send a prediction request to the server?

medium

A. The server returns predictions from the loaded model

B. The server throws an error because model_store is missing

C. The server starts but does not load any model

D. The server converts the model to ONNX format

Solution

1. Step 1: Understand the archiving and serving commands

   The first command creates a .mar file in model_store. The second starts TorchServe loading that model.

2. Step 2: Predict server behavior on request

   Since the model is loaded, the server will respond with predictions when given input data.

3. Final Answer:

   The server returns predictions from the loaded model -> Option A

4. Quick Check:

   Model archived and loaded = predictions served [OK]

Hint: Archive then start server to serve predictions [OK]

Common Mistakes:

• Assuming model_store folder is missing
• Thinking server starts without loading models
• Believing TorchServe converts model formats

4. You run torchserve --start --model-store model_store --models mymodel=mymodel.mar but get an error: ModelNotFoundError. What is the most likely cause?

medium

A. The handler.py file is missing

B. TorchServe is not installed

C. The .mar file is not in the model_store directory

D. The model name is misspelled in the command

Solution

1. Step 1: Analyze the error message

   ModelNotFoundError means TorchServe cannot find the specified .mar file in the model_store directory.

2. Step 2: Check command and files

   If the .mar file is missing or not placed in model_store, the server cannot load it. Misspelling or missing handler causes different errors.

3. Final Answer:

   The .mar file is not in the model_store directory -> Option C

4. Quick Check:

   ModelNotFoundError = missing .mar file [OK]

Hint: Ensure .mar file is inside model_store folder [OK]

Common Mistakes:

• Ignoring file location for .mar archive
• Assuming TorchServe installation issue
• Confusing model name typo with missing file

5. You want to deploy two different PyTorch models simultaneously using TorchServe. Which setup correctly achieves this?

hard

A. Create two .mar files and start TorchServe with --models model1=model1.mar model2=model2.mar

B. Combine both models into one .mar file and start TorchServe with a single model

C. Start two TorchServe servers on the same port, each serving one model

D. Use torch-model-archiver to merge models into a single .pth file

Solution

1. Step 1: Understand multi-model serving in TorchServe

   TorchServe supports serving multiple models by specifying multiple .mar files with aliases in the --models option.

2. Step 2: Evaluate options

   Create two .mar files and start TorchServe with --models model1=model1.mar model2=model2.mar correctly uses two .mar files with aliases. Combine both models into one .mar file and start TorchServe with a single model is invalid because .mar files cannot combine models. Start two TorchServe servers on the same port, each serving one model fails because two servers cannot share the same port. Use torch-model-archiver to merge models into a single .pth file is incorrect; torch-model-archiver does not merge models.

3. Final Answer:

   Create two .mar files and start TorchServe with --models model1=model1.mar model2=model2.mar -> Option A

4. Quick Check:

   Multiple models = multiple .mar files with aliases [OK]

Hint: Use space-separated model aliases in --models option [OK]

Common Mistakes:

• Trying to merge models into one .mar file
• Running multiple servers on same port
• Misusing torch-model-archiver to combine models