REST API inference in PyTorch - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

What is the output of this Flask REST API inference code?

Consider this simple Flask app that loads a PyTorch model and returns predictions for input data sent as JSON. What will be the JSON response when sending {"input": [1.0, 2.0, 3.0]}?

PyTorch

from flask import Flask, request, jsonify
import torch
import torch.nn as nn

app = Flask(__name__)

class SimpleModel(nn.Module):
    def __init__(self):
        super().__init__()
        self.linear = nn.Linear(3, 1)
    def forward(self, x):
        return self.linear(x)

model = SimpleModel()
model.linear.weight.data.fill_(1.0)
model.linear.bias.data.fill_(0.0)

@app.route('/predict', methods=['POST'])
def predict():
    data = request.get_json()
    x = torch.tensor(data['input'], dtype=torch.float32)
    with torch.no_grad():
        output = model(x).item()
    return jsonify({'prediction': output})

A {"prediction": 1.0}

B {"prediction": 3.0}

C {"prediction": 0.0}

D {"prediction": 6.0}

💡 Hint

Remember the linear layer sums weighted inputs plus bias. Weights are all 1, bias is 0.

The linear layer has weights all set to 1 and bias 0. Input vector is [1.0, 2.0, 3.0]. The output is sum(1*1 + 1*2 + 1*3) + 0 = 6.0.

❓ Model Choice

intermediate

Which PyTorch model is best suited for REST API inference on image classification?

You want to deploy a REST API that receives images and returns predicted labels. Which model architecture below is most appropriate for this task?

A A linear regression model

B A recurrent neural network (RNN) like LSTM

C A convolutional neural network (CNN) like ResNet

D A k-means clustering model

💡 Hint

Think about which model type handles image data well.

CNNs like ResNet are designed to process images and extract spatial features, making them ideal for image classification tasks in REST APIs.

❓ Hyperparameter

advanced

Which batch size is best for fast REST API inference with PyTorch?

You deploy a REST API for inference. To optimize latency and throughput, which batch size should you choose?

A Batch size of 1 for lowest latency per request

B Batch size of 128 to maximize GPU utilization

C Batch size of 0 to disable batching

D Batch size of 1000 to process many requests at once

💡 Hint

Consider that REST API requests usually come one at a time and latency matters.

Batch size 1 processes each request immediately, minimizing latency. Large batch sizes increase throughput but add delay waiting for batch to fill.

🔧 Debug

advanced

Why does this PyTorch REST API inference code raise a RuntimeError?

Given this code snippet, why does the server raise a RuntimeError: "Trying to backward through the graph during inference"?

PyTorch

from flask import Flask, request, jsonify
import torch
import torch.nn as nn

app = Flask(__name__)

model = nn.Linear(2, 1)

@app.route('/predict', methods=['POST'])
def predict():
    data = request.get_json()
    x = torch.tensor(data['input'], dtype=torch.float32)
    with torch.no_grad():
        output = model(x)
    return jsonify({'prediction': output.item()})

A Flask cannot handle POST requests without JSON schema validation

B The code is missing torch.no_grad() context, so autograd tries to track operations

C The input tensor x has wrong dtype causing runtime error

D The model is not in eval() mode causing training behavior

💡 Hint

Inference should not track gradients to save memory and avoid errors.

Without torch.no_grad(), PyTorch tracks operations for gradients. During inference, this can cause errors if backward is called or memory issues.

❓ Metrics

expert

Which metric is best to monitor REST API inference quality for a multi-class classification model?

You deploy a REST API serving a multi-class classifier. Which metric below best measures the model's prediction quality on live data?

A Accuracy - percentage of correct predictions

B Mean Squared Error (MSE)

C Silhouette Score

D BLEU Score

💡 Hint

Think about classification metrics that reflect correct label predictions.

Accuracy directly measures the proportion of correct predictions in classification tasks, making it suitable for monitoring REST API inference quality.

Practice

1. What is the main purpose of using a REST API for PyTorch model inference?

easy

A. To train the model faster using multiple GPUs

B. To send data to the model and receive predictions over the web

C. To visualize the model architecture in a browser

D. To save the model weights to disk

Solution

1. Step 1: Understand REST API role

   A REST API allows communication over the web by sending requests and receiving responses.

2. Step 2: Connect REST API with model inference

   In PyTorch, a REST API is used to send input data and get predictions from the model remotely.

3. Final Answer:

   To send data to the model and receive predictions over the web -> Option B

4. Quick Check:

   REST API = Web data exchange for predictions [OK]

Hint: REST API = send input, get prediction over web [OK]

Common Mistakes:

• Confusing inference with training
• Thinking REST API is for saving models
• Assuming REST API visualizes models

2. Which of the following is the correct way to disable gradient calculation during inference in a Flask REST API using PyTorch?

easy

A. with torch.no_grad():

B. model.train()

C. torch.enable_grad()

D. model.backward()

Solution

1. Step 1: Identify how to disable gradients

   PyTorch uses torch.no_grad() context to turn off gradient tracking during inference.

2. Step 2: Check other options

   model.train() enables training mode, torch.enable_grad() enables gradients, and model.backward() is invalid.

3. Final Answer:

   with torch.no_grad(): -> Option A

4. Quick Check:

   Disable gradients = torch.no_grad() [OK]

Hint: Use torch.no_grad() to save memory during inference [OK]

Common Mistakes:

• Using model.train() instead of model.eval()
• Not disabling gradients causing slow inference
• Calling backward() during inference

3. Given this Flask route for inference, what will be the output if the input JSON is {'input': [1.0, 2.0]} and the model returns tensor([3.0])?

from flask import Flask, request, jsonify
import torch
app = Flask(__name__)
model = torch.nn.Linear(2, 1)
model.eval()

@app.route('/predict', methods=['POST'])
def predict():
    data = request.get_json()
    x = torch.tensor(data['input']).unsqueeze(0).float()
    with torch.no_grad():
        output = model(x)
    return jsonify({'prediction': output.item()})

medium

A. {"prediction": "tensor([3.0])"}

B. {"prediction": [3.0]}

C. Error: model not loaded

D. {"prediction": 3.0}

Solution

1. Step 1: Understand input and output processing

   The input JSON is converted to a tensor with shape (1, 2). The model outputs a tensor with shape (1, 1).

2. Step 2: Check output conversion

   The output tensor's single value is extracted using output.item(), which returns a float, then wrapped in JSON.

3. Final Answer:

   {"prediction": 3.0} -> Option D

4. Quick Check:

   output.item() returns float for JSON [OK]

Hint: Use output.item() to get scalar from tensor for JSON [OK]

Common Mistakes:

• Returning tensor object directly in JSON
• Forgetting to convert input to tensor
• Not using torch.no_grad() causing slow inference

4. You wrote this Flask REST API code for inference but get an error: 'RuntimeError: expected scalar type Float but found Double'. What is the likely fix?

@app.route('/predict', methods=['POST'])
def predict():
    data = request.get_json()
    x = torch.tensor(data['input'])
    with torch.no_grad():
        output = model(x)
    return jsonify({'prediction': output.item()})

medium

A. Convert input tensor to float using .float() before passing to model

B. Call model.train() before inference

C. Remove torch.no_grad() context

D. Change model output to int

Solution

1. Step 1: Understand the error cause

   The model expects float tensors but input tensor is double (default from Python floats).

2. Step 2: Fix input tensor type

   Convert input tensor to float type using .float() before passing to the model.

3. Final Answer:

   Convert input tensor to float using .float() before passing to model -> Option A

4. Quick Check:

   Tensor dtype mismatch fixed by .float() [OK]

Hint: Always convert input to float tensor before model call [OK]

Common Mistakes:

• Not converting input data type
• Calling model.train() during inference
• Removing torch.no_grad() causing slow inference

5. You want to deploy a PyTorch model as a REST API that accepts images, preprocesses them, and returns class predictions. Which sequence of steps is best practice inside your Flask route?

hard

A. Load model globally, call model.eval(), preprocess input image to numpy array, call model backward(), return JSON

B. Load model inside route on every request, call model.train(), preprocess input, get prediction, return JSON

C. Load model once globally, set model.eval(), preprocess input image to tensor, use torch.no_grad(), get prediction, return JSON

D. Preprocess input image after prediction, load model globally, call torch.enable_grad(), return raw tensor

Solution

1. Step 1: Model loading and mode

   Load the model once globally to avoid overhead and set it to evaluation mode with model.eval().

2. Step 2: Input preprocessing and inference

   Preprocess the input image to a tensor, use torch.no_grad() to disable gradients, get prediction, and return JSON response.

3. Final Answer:

   Load model once globally, set model.eval(), preprocess input image to tensor, use torch.no_grad(), get prediction, return JSON -> Option C

4. Quick Check:

   Efficient inference = global load + eval + no_grad + preprocess [OK]

Hint: Load model once, eval mode, no_grad, preprocess input tensor [OK]

Common Mistakes:

• Loading model on every request causing slow response
• Using model.train() during inference
• Not preprocessing input to tensor
• Calling backward() during inference