ONNX Runtime inference in PyTorch - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

ONNX Runtime: Model output shape

Given the following PyTorch model exported to ONNX and loaded with ONNX Runtime, what is the shape of the output tensor after inference?

PyTorch

import torch
import onnxruntime as ort
import numpy as np

class SimpleModel(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.linear = torch.nn.Linear(4, 3)
    def forward(self, x):
        return self.linear(x)

model = SimpleModel()
# Export to ONNX
dummy_input = torch.randn(1, 4)
torch.onnx.export(model, dummy_input, 'simple.onnx', input_names=['input'], output_names=['output'])

# Load with ONNX Runtime
session = ort.InferenceSession('simple.onnx')
input_name = session.get_inputs()[0].name
input_data = np.random.randn(1, 4).astype(np.float32)
outputs = session.run(None, {input_name: input_data})
output = outputs[0]
output.shape

A (3, 1)

B (4, 3)

C (1, 3)

D (1, 4)

💡 Hint

Remember the linear layer transforms input features from 4 to 3, batch size is 1.

The model input has shape (1, 4). The linear layer outputs (1, 3) because it maps 4 features to 3 outputs for each batch item.

❓ Model Choice

intermediate

Choosing ONNX Runtime for inference

Which of the following is the main advantage of using ONNX Runtime for model inference compared to running PyTorch models directly?

A ONNX Runtime supports running models on multiple hardware platforms efficiently.

B ONNX Runtime only supports CPU inference, not GPU.

C ONNX Runtime requires rewriting the model in a new language.

D ONNX Runtime cannot optimize model execution speed.

💡 Hint

Think about hardware compatibility and performance.

ONNX Runtime is designed to run models efficiently on various hardware like CPU, GPU, and specialized accelerators, improving portability and speed.

❓ Hyperparameter

advanced

Batch size effect on ONNX Runtime inference

If you increase the batch size of input data when running inference with ONNX Runtime, which of the following is true?

A ONNX Runtime automatically splits batches into size 1 internally.

B Inference latency per batch decreases and throughput decreases.

C Batch size has no effect on inference speed or throughput.

D Inference latency per batch increases but throughput (samples per second) may improve.

💡 Hint

Think about how processing more samples at once affects speed.

Larger batch sizes mean more data processed at once, increasing latency per batch but often improving overall throughput due to parallelism.

🔧 Debug

advanced

ONNX Runtime inference error diagnosis

You run ONNX Runtime inference with input data shape (1, 5) but the model expects input shape (1, 4). What error will ONNX Runtime most likely raise?

A onnxruntime.capi.onnxruntime_pybind11_state.InvalidArgument: Unexpected input shape

B ZeroDivisionError: division by zero

C FileNotFoundError: model file not found

D TypeError: unsupported operand type(s) for +: 'int' and 'str'

💡 Hint

Consider what happens if input shape does not match model expectation.

ONNX Runtime checks input shapes and raises an InvalidArgument error if the input shape does not match the model's expected shape.

❓ Metrics

expert

Evaluating ONNX Runtime inference speed

You measure ONNX Runtime inference time for 1000 samples with batch size 10 and get 2 seconds total. What is the approximate throughput in samples per second?

A 5000 samples/second

B 500 samples/second

C 50 samples/second

D 5 samples/second

💡 Hint

Throughput = total samples / total time.

Throughput = 1000 samples / 2 seconds = 500 samples per second.