Latency optimization in Prompt Engineering / GenAI - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

Understanding Latency in Model Inference

Which factor most directly affects the latency of a machine learning model during inference?

A The number of layers and parameters in the model

B The size of the training dataset

C The type of optimizer used during training

D The number of epochs used during training

💡 Hint

Think about what happens when the model makes a prediction.

Latency during inference depends mainly on how complex the model is, which is related to its layers and parameters. Larger models take longer to compute predictions.

❓ Predict Output

intermediate

Effect of Batch Size on Latency

What is the output of the following code that simulates latency for different batch sizes?

Prompt Engineering / GenAI

import time

def simulate_latency(batch_size):
    base_time = 0.01  # seconds per sample
    total_time = base_time * batch_size
    time.sleep(total_time)
    return total_time

latencies = {b: simulate_latency(b) for b in [1, 5, 10]}
print(latencies)

A {1: 0.01, 5: 0.05, 10: 0.1}

B {1: 0.01, 5: 0.01, 10: 0.01}

C {1: 0.1, 5: 0.5, 10: 1.0}

D {1: 0.01, 5: 0.1, 10: 0.5}

💡 Hint

Latency scales linearly with batch size in this simulation.

The function multiplies the base time per sample by the batch size, so latency increases linearly.

❓ Model Choice

advanced

Choosing a Model for Low Latency

You need to deploy a model on a device with limited processing power and require very low latency. Which model architecture is best suited?

A A deep convolutional neural network with 50 layers

B A small decision tree model

C A large transformer model with billions of parameters

D A recurrent neural network with multiple LSTM layers

💡 Hint

Think about model size and computation needed for fast predictions.

Decision trees are lightweight and fast to evaluate, making them suitable for low latency on limited hardware.

❓ Hyperparameter

advanced

Hyperparameter Impact on Latency

Which hyperparameter adjustment is most likely to reduce inference latency without retraining the model?

A Reducing the number of training epochs

B Increasing the learning rate

C Lowering the batch size during inference

D Changing the activation function to ReLU

💡 Hint

Consider what happens when you process fewer samples at once.

Lowering batch size reduces the amount of data processed at once, which can reduce latency per batch during inference.

🔧 Debug

expert

Debugging Latency Bottleneck in Code

Given the code below, which line is the main cause of increased latency during inference?

def predict(model, data):
    results = []
    for sample in data:
        processed = preprocess(sample)
        output = model(processed)
        results.append(output)
    return results

# preprocess is slow due to heavy image resizing
# model is optimized and fast

A Line 3: for sample in data:

B Line 6: results.append(output)

C Line 5: output = model(processed)

D Line 4: processed = preprocess(sample)

💡 Hint

Focus on which step is described as slow.

The preprocessing step is slow due to heavy image resizing, causing the latency bottleneck.