Latency optimization in Prompt Engineering / GenAI - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

Understanding Latency in Model Inference

Which factor most directly affects the latency of a machine learning model during inference?

A The number of layers and parameters in the model

B The size of the training dataset

C The type of optimizer used during training

D The number of epochs used during training

💡 Hint

Think about what happens when the model makes a prediction.

Latency during inference depends mainly on how complex the model is, which is related to its layers and parameters. Larger models take longer to compute predictions.

❓ Predict Output

intermediate

Effect of Batch Size on Latency

What is the output of the following code that simulates latency for different batch sizes?

Prompt Engineering / GenAI

import time

def simulate_latency(batch_size):
    base_time = 0.01  # seconds per sample
    total_time = base_time * batch_size
    time.sleep(total_time)
    return total_time

latencies = {b: simulate_latency(b) for b in [1, 5, 10]}
print(latencies)

A {1: 0.01, 5: 0.05, 10: 0.1}

B {1: 0.01, 5: 0.01, 10: 0.01}

C {1: 0.1, 5: 0.5, 10: 1.0}

D {1: 0.01, 5: 0.1, 10: 0.5}

💡 Hint

Latency scales linearly with batch size in this simulation.

The function multiplies the base time per sample by the batch size, so latency increases linearly.

❓ Model Choice

advanced

Choosing a Model for Low Latency

You need to deploy a model on a device with limited processing power and require very low latency. Which model architecture is best suited?

A A deep convolutional neural network with 50 layers

B A small decision tree model

C A large transformer model with billions of parameters

D A recurrent neural network with multiple LSTM layers

💡 Hint

Think about model size and computation needed for fast predictions.

Decision trees are lightweight and fast to evaluate, making them suitable for low latency on limited hardware.

❓ Hyperparameter

advanced

Hyperparameter Impact on Latency

Which hyperparameter adjustment is most likely to reduce inference latency without retraining the model?

A Reducing the number of training epochs

B Increasing the learning rate

C Lowering the batch size during inference

D Changing the activation function to ReLU

💡 Hint

Consider what happens when you process fewer samples at once.

Lowering batch size reduces the amount of data processed at once, which can reduce latency per batch during inference.

🔧 Debug

expert

Debugging Latency Bottleneck in Code

Given the code below, which line is the main cause of increased latency during inference?

def predict(model, data):
    results = []
    for sample in data:
        processed = preprocess(sample)
        output = model(processed)
        results.append(output)
    return results

# preprocess is slow due to heavy image resizing
# model is optimized and fast

A Line 3: for sample in data:

B Line 6: results.append(output)

C Line 5: output = model(processed)

D Line 4: processed = preprocess(sample)

💡 Hint

Focus on which step is described as slow.

The preprocessing step is slow due to heavy image resizing, causing the latency bottleneck.

Practice

1. What is the main goal of latency optimization in AI models?

easy

A. To make AI models respond faster for better user experience

B. To increase the size of the AI model

C. To reduce the accuracy of the AI model

D. To add more layers to the AI model

Solution

1. Step 1: Understand latency meaning

   Latency means the time it takes for a model to give an answer after input.

2. Step 2: Connect latency to user experience

   Lower latency means faster responses, which improves how users feel about the AI.

3. Final Answer:

   To make AI models respond faster for better user experience -> Option A

4. Quick Check:

   Latency optimization = faster response [OK]

Hint: Latency means speed of response, optimize to make it faster [OK]

Common Mistakes:

• Confusing latency with model size
• Thinking latency means accuracy
• Assuming more layers reduce latency

2. Which of the following is a correct Python syntax to measure latency using time module?

easy

A. start = time.sleep(); model.predict(x); end = time.sleep(); latency = end / start

B. start = time.time(); model.predict(x); end = time.time(); latency = end - start

C. start = time.clock(); model.predict(x); end = time.clock(); latency = start - end

D. start = time.now(); model.predict(x); end = time.now(); latency = end + start

Solution

1. Step 1: Identify correct time functions

   Python's time.time() returns current time in seconds; subtracting gives elapsed time.

2. Step 2: Check latency calculation

   Latency = end - start measures duration correctly; other options misuse functions or operations.

3. Final Answer:

   start = time.time(); model.predict(x); end = time.time(); latency = end - start -> Option B

4. Quick Check:

   Latency = end - start time [OK]

Hint: Use time.time() and subtract end-start for latency [OK]

Common Mistakes:

• Using time.now() which does not exist
• Subtracting start - end instead of end - start
• Using time.sleep() which pauses code, not measures time

3. Given this code snippet measuring latency, what will be printed?

import time
start = time.time()
for _ in range(1000000):
    pass
end = time.time()
print(round(end - start, 2))

medium

A. A number close to 1.00

B. An error because of wrong syntax

C. A number close to 10.00

D. A number close to 0.00

Solution

1. Step 1: Understand the loop workload

   The loop runs 1,000,000 times doing nothing (pass), which takes very little time due to Python's loop execution speed.

2. Step 2: Estimate time taken

   On a normal computer, this empty loop takes around 0.03-0.1 seconds, so round(end - start, 2) prints a number close to 0.00.

3. Final Answer:

   A number close to 0.00 -> Option D

4. Quick Check:

   1M empty loops ~0.05s [OK]

Hint: 1 million empty loops take ~0.05 seconds [OK]

Common Mistakes:

• Overestimating time for empty loop (e.g., thinking 1 second)
• Thinking it takes 10 seconds
• Assuming syntax error due to indentation

4. You tried pruning your AI model to reduce latency but latency increased. What is the likely cause?

medium

A. Pruning removed important layers causing slower computation

B. Pruning always increases latency by design

C. Pruning was done incorrectly causing overhead in model execution

D. Latency measurement was done before pruning

Solution

1. Step 1: Understand pruning effect

   Pruning removes less important parts to speed up model, so latency should decrease if done right.

2. Step 2: Identify why latency increased

   If latency increased, pruning likely added overhead or was done incorrectly, causing slower execution.

3. Final Answer:

   Pruning was done incorrectly causing overhead in model execution -> Option C

4. Quick Check:

   Incorrect pruning = more overhead = higher latency [OK]

Hint: Incorrect pruning adds overhead, increasing latency [OK]

Common Mistakes:

• Assuming pruning always slows model
• Ignoring measurement timing
• Thinking pruning removes important layers by default

5. You want to reduce latency of a large AI model for mobile devices. Which combined approach is best?

hard

A. Use quantization to reduce precision and prune unimportant weights

B. Increase model layers and use caching on server

C. Only use caching without changing model size

D. Train a bigger model with more data

Solution

1. Step 1: Identify techniques for latency reduction on mobile

   Quantization reduces number size to speed up computation; pruning removes unneeded parts to shrink model.

2. Step 2: Evaluate options

   Increasing layers or bigger models increase latency; caching helps but alone is not enough for mobile constraints.

3. Final Answer:

   Use quantization to reduce precision and prune unimportant weights -> Option A

4. Quick Check:

   Quantization + pruning = best latency reduction [OK]

Hint: Combine quantization and pruning for mobile latency [OK]

Common Mistakes:

• Thinking bigger models reduce latency
• Relying only on caching for mobile speed
• Ignoring model size impact on mobile devices