Batch vs real-time inference in NLP - Practice Questions

Challenge - 5 Problems

🧠 Conceptual

intermediate

Understanding batch inference

Which of the following best describes batch inference in machine learning?

A Training the model using multiple data points in one go.

B Generating predictions instantly for each data point as it arrives.

C Processing many data points together at once, usually offline, to generate predictions.

D Updating the model weights continuously during prediction.

💡 Hint

Think about when predictions are made: all at once or one by one?

Batch inference means processing many inputs together, often offline, to produce predictions in bulk. Real-time inference is instant per input.

🧠 Conceptual

intermediate

Real-time inference use case

Which scenario is best suited for real-time inference?

A Recommending a product immediately after a user clicks on a website.

B Archiving old prediction results for future analysis.

C Training a language model on millions of sentences.

D Analyzing a large dataset overnight to update customer segments.

💡 Hint

Consider when the prediction is needed: instantly or later?

Real-time inference is used when predictions must be made instantly, such as recommending products right after user actions.

❓ Metrics

advanced

Latency comparison between batch and real-time inference

Given a model that takes 0.01 seconds to predict one input, what is the expected latency for batch inference processing 1000 inputs at once compared to real-time inference processing one input at a time?

A Batch latency ~10 seconds; Real-time latency ~0.01 seconds per input.

B Batch latency ~100 seconds; Real-time latency ~1 second per input.

C Batch latency ~0.1 seconds; Real-time latency ~0.1 seconds per input.

D Batch latency ~0.01 seconds; Real-time latency ~10 seconds per input.

💡 Hint

Multiply prediction time by number of inputs for batch; real-time is per input.

Batch inference processes all inputs together, so total time is 0.01s * 1000 = 10s. Real-time inference processes one input at a time, so latency per input is 0.01s.

🔧 Debug

advanced

Identifying error in real-time inference code snippet

What error will this Python code raise when performing real-time inference?

def predict_real_time(model, inputs):
    results = []
    for input in inputs:
        prediction = model.predict(input)
        results.append(prediction)
    return results

# inputs is a list of data points

A NameError because 'input' is a reserved keyword.

B SyntaxError due to incorrect indentation of results.append(prediction).

C TypeError because model.predict expects a batch, not single input.

D The results list will contain only the last prediction, missing others.

💡 Hint

Check where the prediction is added to results inside the loop.

The append is outside the loop, so only the last prediction is added. The list misses all previous predictions.

❓ Model Choice

expert

Choosing model architecture for batch vs real-time inference

You have two model architectures: Model A is large and accurate but slow; Model B is smaller and faster but less accurate. For a chatbot requiring instant replies, which model is best?

A Model A, because accuracy is most important regardless of speed.

B Model B, because real-time inference needs fast responses even if less accurate.

C Use Model A for batch inference and Model B for real-time inference simultaneously.

D Neither, because chatbots should only use rule-based systems.

💡 Hint

Consider the trade-off between speed and accuracy for instant replies.

Real-time inference requires fast predictions to keep conversations smooth, so a smaller, faster model is preferred despite some accuracy loss.