Why production readiness matters in Prompt Engineering / GenAI - Challenge Your Understanding

Challenge - 5 Problems

🧠 Conceptual

intermediate

Why is monitoring important in production ML systems?

Imagine you deployed a machine learning model that predicts customer churn. Why is it important to monitor this model after deployment?

A Because monitoring automatically improves the model's accuracy without retraining.

B Because monitoring helps to reduce the size of the training dataset.

C Because the model's performance can change over time due to new data patterns.

D Because monitoring replaces the need for testing before deployment.

💡 Hint

Think about how real-world data can change after deployment.

Monitoring is crucial because data in the real world can change, causing the model to perform worse. Detecting this early helps maintain good predictions.

❓ Model Choice

intermediate

Choosing a model for production with limited resources

You need to deploy a model on a device with limited memory and processing power. Which model type is best suited for this production environment?

A A simple linear regression or small decision tree model.

B A model that requires heavy GPU computation for inference.

C An ensemble of many complex models combined.

D A large deep neural network with millions of parameters.

💡 Hint

Think about model size and speed for devices with limited resources.

Simple models like linear regression or small decision trees use less memory and compute, making them suitable for limited devices.

❓ Metrics

advanced

Evaluating model performance in production

After deploying a classification model, you observe the following confusion matrix on new data:

True Positive: 80
False Positive: 20
True Negative: 900
False Negative: 100

What is the precision of the model on this data?

A 0.50

B 0.44

C 0.90

D 0.80

💡 Hint

Precision = True Positives / (True Positives + False Positives)

Precision measures how many predicted positives are actually correct. Here, precision = 80 / (80 + 20) = 0.80.

🔧 Debug

advanced

Identifying production issue from model output drift

Your deployed model's accuracy suddenly drops. You suspect data drift. Which of the following is the most likely cause?

A The model's hyperparameters were tuned incorrectly before deployment.

B The input data distribution has changed compared to training data.

C The model was trained with too many epochs.

D The training data was too large.

💡 Hint

Think about what data drift means in production.

Data drift means the new input data looks different from what the model saw during training, causing performance to drop.

❓ Hyperparameter

expert

Selecting hyperparameters for production model stability

You want to deploy a neural network model that is stable and less likely to overfit in production. Which hyperparameter setting helps achieve this?

A Use dropout layers with a moderate dropout rate during training.

B Use a very high learning rate to speed up training.

C Use no regularization and train for many epochs.

D Use a very small batch size to increase noise during training.

💡 Hint

Think about techniques that prevent overfitting and improve generalization.

Dropout randomly disables neurons during training, which helps the model generalize better and be more stable in production.