A/B testing models in ML Python - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

Purpose of A/B Testing in Machine Learning

What is the main goal of performing A/B testing when comparing two machine learning models?

A To determine which model performs better on a specific metric using real user data

B To combine the predictions of both models to improve accuracy

C To train both models on the same dataset simultaneously

D To reduce the size of the training dataset by splitting it

💡 Hint

Think about how A/B testing helps in decision making with live data.

A/B testing compares two models by exposing them to real users or data to see which performs better on key metrics.

❓ Predict Output

intermediate

Output of A/B Test Metric Calculation

Given the following Python code that calculates conversion rates for two models in an A/B test, what is the printed output?

ML Python

model_a_conversions = 120
model_a_visitors = 1000
model_b_conversions = 150
model_b_visitors = 1200

conversion_rate_a = model_a_conversions / model_a_visitors
conversion_rate_b = model_b_conversions / model_b_visitors

print(f"Model A Conversion Rate: {conversion_rate_a:.3f}")
print(f"Model B Conversion Rate: {conversion_rate_b:.3f}")

A

Model A Conversion Rate: 0.120
Model B Conversion Rate: 0.125

B

Model A Conversion Rate: 0.125
Model B Conversion Rate: 0.120

C

Model A Conversion Rate: 0.012
Model B Conversion Rate: 0.015

D

Model A Conversion Rate: 1.200
Model B Conversion Rate: 1.250

💡 Hint

Conversion rate is conversions divided by visitors.

Model A: 120/1000 = 0.12; Model B: 150/1200 = 0.125; formatted to 3 decimals matches option A.

❓ Hyperparameter

advanced

Choosing Sample Size for A/B Testing

Which factor is most important when deciding the sample size for an A/B test comparing two machine learning models?

A The number of layers in the neural network

B The number of features in the dataset

C The expected difference in performance between the two models

D The training time of the models

💡 Hint

Think about what affects the ability to detect a difference in results.

Sample size depends mainly on the expected effect size (difference in performance) to ensure statistical significance.

❓ Metrics

advanced

Interpreting A/B Test Results with Confidence Intervals

If the 95% confidence interval for the difference in conversion rates between Model B and Model A is [-0.01, 0.03], what can you conclude?

A Model B is significantly better than Model A

B There is no statistically significant difference between the models

C Model A is significantly better than Model B

D The test was invalid due to overlapping intervals

💡 Hint

Check if zero is inside the confidence interval.

Since zero is within the interval, the difference could be zero, so no significant difference is detected.

🔧 Debug

expert

Identifying the Bug in A/B Test Data Split Code

What error will this Python code raise when splitting data for an A/B test, and why? code: import numpy as np users = np.array([1,2,3,4,5,6,7,8,9,10]) np.random.seed(42) mask = np.random.rand(len(users)) > 0.5 ab_test_group = users[mask] control_group = users[~mask] print(f"A/B Test Group: {ab_test_group}") print(f"Control Group: {control_group}") print(f"Total users: {len(ab_test_group) + len(control_group)}")

A IndexError because mask contains values outside valid index range

B TypeError because mask is not a boolean array

C ValueError because the mask length does not match users length

D No error; the total users printed will be 10

💡 Hint

Check the type and length of the mask array.

np.random.rand(len(users)) > 0.5 creates a boolean mask of the same length as users, so indexing works fine and no error occurs.