A/B testing model versions in MLOps - Mini Project: Build & Apply

A/B Testing Model Versions

📖 Scenario: You work in a team that deploys machine learning models. You want to compare two versions of a model to see which one performs better in real user traffic. This is called A/B testing.We will simulate a simple A/B test by assigning users to either model version A or B and counting how many users each version serves.

🎯 Goal: Build a simple A/B testing simulation that assigns users to model versions and counts the number of users served by each version.

📋 What You'll Learn

Create a list of user IDs

Define the percentage split for model versions A and B

Assign each user to a model version based on the split

Count how many users are assigned to each model version

Print the counts for both model versions

💡 Why This Matters

🌍 Real World

A/B testing is used in real machine learning deployments to compare different model versions by splitting user traffic and measuring which version performs better.

💼 Career

Understanding A/B testing helps in roles like MLOps engineer, data scientist, and DevOps engineer to safely deploy and evaluate machine learning models in production.

1

Create a list of user IDs

Create a list called user_ids containing these exact user IDs as strings: 'user1', 'user2', 'user3', 'user4', 'user5'.

MLOps

# Create the list user_ids with the given user IDs
# Your code here

Need a hint?

Use square brackets to create a list and put the user IDs as strings inside.

2

Define the A/B split percentage

Create a variable called ab_split and set it to 0.6 to represent 60% of users assigned to model version A.

MLOps

user_ids = ['user1', 'user2', 'user3', 'user4', 'user5']
# Define the ab_split variable as 0.6
# Your code here

Need a hint?

Use a simple assignment to create the variable ab_split with the value 0.6.

3

Assign users to model versions

Create a dictionary called assignments that assigns each user in user_ids to either 'A' or 'B'. Assign the first 60% of users to 'A' and the rest to 'B' using a for loop with index i and user user.

MLOps

user_ids = ['user1', 'user2', 'user3', 'user4', 'user5']
ab_split = 0.6
# Create assignments dictionary and assign users to 'A' or 'B'
# Your code here

Need a hint?

Use enumerate to get the index and user. Use int(len(user_ids) * ab_split) to find the cutoff.

4

Count and print users per model version

Create two variables count_A and count_B to count how many users are assigned to model versions 'A' and 'B' respectively. Use a for loop over assignments.values(). Then print the counts exactly as: print(f"Model A users: {count_A}") and print(f"Model B users: {count_B}").

MLOps

user_ids = ['user1', 'user2', 'user3', 'user4', 'user5']
ab_split = 0.6
assignments = {}
for i, user in enumerate(user_ids):
    if i < int(len(user_ids) * ab_split):
        assignments[user] = 'A'
    else:
        assignments[user] = 'B'
# Count users assigned to 'A' and 'B' and print the results
# Your code here

Need a hint?

Initialize counts to zero. Loop over assignments.values() and add 1 to the right count. Use f-strings to print.