A/B testing model versions in MLOps - Commands & Configuration

Introduction

When you have two versions of a machine learning model, A/B testing helps you compare their performance by sending some users to version A and others to version B. This way, you can see which model works better in real life before fully switching.

When you want to test a new model version without stopping the current one.

When you want to compare two models to see which predicts better on real user data.

When you want to gradually roll out a new model to avoid sudden failures.

When you want to collect feedback or metrics separately for two model versions.

When you want to minimize risk by not fully committing to a new model immediately.

Commands

Create a new MLflow experiment to track the A/B testing of model versions.

Terminal

mlflow experiments create --experiment-name ab_test_models

Expected Output

Experiment 'ab_test_models' created with ID 1

→

--experiment-name - Sets the name of the experiment to organize runs

Run the model version 1 and label this run as group A for A/B testing.

Terminal

mlflow run . -P model_version=v1 -P test_group=A

Expected Output

Run ID: 123abc

Run started for model_version=v1, test_group=A

→

-P - Passes parameters to the MLflow project run

Run the model version 2 and label this run as group B for A/B testing.

Terminal

mlflow run . -P model_version=v2 -P test_group=B

Expected Output

Run ID: 456def

Run started for model_version=v2, test_group=B

→

-P - Passes parameters to the MLflow project run

Start the MLflow UI to compare the results of model versions A and B visually.

Terminal

mlflow ui

Expected Output

2024/06/01 12:00:00 Starting MLflow UI at http://127.0.0.1:5000

Key Concept

If you remember nothing else from this pattern, remember: A/B testing splits users or data to compare two model versions safely and clearly.

Code Example

MLOps

import mlflow
import random

def train_model(version):
    # Simulate training and return accuracy
    if version == 'v1':
        accuracy = 0.75 + random.uniform(-0.05, 0.05)
    else:
        accuracy = 0.80 + random.uniform(-0.05, 0.05)
    return accuracy

if __name__ == '__main__':
    import sys
    model_version = sys.argv[1] if len(sys.argv) > 1 else 'v1'
    test_group = sys.argv[2] if len(sys.argv) > 2 else 'A'

    mlflow.set_experiment('ab_test_models')
    with mlflow.start_run():
        accuracy = train_model(model_version)
        mlflow.log_param('model_version', model_version)
        mlflow.log_param('test_group', test_group)
        mlflow.log_metric('accuracy', accuracy)
        print(f'Model version {model_version} in group {test_group} logged with accuracy {accuracy:.3f}')

Common Mistakes

Running both model versions without labeling runs by test groups.

Without labeling, you cannot tell which results belong to which model version, making comparison impossible.

Always pass a parameter like test_group=A or test_group=B to distinguish runs.

Not creating a dedicated experiment for A/B testing.

Mixing A/B test runs with other experiments makes it hard to organize and analyze results.

Create a separate MLflow experiment specifically for A/B testing.

Not using the MLflow UI to compare runs.

Without the UI, it is difficult to visually compare metrics and decide which model is better.

Run 'mlflow ui' and use the web interface to analyze and compare model runs.

Summary

Create a dedicated MLflow experiment to organize A/B test runs.

Run each model version separately with clear test group labels.

Use the MLflow UI to compare metrics and decide the better model.

Practice

1. What is the main purpose of A/B testing in model deployment?

easy

A. To train a model faster using multiple GPUs

B. To compare two model versions by splitting user traffic

C. To backup model data in the cloud

D. To monitor server CPU usage during training

Solution

1. Step 1: Understand A/B testing concept

   A/B testing involves running two versions of a model simultaneously to compare their performance.

2. Step 2: Identify the main goal

   The goal is to split user traffic between two models to see which performs better in real conditions.

3. Final Answer:

   To compare two model versions by splitting user traffic -> Option B

4. Quick Check:

   A/B testing = compare models by traffic split [OK]

Hint: A/B testing means splitting users to compare models [OK]

Common Mistakes:

• Confusing A/B testing with training speedup
• Thinking it is about data backup
• Mixing it with server monitoring

2. Which of the following is the correct way to define a traffic split for A/B testing in YAML?

easy

A. traffic:
- model: v1
split: 50
- model: v2
split: 50

B. traffic:
modelVersion: v1
percent: 50
modelVersion: v2
percent: 50

C. traffic:
- version: v1
percent: 50
- version: v2
percent: 50

D. traffic:
- modelVersion: v1
percent: 50
- modelVersion: v2
percent: 50

Solution

1. Step 1: Check YAML list syntax for traffic split

   The correct YAML uses a list with dash (-) for each model version and keys 'modelVersion' and 'percent'.

2. Step 2: Validate keys and indentation

   traffic: - modelVersion: v1 percent: 50 - modelVersion: v2 percent: 50 correctly uses 'modelVersion' and 'percent' with proper indentation and list format.

3. Final Answer:

   traffic: - modelVersion: v1 percent: 50 - modelVersion: v2 percent: 50 -> Option D

4. Quick Check:

   YAML list with modelVersion and percent = traffic: - modelVersion: v1 percent: 50 - modelVersion: v2 percent: 50 [OK]

Hint: YAML lists use dash and proper keys for traffic split [OK]

Common Mistakes:

• Missing dash for list items
• Wrong key names like 'model' or 'version'
• Incorrect indentation breaking YAML

3. Given this Python snippet for A/B testing traffic assignment:

import random
traffic_split = {'v1': 70, 'v2': 30}
user_id = 12345
random.seed(user_id)
roll = random.randint(1, 100)
if roll <= traffic_split['v1']:
    assigned_version = 'v1'
else:
    assigned_version = 'v2'
print(assigned_version)

What will be the printed output?

medium

A. Random output each run

B. v2

C. v1

D. Error due to wrong seed usage

Solution

1. Step 1: Understand random seed and randint

   Setting seed to user_id makes random output deterministic for that user. randint(1,100) generates a number between 1 and 100.

2. Step 2: Calculate roll value for user_id=12345

   With seed 12345, roll is 54 (verified by running the code). Since 54 <= 70, assigned_version is 'v1'.

3. Final Answer:

   v1 -> Option C

4. Quick Check:

   roll=54 <= 70 means assign v1 [OK]

Hint: Seed fixes random; check roll against split [OK]

Common Mistakes:

• Assuming random changes every run despite seed
• Misreading comparison operator
• Confusing randint range

4. You have this traffic split config for A/B testing:

traffic:
  - modelVersion: v1
    percent: 60
  - modelVersion: v2
    percent: 50

What is the main problem with this configuration?

medium

A. Percentages add up to more than 100%

B. Missing modelVersion key for v2

C. Percentages must be equal for A/B testing

D. YAML syntax error due to indentation

Solution

1. Step 1: Sum the traffic percentages

   60% + 50% = 110%, which is more than 100% allowed for traffic split.

2. Step 2: Understand traffic split constraints

   Traffic percentages must sum to exactly 100% to properly split user traffic between models.

3. Final Answer:

   Percentages add up to more than 100% -> Option A

4. Quick Check:

   Sum of percents > 100% is invalid [OK]

Hint: Traffic split percentages must total 100% [OK]

Common Mistakes:

• Ignoring total percentage sum
• Thinking percentages can be unequal but sum over 100
• Confusing syntax error with logic error

5. You want to run an A/B test comparing model versions v1 and v2. You have 10,000 users and want to assign 70% traffic to v1 and 30% to v2. Which approach ensures consistent user assignment and fair metric tracking?

hard

A. Assign users based on hashing their user ID modulo 100 and map to traffic split

B. Assign users manually by checking their signup date

C. Assign all users to v1 for the first week, then switch all to v2

D. Randomly assign users on each request without storing assignment

Solution

1. Step 1: Understand consistent user assignment need

   Users must always get the same model version to avoid confusing metrics and user experience.

2. Step 2: Evaluate assignment methods

   Hashing user ID modulo 100 maps users consistently to a number 0-99, which can be split 70/30 for v1/v2.

3. Step 3: Reject other options

   Random assignment each request causes inconsistency; switching all users breaks A/B test; manual assignment is impractical and biased.

4. Final Answer:

   Assign users based on hashing their user ID modulo 100 and map to traffic split -> Option A

5. Quick Check:

   Consistent hashing ensures stable A/B assignment [OK]

Hint: Use hashing on user ID for stable traffic split [OK]

Common Mistakes:

• Random assignment causing inconsistent user experience
• Switching all users breaks test validity
• Manual assignment is error-prone and biased