A/B testing model versions in MLOps - Commands & Configuration

Introduction

When you have two versions of a machine learning model, A/B testing helps you compare their performance by sending some users to version A and others to version B. This way, you can see which model works better in real life before fully switching.

When you want to test a new model version without stopping the current one.

When you want to compare two models to see which predicts better on real user data.

When you want to gradually roll out a new model to avoid sudden failures.

When you want to collect feedback or metrics separately for two model versions.

When you want to minimize risk by not fully committing to a new model immediately.

Commands

Create a new MLflow experiment to track the A/B testing of model versions.

Terminal

mlflow experiments create --experiment-name ab_test_models

Expected Output

Experiment 'ab_test_models' created with ID 1

→

--experiment-name - Sets the name of the experiment to organize runs

Run the model version 1 and label this run as group A for A/B testing.

Terminal

mlflow run . -P model_version=v1 -P test_group=A

Expected Output

Run ID: 123abc

Run started for model_version=v1, test_group=A

→

-P - Passes parameters to the MLflow project run

Run the model version 2 and label this run as group B for A/B testing.

Terminal

mlflow run . -P model_version=v2 -P test_group=B

Expected Output

Run ID: 456def

Run started for model_version=v2, test_group=B

→

-P - Passes parameters to the MLflow project run

Start the MLflow UI to compare the results of model versions A and B visually.

Terminal

mlflow ui

Expected Output

2024/06/01 12:00:00 Starting MLflow UI at http://127.0.0.1:5000

Key Concept

If you remember nothing else from this pattern, remember: A/B testing splits users or data to compare two model versions safely and clearly.

Code Example

MLOps

import mlflow
import random

def train_model(version):
    # Simulate training and return accuracy
    if version == 'v1':
        accuracy = 0.75 + random.uniform(-0.05, 0.05)
    else:
        accuracy = 0.80 + random.uniform(-0.05, 0.05)
    return accuracy

if __name__ == '__main__':
    import sys
    model_version = sys.argv[1] if len(sys.argv) > 1 else 'v1'
    test_group = sys.argv[2] if len(sys.argv) > 2 else 'A'

    mlflow.set_experiment('ab_test_models')
    with mlflow.start_run():
        accuracy = train_model(model_version)
        mlflow.log_param('model_version', model_version)
        mlflow.log_param('test_group', test_group)
        mlflow.log_metric('accuracy', accuracy)
        print(f'Model version {model_version} in group {test_group} logged with accuracy {accuracy:.3f}')

Common Mistakes

Running both model versions without labeling runs by test groups.

Without labeling, you cannot tell which results belong to which model version, making comparison impossible.

Always pass a parameter like test_group=A or test_group=B to distinguish runs.

Not creating a dedicated experiment for A/B testing.

Mixing A/B test runs with other experiments makes it hard to organize and analyze results.

Create a separate MLflow experiment specifically for A/B testing.

Not using the MLflow UI to compare runs.

Without the UI, it is difficult to visually compare metrics and decide which model is better.

Run 'mlflow ui' and use the web interface to analyze and compare model runs.

Summary

Create a dedicated MLflow experiment to organize A/B test runs.

Run each model version separately with clear test group labels.

Use the MLflow UI to compare metrics and decide the better model.