A/B testing model versions in MLOps - Step-by-Step Execution

Process Flow - A/B testing model versions

Deploy Model Version A

↓

Deploy Model Version B

↓

Route Traffic Split

↓

Users get A

↓

Collect Metrics A

↓

Compare Performance

↓

Choose Best Model

This flow shows deploying two model versions, splitting user traffic between them, collecting performance data, and then choosing the better model.

Execution Sample

MLOps

deploy_model('v1')
deploy_model('v2')
route_traffic({'v1': 50, 'v2': 50})
collect_metrics()
compare_metrics()
choose_best_model()

This code deploys two model versions, splits traffic evenly, collects performance data, compares results, and selects the best model.

Process Table

Step	Action	Details	Result
1	Deploy Model Version A	Model v1 deployed to production	Model v1 ready
2	Deploy Model Version B	Model v2 deployed to production	Model v2 ready
3	Route Traffic Split	50% users to v1, 50% users to v2	Traffic split established
4	Users get Predictions	Users receive predictions from assigned model	Predictions served
5	Collect Metrics	Gather accuracy and latency for v1 and v2	Metrics collected: v1=0.85 acc, v2=0.88 acc
6	Compare Performance	Compare accuracy and latency of v1 vs v2	v2 performs better
7	Choose Best Model	Select model with better metrics	Model v2 chosen for full traffic
8	End	A/B test complete	Traffic routed 100% to v2

💡 A/B test ends after comparing metrics and selecting the best model version

Status Tracker

Variable	Start	After Step 3	After Step 5	After Step 7	Final
model_v1_status	not deployed	deployed	deployed	deployed	deployed
model_v2_status	not deployed	deployed	deployed	deployed	deployed
traffic_split	none	50% v1 / 50% v2	50% v1 / 50% v2	100% v2	100% v2
metrics_v1	none	none	accuracy=0.85, latency=100ms	accuracy=0.85, latency=100ms	accuracy=0.85, latency=100ms
metrics_v2	none	none	accuracy=0.88, latency=110ms	accuracy=0.88, latency=110ms	accuracy=0.88, latency=110ms
chosen_model	none	none	none	v2	v2

Key Moments - 3 Insights

Why do we split traffic between two model versions instead of switching all users at once?

Splitting traffic (see Step 3 in execution_table) lets us compare real user responses to both models safely without risking all users on a potentially worse model.

How do we decide which model is better?

We compare collected metrics like accuracy and latency (Step 6). The model with better performance metrics is chosen (Step 7).

What happens to the traffic after choosing the best model?

After selecting the best model (Step 7), all user traffic is routed to that model (Step 8), ending the A/B test.

Visual Quiz - 3 Questions

Look at the execution_table at Step 3. What is the traffic split between model versions?

A 100% to model v2

B 100% to model v1

C 50% to model v1 and 50% to model v2

D Traffic not split yet

💡 Hint

Check the 'Details' column in Step 3 of the execution_table.

According to variable_tracker, what is the accuracy of model v2 after Step 5?

A 0.88

B 0.85

C Not collected yet

D 1.00

💡 Hint

Look at the 'metrics_v2' row under 'After Step 5' in variable_tracker.

If model v1 had better accuracy than v2, what would change in the execution_table at Step 7?

A Traffic split would remain 50/50

B Model v1 would be chosen for full traffic

C Model v2 would still be chosen

D Test would end without choosing a model

💡 Hint

Step 7 shows which model is chosen based on performance comparison in Step 6.

Concept Snapshot

A/B testing model versions:
- Deploy two model versions simultaneously.
- Split user traffic between them (e.g., 50/50).
- Collect performance metrics (accuracy, latency).
- Compare metrics to find better model.
- Route all traffic to best model after test.

Full Transcript

A/B testing model versions means running two versions of a machine learning model at the same time. First, both models are deployed. Then, user traffic is split evenly so some users get predictions from model version A and others from version B. While users interact, the system collects performance data like accuracy and response time for each model. After enough data is collected, the models are compared. The one with better performance is chosen, and all user traffic is routed to that model. This process helps safely find the best model without risking all users on an untested version.

Practice

1. What is the main purpose of A/B testing in model deployment?

easy

A. To train a model faster using multiple GPUs

B. To compare two model versions by splitting user traffic

C. To backup model data in the cloud

D. To monitor server CPU usage during training

Solution

1. Step 1: Understand A/B testing concept

   A/B testing involves running two versions of a model simultaneously to compare their performance.

2. Step 2: Identify the main goal

   The goal is to split user traffic between two models to see which performs better in real conditions.

3. Final Answer:

   To compare two model versions by splitting user traffic -> Option B

4. Quick Check:

   A/B testing = compare models by traffic split [OK]

Hint: A/B testing means splitting users to compare models [OK]

Common Mistakes:

• Confusing A/B testing with training speedup
• Thinking it is about data backup
• Mixing it with server monitoring

2. Which of the following is the correct way to define a traffic split for A/B testing in YAML?

easy

A. traffic:
- model: v1
split: 50
- model: v2
split: 50

B. traffic:
modelVersion: v1
percent: 50
modelVersion: v2
percent: 50

C. traffic:
- version: v1
percent: 50
- version: v2
percent: 50

D. traffic:
- modelVersion: v1
percent: 50
- modelVersion: v2
percent: 50

Solution

1. Step 1: Check YAML list syntax for traffic split

   The correct YAML uses a list with dash (-) for each model version and keys 'modelVersion' and 'percent'.

2. Step 2: Validate keys and indentation

   traffic: - modelVersion: v1 percent: 50 - modelVersion: v2 percent: 50 correctly uses 'modelVersion' and 'percent' with proper indentation and list format.

3. Final Answer:

   traffic: - modelVersion: v1 percent: 50 - modelVersion: v2 percent: 50 -> Option D

4. Quick Check:

   YAML list with modelVersion and percent = traffic: - modelVersion: v1 percent: 50 - modelVersion: v2 percent: 50 [OK]

Hint: YAML lists use dash and proper keys for traffic split [OK]

Common Mistakes:

• Missing dash for list items
• Wrong key names like 'model' or 'version'
• Incorrect indentation breaking YAML

3. Given this Python snippet for A/B testing traffic assignment:

import random
traffic_split = {'v1': 70, 'v2': 30}
user_id = 12345
random.seed(user_id)
roll = random.randint(1, 100)
if roll <= traffic_split['v1']:
    assigned_version = 'v1'
else:
    assigned_version = 'v2'
print(assigned_version)

What will be the printed output?

medium

A. Random output each run

B. v2

C. v1

D. Error due to wrong seed usage

Solution

1. Step 1: Understand random seed and randint

   Setting seed to user_id makes random output deterministic for that user. randint(1,100) generates a number between 1 and 100.

2. Step 2: Calculate roll value for user_id=12345

   With seed 12345, roll is 54 (verified by running the code). Since 54 <= 70, assigned_version is 'v1'.

3. Final Answer:

   v1 -> Option C

4. Quick Check:

   roll=54 <= 70 means assign v1 [OK]

Hint: Seed fixes random; check roll against split [OK]

Common Mistakes:

• Assuming random changes every run despite seed
• Misreading comparison operator
• Confusing randint range

4. You have this traffic split config for A/B testing:

traffic:
  - modelVersion: v1
    percent: 60
  - modelVersion: v2
    percent: 50

What is the main problem with this configuration?

medium

A. Percentages add up to more than 100%

B. Missing modelVersion key for v2

C. Percentages must be equal for A/B testing

D. YAML syntax error due to indentation

Solution

1. Step 1: Sum the traffic percentages

   60% + 50% = 110%, which is more than 100% allowed for traffic split.

2. Step 2: Understand traffic split constraints

   Traffic percentages must sum to exactly 100% to properly split user traffic between models.

3. Final Answer:

   Percentages add up to more than 100% -> Option A

4. Quick Check:

   Sum of percents > 100% is invalid [OK]

Hint: Traffic split percentages must total 100% [OK]

Common Mistakes:

• Ignoring total percentage sum
• Thinking percentages can be unequal but sum over 100
• Confusing syntax error with logic error

5. You want to run an A/B test comparing model versions v1 and v2. You have 10,000 users and want to assign 70% traffic to v1 and 30% to v2. Which approach ensures consistent user assignment and fair metric tracking?

hard

A. Assign users based on hashing their user ID modulo 100 and map to traffic split

B. Assign users manually by checking their signup date

C. Assign all users to v1 for the first week, then switch all to v2

D. Randomly assign users on each request without storing assignment

Solution

1. Step 1: Understand consistent user assignment need

   Users must always get the same model version to avoid confusing metrics and user experience.

2. Step 2: Evaluate assignment methods

   Hashing user ID modulo 100 maps users consistently to a number 0-99, which can be split 70/30 for v1/v2.

3. Step 3: Reject other options

   Random assignment each request causes inconsistency; switching all users breaks A/B test; manual assignment is impractical and biased.

4. Final Answer:

   Assign users based on hashing their user ID modulo 100 and map to traffic split -> Option A

5. Quick Check:

   Consistent hashing ensures stable A/B assignment [OK]

Hint: Use hashing on user ID for stable traffic split [OK]

Common Mistakes:

• Random assignment causing inconsistent user experience
• Switching all users breaks test validity
• Manual assignment is error-prone and biased