A/B testing model versions in MLOps - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

Understanding A/B Testing Traffic Split

You want to run an A/B test for two model versions: v1 and v2. You configure the traffic split as 70% to v1 and 30% to v2. What will happen to incoming user requests?

A 70% of user requests will be served by model v1, and 30% by model v2.

B All user requests will be served by model v1 until 70 requests are served, then switch to v2.

C Only 30% of users will get any response, the rest will be dropped.

D User requests will be randomly served by either model without respecting the 70/30 ratio.

💡 Hint

Think about how traffic splitting works in A/B testing to distribute load.

In A/B testing, traffic split means the percentage of incoming requests routed to each model version. Here, 70% go to v1 and 30% to v2, so requests are distributed accordingly.

💻 Command Output

intermediate

Interpreting A/B Test Metrics Output

You run an A/B test comparing model v1 and v2. The output metrics show:

{"v1_accuracy": 0.85, "v2_accuracy": 0.88, "v1_requests": 700, "v2_requests": 300}

What does this output tell you?

A Model v1 has higher accuracy and more requests; traffic split is 50/50.

B Both models have the same accuracy; traffic split is incorrect.

C Model v2 has higher accuracy but fewer requests; traffic split matches 70/30 ratio.

D Model v2 has lower accuracy but more requests; traffic split is reversed.

💡 Hint

Look at accuracy values and request counts carefully.

The metrics show v2 has better accuracy (0.88 vs 0.85) and received 300 requests, which is 30% of total 1000 requests, matching the 70/30 traffic split.

❓ Configuration

advanced

Configuring Canary Deployment for Model Version

You want to deploy a new model version v3 as a canary with 10% traffic, while 90% remains on v2. Which YAML snippet correctly configures this traffic split in a Kubernetes service mesh?

A

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
spec:
  http:
  - route:
    - destination:
        host: model-service
        subset: v3
      weight: 50
    - destination:
        host: model-service
        subset: v2
      weight: 50

B

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
spec:
  http:
  - route:
    - destination:
        host: model-service
        subset: v3
      weight: 90
    - destination:
        host: model-service
        subset: v2
      weight: 10

C

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
spec:
  http:
  - route:
    - destination:
        host: model-service
        subset: v2
      weight: 10
    - destination:
        host: model-service
        subset: v3
      weight: 90

D

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
spec:
  http:
  - route:
    - destination:
        host: model-service
        subset: v2
      weight: 90
    - destination:
        host: model-service
        subset: v3
      weight: 10

💡 Hint

Remember weights must add up to 100 and match the desired traffic percentages.

The correct config routes 90% traffic to v2 and 10% to v3. Option D matches this exactly. Others reverse weights or split incorrectly.

❓ Troubleshoot

advanced

Diagnosing A/B Test Data Skew

After running an A/B test for 24 hours, you notice model v2 received 80% of traffic instead of the configured 50%. What is the most likely cause?

A The monitoring tool aggregated data incorrectly, actual traffic was 50/50.

B The traffic routing configuration was updated incorrectly, causing uneven weights.

C User sessions were sticky, causing repeated routing to v2 for returning users.

D Model v1 crashed, so all traffic was rerouted to v2 automatically.

💡 Hint

Consider what controls traffic distribution and what might cause imbalance.

Incorrect traffic routing weights or config changes can cause skew. Crashes or sticky sessions could cause issues but would not consistently cause 80% traffic to v2 unless config changed. Monitoring errors do not affect actual traffic.

🔀 Workflow

expert

Order of Steps in A/B Testing Model Deployment

Arrange the following steps in the correct order to perform a safe A/B test deployment of a new model version:

1. Analyze test results and decide winner
2. Deploy new model version alongside existing
3. Configure traffic split between old and new versions
4. Monitor performance and collect metrics

A 1,2,3,4

B 2,1,3,4

C 1,3,2,4

D 2,3,1,4

💡 Hint

Think about the logical flow from deployment to decision.

First deploy the new model alongside the old (1), then configure traffic split (2) to start sending requests to both. Next monitor and collect metrics (3), and finally analyze results to decide the winner (4).

Practice

1. What is the main purpose of A/B testing in model deployment?

easy

A. To train a model faster using multiple GPUs

B. To compare two model versions by splitting user traffic

C. To backup model data in the cloud

D. To monitor server CPU usage during training

Solution

1. Step 1: Understand A/B testing concept

   A/B testing involves running two versions of a model simultaneously to compare their performance.

2. Step 2: Identify the main goal

   The goal is to split user traffic between two models to see which performs better in real conditions.

3. Final Answer:

   To compare two model versions by splitting user traffic -> Option B

4. Quick Check:

   A/B testing = compare models by traffic split [OK]

Hint: A/B testing means splitting users to compare models [OK]

Common Mistakes:

• Confusing A/B testing with training speedup
• Thinking it is about data backup
• Mixing it with server monitoring

2. Which of the following is the correct way to define a traffic split for A/B testing in YAML?

easy

A. traffic:
- model: v1
split: 50
- model: v2
split: 50

B. traffic:
modelVersion: v1
percent: 50
modelVersion: v2
percent: 50

C. traffic:
- version: v1
percent: 50
- version: v2
percent: 50

D. traffic:
- modelVersion: v1
percent: 50
- modelVersion: v2
percent: 50

Solution

1. Step 1: Check YAML list syntax for traffic split

   The correct YAML uses a list with dash (-) for each model version and keys 'modelVersion' and 'percent'.

2. Step 2: Validate keys and indentation

   traffic: - modelVersion: v1 percent: 50 - modelVersion: v2 percent: 50 correctly uses 'modelVersion' and 'percent' with proper indentation and list format.

3. Final Answer:

   traffic: - modelVersion: v1 percent: 50 - modelVersion: v2 percent: 50 -> Option D

4. Quick Check:

   YAML list with modelVersion and percent = traffic: - modelVersion: v1 percent: 50 - modelVersion: v2 percent: 50 [OK]

Hint: YAML lists use dash and proper keys for traffic split [OK]

Common Mistakes:

• Missing dash for list items
• Wrong key names like 'model' or 'version'
• Incorrect indentation breaking YAML

3. Given this Python snippet for A/B testing traffic assignment:

import random
traffic_split = {'v1': 70, 'v2': 30}
user_id = 12345
random.seed(user_id)
roll = random.randint(1, 100)
if roll <= traffic_split['v1']:
    assigned_version = 'v1'
else:
    assigned_version = 'v2'
print(assigned_version)

What will be the printed output?

medium

A. Random output each run

B. v2

C. v1

D. Error due to wrong seed usage

Solution

1. Step 1: Understand random seed and randint

   Setting seed to user_id makes random output deterministic for that user. randint(1,100) generates a number between 1 and 100.

2. Step 2: Calculate roll value for user_id=12345

   With seed 12345, roll is 54 (verified by running the code). Since 54 <= 70, assigned_version is 'v1'.

3. Final Answer:

   v1 -> Option C

4. Quick Check:

   roll=54 <= 70 means assign v1 [OK]

Hint: Seed fixes random; check roll against split [OK]

Common Mistakes:

• Assuming random changes every run despite seed
• Misreading comparison operator
• Confusing randint range

4. You have this traffic split config for A/B testing:

traffic:
  - modelVersion: v1
    percent: 60
  - modelVersion: v2
    percent: 50

What is the main problem with this configuration?

medium

A. Percentages add up to more than 100%

B. Missing modelVersion key for v2

C. Percentages must be equal for A/B testing

D. YAML syntax error due to indentation

Solution

1. Step 1: Sum the traffic percentages

   60% + 50% = 110%, which is more than 100% allowed for traffic split.

2. Step 2: Understand traffic split constraints

   Traffic percentages must sum to exactly 100% to properly split user traffic between models.

3. Final Answer:

   Percentages add up to more than 100% -> Option A

4. Quick Check:

   Sum of percents > 100% is invalid [OK]

Hint: Traffic split percentages must total 100% [OK]

Common Mistakes:

• Ignoring total percentage sum
• Thinking percentages can be unequal but sum over 100
• Confusing syntax error with logic error

5. You want to run an A/B test comparing model versions v1 and v2. You have 10,000 users and want to assign 70% traffic to v1 and 30% to v2. Which approach ensures consistent user assignment and fair metric tracking?

hard

A. Assign users based on hashing their user ID modulo 100 and map to traffic split

B. Assign users manually by checking their signup date

C. Assign all users to v1 for the first week, then switch all to v2

D. Randomly assign users on each request without storing assignment

Solution

1. Step 1: Understand consistent user assignment need

   Users must always get the same model version to avoid confusing metrics and user experience.

2. Step 2: Evaluate assignment methods

   Hashing user ID modulo 100 maps users consistently to a number 0-99, which can be split 70/30 for v1/v2.

3. Step 3: Reject other options

   Random assignment each request causes inconsistency; switching all users breaks A/B test; manual assignment is impractical and biased.

4. Final Answer:

   Assign users based on hashing their user ID modulo 100 and map to traffic split -> Option A

5. Quick Check:

   Consistent hashing ensures stable A/B assignment [OK]

Hint: Use hashing on user ID for stable traffic split [OK]

Common Mistakes:

• Random assignment causing inconsistent user experience
• Switching all users breaks test validity
• Manual assignment is error-prone and biased