Canary releases for model updates in MLOps - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

Understanding Canary Release Purpose

Why is a canary release used when updating machine learning models in production?

A To deploy the new model only in a testing environment without any user traffic

B To immediately replace the old model with the new one for all users to save time

C To run both old and new models in parallel and average their predictions for all users

D To gradually expose the new model to a small portion of users to monitor performance before full deployment

💡 Hint

Think about minimizing risk when introducing changes.

A canary release gradually exposes the new model to a small user group. This helps detect issues early without affecting all users.

💻 Command Output

intermediate

Interpreting Traffic Routing Command Output

You run a command to route traffic to the new model version in a canary release. What is the output of this command?

MLOps

kubectl rollout status deployment/model-v2
kubectl patch service model-service -p '{"spec":{"selector":{"version":"v2"}}}'

A

deployment "model-v2" rolled out with warnings
service selector unchanged

B

error: deployment "model-v2" not found
service patch failed due to syntax error

C

deployment "model-v2" successfully rolled out
service patched to route traffic to version v2

D

deployment "model-v2" rolled out
service deleted accidentally

💡 Hint

Check for success messages and correct patch syntax.

The command confirms the new model deployment is ready and updates the service selector to route traffic to the new version.

🔀 Workflow

advanced

Correct Order of Canary Release Steps

Arrange the steps in the correct order for performing a canary release of a new ML model version.

A 3,1,2,4

B 1,2,3,4

C 1,3,2,4

D 2,1,3,4

💡 Hint

Think about deploying first, then routing traffic, then monitoring.

The canary release starts by deploying the new version, then routing a small traffic portion, monitoring results, and finally adjusting traffic or rolling back.

❓ Troubleshoot

advanced

Identifying Issue in Canary Release Traffic Routing

After deploying a new model version with a canary release, users report no change in predictions. What is the most likely cause?

A Traffic routing rules still direct 100% traffic to the old model version

B The new model version failed to build and was never deployed

C The monitoring system is not collecting data properly

D The new model version is slower but still serving predictions

💡 Hint

If users see no change, check where traffic is going.

If traffic routing is not updated, all user requests go to the old model, so predictions remain unchanged.

✅ Best Practice

expert

Best Practice for Canary Release Rollback

During a canary release, monitoring detects a significant drop in model accuracy. What is the best immediate action?

A Rollback the canary release by routing all traffic back to the stable model version

B Increase traffic to the new model to gather more data for analysis

C Ignore the drop and continue rollout to all users

D Deploy a third model version immediately without rollback

💡 Hint

When performance drops, safety first.

Rolling back quickly to the stable version prevents poor user experience and allows time to fix the new model.

Practice

1. What is the main purpose of a canary release when updating machine learning models?

easy

A. To train the model faster using more data

B. To immediately replace the old model with the new one for all users

C. To test the new model on a small group of users before full deployment

D. To reduce the size of the model for faster inference

Solution

1. Step 1: Understand canary release concept

   Canary releases deploy a new model to a small subset of users first to test its performance safely.

2. Step 2: Compare options

   Only To test the new model on a small group of users before full deployment describes testing on a small group before full rollout, which is the main purpose.

3. Final Answer:

   To test the new model on a small group of users before full deployment -> Option C

4. Quick Check:

   Canary release = small group test [OK]

Hint: Canary means small test group before full rollout [OK]

Common Mistakes:

• Thinking canary releases replace models immediately
• Confusing canary with model training speed
• Assuming canary reduces model size

2. Which of the following is the correct way to specify 10% traffic to a new model version in a deployment configuration?

easy

A. "traffic_split": {"new_model": 10, "old_model": 90}

B. "traffic_split": {"new_model": 0.1, "old_model": 0.9}

C. "traffic_split": {"new_model": "10%", "old_model": "90%"}

D. "traffic_split": {"new_model": 1, "old_model": 9}

Solution

1. Step 1: Understand traffic split format

   Traffic splits are usually specified as fractions summing to 1.0, representing percentages as decimals.

2. Step 2: Evaluate options

   "traffic_split": {"new_model": 0.1, "old_model": 0.9} uses decimal fractions (0.1 and 0.9) correctly. "traffic_split": {"new_model": 10, "old_model": 90} uses integers but not fractions. "traffic_split": {"new_model": "10%", "old_model": "90%"} uses strings with percent signs, which is invalid syntax. "traffic_split": {"new_model": 1, "old_model": 9} sums to 10, not 1.

3. Final Answer:

   "traffic_split": {"new_model": 0.1, "old_model": 0.9} -> Option B

4. Quick Check:

   Traffic split decimals sum to 1 [OK]

Hint: Use decimals summing to 1 for traffic percentages [OK]

Common Mistakes:

• Using integers instead of decimals for traffic split
• Including percent signs in values
• Traffic splits not summing to 1

3. Given this simplified code snippet for routing traffic in a canary release:

def route_request(user_id):
    if user_id % 10 == 0:
        return "new_model"
    else:
        return "old_model"

print(route_request(20))
print(route_request(23))

What will be the output?

medium

A. new_model\nold_model

B. old_model\nnew_model

C. new_model\nnew_model

D. old_model\nold_model

Solution

1. Step 1: Analyze routing logic

   The function sends users with user_id divisible by 10 to the new model, others to old model.

2. Step 2: Evaluate given user_ids

   For user_id 20: 20 % 10 == 0, so returns "new_model". For user_id 23: 23 % 10 == 3, so returns "old_model".

3. Final Answer:

   new_model old_model -> Option A

4. Quick Check:

   Divisible by 10 = new_model [OK]

Hint: Check modulo condition for routing [OK]

Common Mistakes:

• Misunderstanding modulo operator
• Swapping outputs for user IDs
• Assuming all users get new model

4. You deployed a canary release but noticed the new model is receiving 100% of traffic instead of 10%. Which fix will correct this issue?

medium

A. Change traffic split from {"new_model": 1, "old_model": 0} to {"new_model": 0.1, "old_model": 0.9}

B. Increase the new model traffic to 50% to balance load

C. Restart the deployment without changing traffic split

D. Remove the old model from deployment

Solution

1. Step 1: Identify traffic split error

   Current split {"new_model": 1, "old_model": 0} sends all traffic to new model, causing 100% traffic.

2. Step 2: Correct traffic split values

   Setting split to {"new_model": 0.1, "old_model": 0.9} correctly routes 10% traffic to new model and 90% to old model.

3. Final Answer:

   Change traffic split from {"new_model": 1, "old_model": 0} to {"new_model": 0.1, "old_model": 0.9} -> Option A

4. Quick Check:

   Traffic split controls user percentage [OK]

Hint: Check traffic split decimals sum to 1 [OK]

Common Mistakes:

• Restarting without fixing traffic split
• Increasing new model traffic without reason
• Removing old model prematurely

5. You want to safely update a model with a canary release. The new model shows better accuracy but higher latency. What is the best approach to decide whether to proceed with full rollout?

hard

A. Deploy new model only to internal users without monitoring

B. Ignore latency since accuracy is more important; rollout immediately

C. Increase traffic to new model to 100% to gather more data quickly

D. Monitor both accuracy and latency metrics during canary; rollback if latency impact is unacceptable

Solution

1. Step 1: Understand trade-offs in canary release

   Canary releases test new model performance including accuracy and latency to ensure overall user experience.

2. Step 2: Choose monitoring and rollback strategy

   Monitoring both metrics allows informed decision; rollback if latency harms user experience despite accuracy gains.

3. Final Answer:

   Monitor both accuracy and latency metrics during canary; rollback if latency impact is unacceptable -> Option D

4. Quick Check:

   Balance metrics and rollback if needed [OK]

Hint: Watch all key metrics before full rollout [OK]

Common Mistakes:

• Ignoring latency impact
• Rushing full rollout without monitoring
• Skipping rollback plans