Blue-green deployment for models in MLOps - Step-by-Step Execution

Process Flow - Blue-green deployment for models

Start with Blue environment live

↓

Deploy new model to Green environment

↓

Test Green environment model

↓

Switch traffic

↓

Green becomes live

↓

Blue environment idle, ready for next update

This flow shows how a new model is deployed to a separate environment (Green) while the current model runs in Blue. After testing, traffic switches to Green if successful, else fixes are made.

Execution Sample

MLOps

Deploy model v2 to Green
Test Green model
If test passes:
  Switch traffic to Green
Else:
  Fix Green model

This pseudo-code shows the steps to deploy a new model version to the Green environment, test it, and switch traffic if tests pass.

Process Table

Step	Action	Environment State	Test Result	Traffic Routing
1	Blue environment live with model v1	Blue: v1 live, Green: idle	N/A	Traffic -> Blue
2	Deploy model v2 to Green	Blue: v1 live, Green: v2 deployed	N/A	Traffic -> Blue
3	Test Green model v2	Blue: v1 live, Green: v2 deployed	Pass	Traffic -> Blue
4	Switch traffic to Green	Blue: v1 idle, Green: v2 live	Pass	Traffic -> Green
5	Blue environment idle, ready for next update	Blue: idle, Green: v2 live	N/A	Traffic -> Green

💡 Traffic switched to Green environment after successful test; Blue environment is idle.

Status Tracker

Variable	Start	After Step 2	After Step 3	After Step 4	Final
Blue environment model	v1 live	v1 live	v1 live	idle	idle
Green environment model	idle	v2 deployed	v2 deployed	v2 live	v2 live
Traffic routing	Blue	Blue	Blue	Green	Green
Test result	N/A	N/A	Pass	Pass	N/A

Key Moments - 3 Insights

Why do we deploy the new model to the Green environment instead of updating Blue directly?

Deploying to Green keeps Blue live and serving traffic, so users are not affected if the new model has issues. This is shown in execution_table steps 1 and 2.

What happens if the Green model test fails?

If the test fails, traffic stays on Blue and the Green model is fixed before switching. This is implied by the decision branch in concept_flow and the absence of traffic switch in execution_table.

Why is the Blue environment kept idle after switching traffic?

Blue is kept idle as a backup to quickly roll back if needed, ensuring reliability. This is shown in execution_table step 5.

Visual Quiz - 3 Questions

Look at the execution_table at step 3, what is the test result for the Green model?

A Pass

B Fail

C Not tested

D Unknown

💡 Hint

Check the 'Test Result' column in execution_table row for step 3.

At which step does traffic switch from Blue to Green?

A Step 2

B Step 3

C Step 4

D Step 5

💡 Hint

Look at the 'Traffic Routing' column in execution_table to find when it changes to Green.

If the Green model test failed at step 3, what would happen to traffic routing?

A Switch to Green anyway

B Keep traffic on Blue

C Split traffic between Blue and Green

D Stop all traffic

💡 Hint

Refer to key_moments about test failure and traffic routing behavior.

Concept Snapshot

Blue-green deployment for models:
- Two environments: Blue (live) and Green (new)
- Deploy new model to Green, test it
- If tests pass, switch traffic to Green
- Blue becomes idle backup
- Enables zero downtime and quick rollback

Full Transcript

Blue-green deployment for models means having two environments: Blue and Green. Blue runs the current model live. We deploy the new model to Green and test it without affecting users. If tests pass, we switch user traffic to Green, making it live. Blue then becomes idle, ready to be a backup. This method avoids downtime and allows quick rollback if needed.

Practice

1. What is the main purpose of blue-green deployment in model updates?

easy

A. To run two models at the same time and combine their outputs

B. To switch traffic to a new model only after it is fully tested and ready

C. To update the model directly in the production environment without backup

D. To deploy models only during off-peak hours

Solution

1. Step 1: Understand blue-green deployment concept

   Blue-green deployment uses two separate environments to avoid downtime and risk during updates.

2. Step 2: Identify the key purpose

   The main goal is to switch traffic to the new model only after it is fully tested and ready, ensuring safety.

3. Final Answer:

   To switch traffic to a new model only after it is fully tested and ready -> Option B

4. Quick Check:

   Safe model update = A [OK]

Hint: Blue-green means switch only after testing new model [OK]

Common Mistakes:

• Thinking both models run and combine outputs
• Updating production without backup
• Deploying only during off-peak hours

2. Which command correctly switches traffic from the blue environment to the green environment in a Kubernetes service?

easy

A. kubectl set image deployment/model-deploy model=green-model:latest

B. kubectl delete deployment model-deploy-blue

C. kubectl rollout restart deployment/model-deploy-green

D. kubectl patch service model-service -p '{"spec":{"selector":{"env":"green"}}}'

Solution

1. Step 1: Understand traffic switching in Kubernetes

   Traffic is routed by the service selector labels pointing to the correct deployment environment.

2. Step 2: Identify the command that changes service selector to green

   The patch command updates the service selector to point to pods labeled with "env=green".

3. Final Answer:

   kubectl patch service model-service -p '{"spec":{"selector":{"env":"green"}}}' -> Option D

4. Quick Check:

   Change service selector = B [OK]

Hint: Patch service selector to green environment label [OK]

Common Mistakes:

• Restarting deployment does not switch traffic
• Setting image changes deployment but not traffic
• Deleting blue deployment before switch causes downtime

3. Given this simplified deployment script snippet, what will be the output after running it?

current_env = "blue"
new_env = "green"
if current_env == "blue":
    print(f"Switching traffic to {new_env} environment")
else:
    print(f"Switching traffic to {current_env} environment")

medium

A. Switching traffic to green environment

B. Switching traffic to undefined environment

C. Switching traffic to blue environment

D. No output

Solution

1. Step 1: Analyze the condition in the script

   The variable current_env is "blue", so the if condition is true.

2. Step 2: Determine the printed output

   Since current_env is "blue", it prints "Switching traffic to green environment" using new_env.

3. Final Answer:

   Switching traffic to green environment -> Option A

4. Quick Check:

   current_env == "blue" triggers green switch = C [OK]

Hint: Check if current_env is blue, then print green [OK]

Common Mistakes:

• Confusing current_env and new_env variables
• Assuming else branch runs when condition is true
• Ignoring f-string formatting

4. You tried to switch traffic to the green model environment but users still hit the blue model. What is the most likely error?

medium

A. The service selector was not updated to point to the green environment

B. The green model deployment was deleted accidentally

C. The blue environment pods crashed and restarted

D. The model version in green environment is outdated

Solution

1. Step 1: Understand traffic routing in blue-green deployment

   Traffic is controlled by the service selector labels pointing to the active environment.

2. Step 2: Identify why traffic still hits blue

   If users still hit blue, the service selector likely was not updated to green, so traffic stays on blue.

3. Final Answer:

   The service selector was not updated to point to the green environment -> Option A

4. Quick Check:

   Traffic routing depends on service selector = D [OK]

Hint: Check if service selector changed to green environment [OK]

Common Mistakes:

• Assuming green deployment deletion causes traffic to blue
• Confusing pod crashes with traffic routing
• Thinking model version affects routing directly

5. You want to implement blue-green deployment for a machine learning model with minimal downtime. Which sequence of steps is correct?

hard

A. Deploy new model to green, switch traffic immediately, then test and monitor

B. Delete blue environment, deploy new model to green, switch traffic, monitor performance

C. Deploy new model to green environment, test it, switch service selector to green, monitor, then delete blue

D. Deploy new model to blue environment, test it, switch service selector to blue, monitor, then delete green

Solution

1. Step 1: Deploy and test new model in green environment

   Deploying and testing in green ensures the new model works without affecting users.

2. Step 2: Switch traffic to green, monitor, then clean up blue

   Switching traffic only after testing reduces risk. Monitoring ensures stability before deleting blue.

3. Final Answer:

   Deploy new model to green environment, test it, switch service selector to green, monitor, then delete blue -> Option C

4. Quick Check:

   Test before switch, monitor after = A [OK]

Hint: Test green first, then switch traffic, then delete blue [OK]

Common Mistakes:

• Deleting blue before green is ready
• Switching traffic before testing
• Deploying new model to blue environment instead of green