Promoting models between stages in MLOps - Deep Dive

Overview - Promoting models between stages

What is it?

Promoting models between stages means moving a machine learning model from one phase of its lifecycle to the next, such as from development to testing, and then to production. Each stage represents a level of readiness and confidence in the model's quality and performance. This process ensures that only well-tested and reliable models are used in real-world applications. It helps teams manage model versions and control deployment safely.

Why it matters

Without promoting models carefully, unreliable or untested models might be used in production, causing wrong decisions or failures. This can lead to loss of trust, wasted resources, or even harm if the model controls critical systems. Promotion creates a clear path for quality checks and approvals, making sure models improve step-by-step before affecting users. It also helps teams track progress and rollback if needed.

Where it fits

Before learning model promotion, you should understand basic machine learning workflows and version control concepts. After mastering promotion, you can explore automated deployment pipelines, monitoring models in production, and continuous retraining strategies.

Mental Model

Core Idea

Model promotion is like passing a baton in a relay race, where each stage hands off the model only after confirming it is ready for the next challenge.

Think of it like...

Imagine baking a cake in stages: first you prepare the batter (development), then you bake and taste it (testing), and finally you serve it to guests (production). You only move to the next step when the current one is successful, ensuring the cake is delicious and safe to eat.

┌─────────────┐    ┌─────────────┐    ┌──────────────┐
│ Development │───▶│   Testing   │───▶│  Production  │
└─────────────┘    └─────────────┘    └──────────────┘
       │                  │                  │
       ▼                  ▼                  ▼
  Model created      Model validated    Model deployed
  and trained       and approved       for real use

Build-Up - 7 Steps

1

Foundation

Understanding model lifecycle stages

Concept: Introduce the basic stages a machine learning model goes through from creation to deployment.

Models start in development where they are trained and tuned. Then they move to testing where their performance is checked on new data. Finally, models that pass tests are deployed to production to make real predictions.

Result

Learners can identify and name the main stages in a model's lifecycle.

Knowing the stages helps organize work and ensures models are reliable before real use.

2

Foundation

What model promotion means

Concept: Explain the idea of moving a model from one stage to another as a controlled process.

Promotion means officially moving a model from development to testing or from testing to production. It involves checks and approvals to confirm the model is ready for the next step.

Result

Learners understand promotion as a quality gate, not just copying files.

Seeing promotion as a checkpoint prevents premature deployment of bad models.

3

Intermediate

Versioning models for promotion

🤔Before reading on: do you think model promotion requires keeping track of model versions? Commit to your answer.

Concept: Introduce model versioning as a key practice to manage promotions safely.

Each model version has a unique ID or tag. When promoting, you move a specific version through stages. This helps track changes, compare models, and rollback if needed.

Result

Learners see how versioning supports clear promotion paths and traceability.

Understanding versioning prevents confusion and errors when multiple models exist.

4

Intermediate

Using metadata and approval workflows

🤔Before reading on: do you think promotion is automatic or requires human approval? Commit to your answer.

Concept: Explain how metadata and approval steps control promotion decisions.

Metadata stores info like performance metrics and test results. Approval workflows require experts to review this data before promotion. This ensures only good models move forward.

Result

Learners grasp the importance of human checks combined with automation.

Knowing approval workflows reduces risks of deploying faulty models.

5

Intermediate

Automating promotion with pipelines

Concept: Show how automation tools can move models between stages based on rules.

CI/CD pipelines can run tests and if passed, automatically promote models. This speeds up delivery and reduces manual errors. Pipelines can also notify teams about promotions.

Result

Learners understand how automation makes promotion faster and safer.

Seeing automation as a guardrail helps scale model deployment reliably.

6

Advanced

Handling rollback and staging environments

🤔Before reading on: do you think once a model is promoted to production, it can’t be undone? Commit to your answer.

Concept: Discuss strategies to revert to previous models and use staging areas.

If a promoted model causes issues, rollback means switching back to a prior stable version. Staging environments mimic production to test models before full deployment, reducing risks.

Result

Learners appreciate safety nets in model promotion.

Knowing rollback and staging prevents costly production failures.

7

Expert

Challenges with multi-model and continuous promotion

🤔Before reading on: do you think promoting multiple models simultaneously is straightforward? Commit to your answer.

Concept: Explore complexities when many models or frequent updates require coordinated promotion.

In real systems, many models serve different tasks or customers. Continuous promotion means models update often. Managing dependencies, conflicts, and monitoring becomes critical to avoid errors or degraded service.

Result

Learners see the real-world complexity beyond simple promotion.

Understanding these challenges prepares learners for scaling MLOps in production.

Under the Hood

Model promotion systems track model versions and metadata in a registry or database. When a promotion request occurs, the system verifies criteria like test results and approvals. It then updates the model's stage label and triggers deployment or notifications. Underneath, pipelines orchestrate these steps using scripts or tools like Jenkins, GitHub Actions, or specialized MLOps platforms.

Why designed this way?

Promotion was designed to prevent accidental deployment of untested models and to provide traceability. Early ML projects suffered from chaotic deployments causing errors. Structured promotion with versioning and approvals balances speed and safety. Alternatives like manual copying were error-prone and lacked audit trails.

┌───────────────┐       ┌───────────────┐       ┌───────────────┐
│ Model Registry│──────▶│ Promotion Logic│──────▶│ Deployment Env│
│ (versions +   │       │ (checks +     │       │ (production,  │
│ metadata)     │       │ approvals)    │       │ staging)      │
└───────────────┘       └───────────────┘       └───────────────┘
         ▲                      │                       │
         │                      ▼                       ▼
   Model Training          Notification             Model Serving

Myth Busters - 4 Common Misconceptions

Quick: Is model promotion just copying files from one folder to another? Commit yes or no.

Common Belief: Model promotion is simply moving model files between folders or servers.

Reality: Promotion involves version control, metadata checks, approvals, and often automation pipelines, not just file copying.

Why it matters: Treating promotion as file copying risks deploying wrong or untested models, causing failures.

Quick: Do you think once a model is promoted to production, it can’t be changed? Commit yes or no.

Common Belief: Once a model is in production, it is fixed and cannot be rolled back or replaced easily.

Reality: Models can and should be rolled back or replaced quickly if issues arise, using versioning and deployment strategies.

Why it matters: Believing models are permanent leads to slow fixes and prolonged errors in production.

Quick: Do you think human approval is unnecessary if tests pass? Commit yes or no.

Common Belief: Automated tests alone are enough; human approval is not needed for promotion.

Reality: Human review is often essential to catch issues tests miss, such as ethical concerns or business impact.

Why it matters: Skipping human approval can cause harmful or biased models to reach users.

Quick: Is promoting multiple models at once as simple as promoting one? Commit yes or no.

Common Belief: Promoting many models simultaneously is just like promoting a single model repeatedly.

Reality: Multi-model promotion requires coordination to handle dependencies, conflicts, and resource limits.

Why it matters: Ignoring this leads to deployment conflicts, degraded performance, or inconsistent results.

Expert Zone

1

Promotion metadata often includes lineage info showing which data and code produced the model, aiding audits.

2

Some systems use canary promotion, deploying models to a small user subset first to monitor impact before full rollout.

3

Promotion pipelines may integrate with feature stores and monitoring tools to automate feedback loops for retraining.

When NOT to use

Model promotion is less useful in experimental or research settings where rapid iteration without strict controls is preferred. Instead, use ad-hoc testing and manual deployment. Also, for very simple models or scripts, direct deployment without formal promotion may suffice.

Production Patterns

In production, teams use model registries like MLflow or SageMaker Model Registry to track versions and stages. Automated CI/CD pipelines run tests and trigger promotions with human approvals via dashboards. Canary deployments and blue-green deployments are common to reduce risk. Monitoring alerts trigger rollbacks if performance drops.

Connections

Software Continuous Integration/Continuous Deployment (CI/CD)

Model promotion builds on CI/CD principles by applying them to machine learning models instead of code.

Understanding software CI/CD helps grasp how automation and approvals improve model deployment safety and speed.

Quality Control in Manufacturing

Both involve staged inspections and approvals before a product moves to the next phase or market.

Seeing model promotion as quality control clarifies why checks and approvals prevent defects reaching customers.

Project Management Stage Gates

Model promotion is like stage gates in projects where progress requires meeting criteria and approvals.

Knowing stage gates helps understand promotion as a formal decision point, not just a technical step.

Common Pitfalls

#1 Promoting models without version control

Wrong approach:

Copying model files manually to production without tagging or tracking versions.

Correct approach:

Use a model registry to assign unique version IDs and promote specific versions through stages.

Root cause:

Misunderstanding that promotion requires traceability and control over model versions.

#2 Skipping testing stage and promoting directly to production

Wrong approach:

Deploying a newly trained model straight to production without validation.

Correct approach:

First promote the model to a testing stage, run validations, then promote to production after approval.

Root cause:

Underestimating the risk of untested models causing failures.

#3 Relying solely on automated tests without human review

Wrong approach:

Automating promotion immediately after tests pass, no manual checks.

Correct approach:

Include human approval steps in the promotion pipeline to review metrics and business impact.

Root cause:

Overconfidence in automated tests missing ethical or contextual issues.

Key Takeaways

Promoting models between stages ensures only tested and approved models reach production, reducing risks.

Model versioning and metadata tracking are essential to manage promotions safely and enable rollbacks.

Combining automation with human approvals balances speed and quality in model deployment.

Staging environments and rollback strategies provide safety nets for production model updates.

Real-world promotion involves handling multiple models and continuous updates, requiring coordination and monitoring.

Practice

1. What is the main purpose of promoting a machine learning model between stages like testing and production?

easy

A. To move a model to a more stable and reliable environment

B. To delete old versions of the model

C. To retrain the model with new data automatically

D. To change the model's algorithm

Solution

1. Step 1: Understand model promotion

   Promoting a model means moving it from one stage to another, like from testing to production.

2. Step 2: Identify the purpose of promotion

   This process ensures the model is stable and reliable before it is used live.

3. Final Answer:

   To move a model to a more stable and reliable environment -> Option A

4. Quick Check:

   Model promotion = move to stable environment [OK]

Hint: Promotion means moving model to a safer, stable stage [OK]

Common Mistakes:

• Confusing promotion with retraining
• Thinking promotion deletes models
• Assuming promotion changes model algorithms

2. Which of the following is the correct command syntax to promote a model named my_model version 3 to the Production stage using MLflow CLI?

easy

A. mlflow models promote --name my_model --version 3 --stage Production

B. mlflow model promote --model my_model --ver 3 --to Production

C. mlflow models transition --name my_model --version 3 --stage Production

D. mlflow models transition-stage --model-name my_model --version 3 --stage Production

Solution

1. Step 1: Identify correct MLflow CLI command

   The MLflow CLI uses mlflow models transition-stage to promote models between stages.

2. Step 2: Check command options

   The correct options are --model-name, --version, and --stage to specify the model, version, and target stage.

3. Final Answer:

   mlflow models transition-stage --model-name my_model --version 3 --stage Production -> Option D

4. Quick Check:

   MLflow promote command = transition-stage with correct flags [OK]

Hint: Use 'transition-stage' with --model-name, --version, --stage flags [OK]

Common Mistakes:

• Using 'promote' instead of 'transition-stage'
• Wrong flag names like --name or --ver
• Mixing singular/plural 'model' vs 'models'

3. Given the following MLflow CLI command:
mlflow models transition-stage --model-name sales_forecast --version 5 --stage Staging
What will be the result of running this command?

medium

A. The model version 5 of sales_forecast is moved to the Staging stage

B. The model sales_forecast version 5 is deleted

C. A new version 6 of sales_forecast is created in Staging

D. The model sales_forecast version 5 is retrained automatically

Solution

1. Step 1: Understand the command purpose

   The command transition-stage moves a specific model version to a new stage.

2. Step 2: Analyze the command parameters

   It targets model sales_forecast, version 5, moving it to Staging stage.

3. Final Answer:

   The model version 5 of sales_forecast is moved to the Staging stage -> Option A

4. Quick Check:

   transition-stage moves model version to new stage [OK]

Hint: transition-stage moves specified version to target stage [OK]

Common Mistakes:

• Thinking it deletes or retrains the model
• Assuming it creates a new version
• Confusing model name and version

4. You run the command mlflow models transition-stage --model-name my_model --version 2 --stage Production but get an error saying "Stage 'Production' does not exist." What is the most likely cause and fix?

medium

A. The stage name is case-sensitive; change 'Production' to 'production'

B. The stage 'Production' is not registered; create the stage before promotion

C. The MLflow server is down; restart the server

D. The model version 2 does not exist; create it first

Solution

1. Step 1: Analyze the error message

   The error says the stage 'Production' does not exist, meaning it is not registered in MLflow.

2. Step 2: Determine the fix

   You must create or register the 'Production' stage before promoting a model to it.

3. Final Answer:

   The stage 'Production' is not registered; create the stage before promotion -> Option B

4. Quick Check:

   Stage must exist before promotion [OK]

Hint: Check if target stage exists before promoting model [OK]

Common Mistakes:

• Assuming stage names are case-insensitive
• Blaming model version existence
• Ignoring server status

5. You want to automate promoting the best performing model version to Production only if it passes testing. Which approach best fits this requirement?

hard

A. Automatically promote every new model version to Production without testing

B. Manually run mlflow models transition-stage after testing

C. Use a CI/CD pipeline that runs tests, then promotes the model version to Production stage if tests pass

D. Delete all previous versions and keep only the latest model

Solution

1. Step 1: Understand automation and testing requirements

   Automation requires a pipeline that runs tests and promotes models only if tests pass.

2. Step 2: Evaluate options for automation

   Use a CI/CD pipeline that runs tests, then promotes the model version to Production stage if tests pass describes a CI/CD pipeline that tests and promotes automatically, matching the requirement.

3. Final Answer:

   Use a CI/CD pipeline that runs tests, then promotes the model version to Production stage if tests pass -> Option C

4. Quick Check:

   Automate with CI/CD pipeline and conditional promotion [OK]

Hint: Automate promotion with CI/CD pipeline after tests pass [OK]

Common Mistakes:

• Promoting without testing
• Manual promotion defeats automation
• Deleting versions unnecessarily