Bird
Raised Fist0

Model Hub and Model Registry - Versioning, Lineage, Governance

SYSTEM-DESIGNhardInterview freq: high
Choose your preparation mode6 modes available
LearnTraceDebugTrade-offChallenge
🤖
Model Hub and Model Registry - Versioning, Lineage, Governance
Ensure robust model management with versioning, lineage, and governance.
system-designhardFreq: highML Platforms
Prerequisites:Experiment TrackingMLOps CI/CD
🎯
Problem Framing
Business Objective

Ensure compliance and reproducibility in ML model deployment across teams.

ML Objective

Implement a system for model versioning, lineage tracking, and governance.

Constraints
  • Latency SLA for model retrieval: P99 < 100ms
  • Regulatory compliance for model usage and data privacy
  • Scalability to handle 1000s of models
  • Auditability for model changes and deployments
  • Integration with existing CI/CD pipelines
Clarifying Questions
  • What are the regulatory requirements for model governance?
  • How frequently do models need to be updated or retrained?
  • What is the expected scale in terms of number of models and users?
  • What existing infrastructure needs to be integrated with the model hub?
  • Are there specific compliance standards (e.g., GDPR) that must be adhered to?
Out of Scope
✗ Feature engineering specifics
✗ Model training algorithms
✗ Real-time inference optimization
✗ Data labeling strategies
📊
Offline Metrics
Versioning accuracy: Percentage of models correctly versioned and tracked
Lineage completeness: Percentage of models with complete lineage information
📈
Online Metrics
Model retrieval latency: Average time to retrieve a model version
Deployment success rate: Percentage of successful model deployments
⚖️
Trade-offs

High versioning accuracy may increase retrieval latency due to additional checks.

💼
Business Metric

Compliance adherence rate: Percentage of models meeting regulatory standards

🗄️
Data Sources
  • Model metadata database: Stores model versions, lineage, and metadata
  • CI/CD logs: Tracks deployment and versioning events
  • User feedback logs: Captures model performance feedback
🏷️
Data Properties
Labeling Strategy

Programmatic labeling for model metadata and lineage tracking.

Data Volume

Approximately 10,000 model versions/month

Freshness

Model metadata must be updated in real-time.

⚠️
Data Challenges
  • Ensuring metadata consistency across versions
  • Handling large volumes of model lineage data
  • Integrating with diverse CI/CD systems
  • Maintaining data privacy and compliance
🔧
Key Features
categorical
Model Version ID
Uniquely identifies each model version for tracking.
numerical
Deployment Timestamp
Tracks when a model was deployed for audit purposes.
numerical
Model Performance Metrics
Evaluates model effectiveness post-deployment.
categorical
Lineage Path
Shows the model's development and deployment history.
categorical
Compliance Tags
Indicates regulatory compliance status.
⚡
Feature Engineering Notes
Real-time vs Batch

Real-time computation for deployment timestamps; batch updates for performance metrics.

Feature Store

Required

🐍
Feature Snippet
Python
def compute_lineage_path(model_id):
    # Fetch lineage from database
    lineage = db.get_lineage(model_id)
    return lineage
🤖
Model Selection
Baseline

Simple metadata storage system

Proposed Model

Comprehensive model registry with versioning and lineage tracking

Ensures compliance and auditability while supporting scalability and integration.
✗
Alternatives Rejected
Basic file-based versioning
Fails to scale and lacks lineage tracking capabilities.
Manual tracking via spreadsheets
Prone to errors and not feasible for large-scale operations.
🏋️
Training Strategy

Offline updates with real-time metadata synchronization

🚀
Serving Infrastructure
LatencyP99 < 100ms for model retrieval
Throughput1000 QPS for model metadata queries
ArchitectureSingle-stage retrieval for simplicity and speed
CachingCache recent model metadata with a TTL of 5 minutes
FallbackUse last known good model version if retrieval fails
📡
Data Drift Signals
Change in model metadata schema
Unexpected increase in model version count
Discrepancy in lineage paths
📉
Model Drift Signals
Drop in deployment success rate
Increase in retrieval latency
Mismatch in compliance tags
🔔
Alerting & Retraining
Retraining Trigger

Significant drop in compliance adherence rate

Alerting Strategy

Page on-call engineer for compliance breaches; auto-recover from metadata cache

🎯
Interview Angles
  • How to ensure compliance and auditability in model management?
  • What strategies can be used for effective model versioning?
  • How to handle scalability in a model registry system?
⚠️
Common Mistakes
Ignoring regulatory compliance requirements
Underestimating the complexity of lineage tracking
Failing to integrate with existing CI/CD pipelines
Overlooking the need for real-time metadata updates
📌
Key Decisions
  • Choosing a scalable model registry architecture
  • Implementing robust lineage tracking mechanisms
  • Ensuring real-time metadata synchronization

Practice

(1/5)
1. For an agent system designed to optimize task completion time, which metric should be prioritized?
easy
A. Latency
B. Accuracy
C. Precision
D. Recall

Solution

  1. Step 1: Understand the objective

    The goal is to optimize task completion time, which relates to how quickly tasks are executed.

  2. Step 2: Match metrics to objectives

    Latency measures the time delay in task execution, making it the relevant metric.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Task time = Latency focus [OK]
Hint: Task time optimization = focus on latency. [OK]
Common Mistakes:
2. For a RAG system focused on improving user satisfaction, which metric should be prioritized?
easy
A. Precision
B. Recall
C. User Engagement Rate
D. F1 Score

Solution

  1. Step 1: Align metrics with business objectives.

    User satisfaction is best measured by user engagement, reflecting how users interact with the system.

  2. Step 2: Eliminate wrong options.

    Precision, recall, and F1 score are technical metrics that don't directly measure user satisfaction.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    User satisfaction = engagement. [OK]
Hint: User satisfaction = engagement. [OK]
Common Mistakes:
3. In a text embedding pipeline, what is a likely failure mode if embeddings are not updated regularly with new data?
medium
A. Decreased model relevance
B. Model overfitting
C. Faster inference times
D. Increased computational cost

Solution

  1. Step 1: Consider the impact of stale embeddings.

    Embeddings that are not updated may not capture recent trends, leading to decreased relevance.

  2. Step 2: Eliminate wrong options.

    Increased cost and overfitting are unrelated to stale embeddings, and inference times are not directly affected.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Stale embeddings reduce relevance. [OK]
Hint: Stale embeddings reduce relevance. [OK]
Common Mistakes:
4. In a delivery forecasting system, when should you prefer using a simple linear regression model over a complex deep learning model?
medium
A. When the dataset is extremely large and complex
B. When interpretability and simplicity are more important than accuracy
C. When the data has non-linear patterns
D. When the model needs to capture intricate temporal dependencies

Solution

  1. Step 1: Consider the trade-offs between model complexity and interpretability

    Simple models are easier to interpret and explain.

  2. Step 2: Eliminate wrong options

    Complex models are needed for large, complex datasets (A), non-linear patterns (C), and intricate dependencies (D).

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Choose simplicity for interpretability. [OK]
Hint: Choose simplicity for interpretability. [OK]
Common Mistakes:
5. In a two-stage retrieval system, when should you prioritize a simpler candidate generation model over a complex one?
medium
A. When the ranking stage is computationally expensive and needs more resources.
B. When precision is more important than recall in the candidate generation stage.
C. When the dataset is small and computation resources are abundant.
D. When the ranking model is not capable of handling large candidate sets.

Solution

  1. Step 1: Consider resource allocation

    If the ranking stage is computationally expensive, a simpler candidate generation model can save resources for the ranking stage.

  2. Step 2: Eliminate wrong options

    When the dataset is small and computation resources are abundant. is incorrect because a small dataset doesn't necessitate a simpler model. When precision is more important than recall in the candidate generation stage. is incorrect because recall is typically prioritized in candidate generation. When the ranking model is not capable of handling large candidate sets. is incorrect because the ranking model should handle the candidate set size.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Save resources for ranking if it's costly. [OK]
Hint: Save resources for ranking if it's costly. [OK]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard