Model registry in ML Python - ML Experiment: Train & Evaluate

Experiment - Model registry

Problem:You have trained multiple versions of a machine learning model, but you lack a system to organize, track, and manage these models effectively.

Current Metrics:No model tracking or versioning system is in place, leading to confusion about which model is best or currently deployed.

Issue:Without a model registry, it is hard to manage model versions, track metadata, and deploy the best model reliably.

Your Task

Set up a simple model registry system to save, version, and load models with metadata, enabling easy tracking and deployment.

Use Python and a local file system for storing models and metadata.

Do not use any external model registry services or databases.

Keep the solution simple and beginner-friendly.

Hint 1

Hint 2

Hint 3

Solution

ML Python

import os
import json
import joblib
from datetime import datetime

class SimpleModelRegistry:
    def __init__(self, registry_dir='model_registry'):
        self.registry_dir = registry_dir
        os.makedirs(self.registry_dir, exist_ok=True)
        self.metadata_file = os.path.join(self.registry_dir, 'metadata.json')
        if not os.path.exists(self.metadata_file):
            with open(self.metadata_file, 'w') as f:
                json.dump([], f)

    def register_model(self, model, version, metrics):
        model_path = os.path.join(self.registry_dir, f'model_v{version}.joblib')
        joblib.dump(model, model_path)
        with open(self.metadata_file, 'r') as f:
            metadata = json.load(f)
        metadata.append({
            'version': version,
            'model_path': model_path,
            'metrics': metrics,
            'registered_at': datetime.now().isoformat()
        })
        with open(self.metadata_file, 'w') as f:
            json.dump(metadata, f, indent=2)
        print(f'Model version {version} registered successfully.')

    def load_model(self, version=None):
        with open(self.metadata_file, 'r') as f:
            metadata = json.load(f)
        if not metadata:
            print('No models registered yet.')
            return None
        if version is None:
            # Load latest version
            latest = max(metadata, key=lambda x: x['version'])
            version = latest['version']
        else:
            latest = next((m for m in metadata if m['version'] == version), None)
            if latest is None:
                print(f'Model version {version} not found.')
                return None
        model = joblib.load(latest['model_path'])
        print(f'Model version {version} loaded.')
        return model

# Example usage:
if __name__ == '__main__':
    from sklearn.datasets import load_iris
    from sklearn.tree import DecisionTreeClassifier
    from sklearn.model_selection import train_test_split
    from sklearn.metrics import accuracy_score

    data = load_iris()
    X_train, X_test, y_train, y_test = train_test_split(data.data, data.target, test_size=0.3, random_state=42)

    model = DecisionTreeClassifier(random_state=42)
    model.fit(X_train, y_train)
    preds = model.predict(X_test)
    acc = accuracy_score(y_test, preds)

    registry = SimpleModelRegistry()
    registry.register_model(model, version=1, metrics={'accuracy': acc})

    loaded_model = registry.load_model()  # Loads latest
    loaded_preds = loaded_model.predict(X_test)
    loaded_acc = accuracy_score(y_test, loaded_preds)
    print(f'Loaded model accuracy: {loaded_acc:.2f}')

Created a SimpleModelRegistry class to save models and metadata in a local folder.

Used joblib to save and load model files with version numbers.

Stored metadata including version, path, metrics, and registration time in a JSON file.

Added methods to register new models and load models by version or latest.

Provided example usage with a decision tree on iris dataset showing registration and loading.

Results Interpretation

Before: No model tracking, no version control, confusion about best model.

After: Models saved with versions and metrics, easy to load specific or latest model reliably.

A model registry helps organize and manage machine learning models by saving versions and metadata, making deployment and tracking easier and less error-prone.

Bonus Experiment

Extend the model registry to keep track of model deployment status (e.g., 'staging', 'production') and allow switching the production model.

💡 Hint

Add a 'status' field in metadata and create methods to update and retrieve models by status.

Practice

1.

What is the main purpose of a model registry in machine learning?

easy

A. To store and organize machine learning models

B. To train machine learning models faster

C. To clean and prepare data for training

D. To visualize model predictions

Solution

1. Step 1: Understand the role of a model registry

   A model registry is designed to keep track of machine learning models, their versions, and metadata.

2. Step 2: Compare options with the definition

   Only To store and organize machine learning models describes storing and organizing models, which matches the main purpose of a model registry.

3. Final Answer:

   To store and organize machine learning models -> Option A

4. Quick Check:

   Model registry = store and organize models [OK]

Hint: Model registry = place to keep models safe and sorted [OK]

Common Mistakes:

• Confusing model registry with data cleaning
• Thinking it speeds up training
• Mixing it with visualization tools

2.

Which of the following is the correct way to add a model to a simple Python dictionary used as a model registry?

model_registry = {}
# Add model here

easy

A. model_registry['model_v1'] = trained_model

B. model_registry.add('model_v1', trained_model)

C. model_registry.append('model_v1', trained_model)

D. model_registry.insert('model_v1', trained_model)

Solution

1. Step 1: Recall how to add key-value pairs to a dictionary

   In Python, adding an item to a dictionary uses the syntax dict[key] = value.

2. Step 2: Check each option for correct dictionary syntax

   The incorrect options use methods like .add(), .append(), and .insert() which are not available for dictionaries. Only model_registry['model_v1'] = trained_model uses the correct assignment syntax.

3. Final Answer:

   model_registry['model_v1'] = trained_model -> Option A

4. Quick Check:

   Dictionary add = dict[key] = value [OK]

Hint: Use dict[key] = value to add items in Python [OK]

Common Mistakes:

• Using list methods like append on dictionaries
• Trying to call add or insert on dict
• Confusing dictionary with list syntax

3.

Given the following code, what will be the output?

model_registry = {}
model_registry['v1'] = {'accuracy': 0.8}
model_registry['v2'] = {'accuracy': 0.85}
print(model_registry['v2']['accuracy'])

medium

A. 0.8

B. 0.85

C. {'accuracy': 0.85}

D. KeyError

Solution

1. Step 1: Understand the dictionary structure

   model_registry has keys 'v1' and 'v2', each storing a dictionary with an 'accuracy' key.

2. Step 2: Access the nested value

   model_registry['v2']['accuracy'] accesses the accuracy value for version 'v2', which is 0.85.

3. Final Answer:

   0.85 -> Option B

4. Quick Check:

   Access nested dict value = 0.85 [OK]

Hint: Access nested dict with dict[key1][key2] [OK]

Common Mistakes:

• Confusing keys and values
• Expecting full dict instead of value
• Mistyping keys causing KeyError

4.

Identify the error in this code snippet that tries to update a model's accuracy in a registry:

model_registry = {'v1': {'accuracy': 0.8}}
model_registry.update('v1', {'accuracy': 0.85})

medium

A. Syntax error due to missing colon

B. update cannot update nested dictionaries

C. The key 'v1' does not exist in the dictionary

D. update method expects a dictionary, not two arguments

Solution

1. Step 1: Check the usage of dictionary update method

   The update() method takes a single dictionary or iterable of key-value pairs, not separate arguments.

2. Step 2: Identify the correct way to update nested dictionary

   To update nested dict, access it first: model_registry['v1'].update({'accuracy': 0.85}).

3. Final Answer:

   update method expects a dictionary, not two arguments -> Option D

4. Quick Check:

   dict.update() needs one dict argument [OK]

Hint: dict.update() takes one dict, not multiple args [OK]

Common Mistakes:

• Passing multiple arguments to update()
• Trying to update nested dict without accessing it
• Assuming update() works like assignment

5.

You want to keep track of multiple versions of a model with their accuracy and deployment status. Which data structure best suits a simple model registry in Python?

hard

A. A set of model names without details

B. A list of tuples with version and accuracy only

C. A dictionary with version keys and nested dictionaries for metrics and status

D. A single string describing the latest model

Solution

1. Step 1: Identify the need for storing multiple details per version

   You need to store version keys, accuracy, and deployment status, which requires nested information.

2. Step 2: Evaluate data structures for this need

   A dictionary with version keys and nested dictionaries for details fits best. Lists or sets lack easy key-based access or nested info.

3. Final Answer:

   A dictionary with version keys and nested dictionaries for metrics and status -> Option C

4. Quick Check:

   Nested dicts store complex model info best [OK]

Hint: Use nested dicts for versions and details [OK]

Common Mistakes:

• Using lists or sets that lack keys
• Storing only one detail per model
• Using strings that can't hold structured data