Technical debt in ML systems in MLOps - Mini Project: Build & Apply

Managing Technical Debt in ML Systems

📖 Scenario: You are working as a machine learning engineer in a team that builds ML models for predicting customer churn. Over time, the ML system has grown complex and hard to maintain. Your task is to create a simple Python script that helps identify and manage technical debt by tracking model versions and their metadata.

🎯 Goal: Build a Python script that stores model metadata, sets a threshold for maximum allowed model versions, filters out old models exceeding the threshold, and prints the list of active models. This helps keep the ML system clean and manageable.

📋 What You'll Learn

Create a dictionary called model_versions with exact keys as model version names and values as their accuracy scores.

Create a variable called max_versions set to the integer 3 to limit the number of active models.

Use a dictionary comprehension to create a new dictionary called active_models that includes only the top max_versions models by accuracy.

Print the active_models dictionary to display the current active models.

💡 Why This Matters

🌍 Real World

ML systems often accumulate many model versions and metadata, which can cause confusion and errors if not managed well. This project simulates a simple way to track and limit active models to reduce technical debt.

💼 Career

Understanding how to manage model versions and technical debt is crucial for ML engineers and MLOps specialists to maintain reliable and maintainable ML systems.

1

Create the initial model versions dictionary

Create a dictionary called model_versions with these exact entries: 'v1.0': 0.82, 'v1.1': 0.85, 'v2.0': 0.88, 'v2.1': 0.90, 'v3.0': 0.87.

MLOps

# Create the model_versions dictionary with exact version names and accuracy scores
# Your code here

Hint

Use curly braces to create a dictionary with version names as keys and accuracy scores as values.

2

Set the maximum allowed model versions

Create a variable called max_versions and set it to the integer 3.

MLOps

model_versions = {'v1.0': 0.82, 'v1.1': 0.85, 'v2.0': 0.88, 'v2.1': 0.90, 'v3.0': 0.87}
# Create max_versions variable and set it to 3
# Your code here

Hint

Just assign the number 3 to the variable max_versions.

3

Filter active models using dictionary comprehension

Use a dictionary comprehension to create a new dictionary called active_models that contains only the top max_versions models by accuracy from model_versions. Sort the models by accuracy in descending order and include only the top 3.

MLOps

model_versions = {'v1.0': 0.82, 'v1.1': 0.85, 'v2.0': 0.88, 'v2.1': 0.90, 'v3.0': 0.87}
max_versions = 3
# Create active_models dictionary with top 3 models by accuracy
# Your code here

Hint

Use sorted() with key=lambda item: item[1] and reverse=True to sort by accuracy descending. Then slice the first max_versions items and build a dictionary comprehension.

4

Print the active models

Write a print statement to display the active_models dictionary.

MLOps

model_versions = {'v1.0': 0.82, 'v1.1': 0.85, 'v2.0': 0.88, 'v2.1': 0.90, 'v3.0': 0.87}
max_versions = 3
active_models = {k: v for k, v in sorted(model_versions.items(), key=lambda item: item[1], reverse=True)[:max_versions]}
# Print the active_models dictionary
# Your code here

Hint

Use print(active_models) to show the filtered dictionary.

Practice

1. What does technical debt in ML systems usually mean?

easy

A. Extra documentation for ML models

B. Using the latest ML algorithms

C. Quick fixes that cause problems later

D. Adding more hardware resources

Solution

1. Step 1: Understand the meaning of technical debt

   Technical debt refers to shortcuts or quick fixes made during development that cause issues later.

2. Step 2: Relate to ML systems context

   In ML systems, this means messy code, missing tests, or poor design that slows future work.

3. Final Answer:

   Quick fixes that cause problems later -> Option C

4. Quick Check:

   Technical debt = Quick fixes causing future problems [OK]

Hint: Technical debt means quick fixes causing future issues [OK]

Common Mistakes:

• Confusing technical debt with adding features
• Thinking it means more hardware
• Assuming it is about documentation only

2. Which of the following is a sign of technical debt in ML code?

easy

A. Messy code with no tests

B. Well-documented and tested code

C. Using version control properly

D. Automated deployment pipelines

Solution

1. Step 1: Identify characteristics of technical debt

   Technical debt often shows as messy code and missing tests.

2. Step 2: Match options to these characteristics

   Messy code with no tests describes messy code with no tests, which is a clear sign of technical debt.

3. Final Answer:

   Messy code with no tests -> Option A

4. Quick Check:

   Messy code + no tests = Technical debt [OK]

Hint: Look for messy code and missing tests as debt signs [OK]

Common Mistakes:

• Choosing well-documented code as debt
• Confusing deployment pipelines with debt
• Thinking version control causes debt

3. Consider this ML pipeline code snippet:

def train_model(data):
    model = Model()
    model.train(data)
    return model

model1 = train_model(data1)
model2 = train_model(data2)

# Later code uses model1 and model2

What technical debt risk does this code have?

medium

A. Model objects are not saved for reuse

B. Duplicate training code causing maintenance issues

C. No risk, code is clean and reusable

D. Data is not validated before training

Solution

1. Step 1: Analyze the code behavior

   The function trains models but does not save them to disk or persistent storage.

2. Step 2: Identify technical debt risk

   Not saving models means retraining is needed every time, causing inefficiency and maintenance problems.

3. Final Answer:

   Model objects are not saved for reuse -> Option A

4. Quick Check:

   Models not saved = Technical debt risk [OK]

Hint: Check if models are saved to avoid retraining debt [OK]

Common Mistakes:

• Assuming code is clean without checking persistence
• Ignoring data validation as debt here
• Confusing duplicate code with saving models

4. You find this error in your ML system logs:

AttributeError: 'NoneType' object has no attribute 'predict'

Which technical debt issue is most likely causing this?

medium

A. Deployment pipeline is missing environment variables

B. Data preprocessing step failed silently

C. Training function has a syntax error

D. Model object was not properly saved or loaded

Solution

1. Step 1: Understand the error message

   The error means the model variable is None, so it was not loaded or saved correctly.

2. Step 2: Link error to technical debt

   Not saving/loading models properly is a common technical debt causing runtime failures.

3. Final Answer:

   Model object was not properly saved or loaded -> Option D

4. Quick Check:

   None model = save/load issue = Technical debt [OK]

Hint: None model means save/load problem causing debt [OK]

Common Mistakes:

• Blaming syntax errors for runtime NoneType errors
• Assuming data preprocessing caused this error
• Ignoring model persistence issues

5. You want to reduce technical debt in your ML system. Which approach best helps improve reliability and speed of updates?

hard

A. Train models faster by skipping data validation

B. Add automated tests and version control for models and code

C. Use complex code shortcuts to save development time

D. Avoid documentation to focus on coding

Solution

1. Step 1: Identify best practices to reduce technical debt

   Automated tests and version control improve code quality and track changes, reducing debt.

2. Step 2: Evaluate options for reliability and update speed

   Add automated tests and version control for models and code supports reliability and faster updates by preventing errors and managing versions.

3. Final Answer:

   Add automated tests and version control for models and code -> Option B

4. Quick Check:

   Tests + version control = Less technical debt [OK]

Hint: Tests and version control reduce technical debt fast [OK]

Common Mistakes:

• Skipping validation to save time increases debt
• Using shortcuts adds more debt
• Ignoring documentation harms maintainability