Experiment tracking (MLflow) in ML Python - Deep Dive

Overview - Experiment tracking (MLflow)

What is it?

Experiment tracking with MLflow is a way to keep a clear record of all your machine learning tests and results. It helps you save details like model settings, data used, and performance scores in one place. This makes it easy to compare different tries and pick the best model. MLflow is a popular tool that organizes this information automatically.

Why it matters

Without experiment tracking, machine learning projects become confusing and hard to manage. You might forget which settings worked best or lose track of your progress. This slows down learning and wastes time. MLflow solves this by making every experiment easy to find, compare, and reproduce, helping teams build better models faster and with fewer mistakes.

Where it fits

Before learning MLflow experiment tracking, you should understand basic machine learning concepts like models, training, and evaluation. After mastering experiment tracking, you can explore model deployment and monitoring to complete the machine learning lifecycle.

Mental Model

Core Idea

Experiment tracking is like keeping a detailed lab notebook that records every test, setting, and result so you can review and improve your machine learning work systematically.

Think of it like...

Imagine you are baking cookies and trying different recipes. You write down each recipe’s ingredients, oven temperature, and baking time, along with how the cookies turned out. This way, you can find the best recipe later without guessing.

┌─────────────────────────────┐
│       MLflow Tracking       │
├─────────────┬───────────────┤
│ Experiment  │   Metadata    │
│  Runs       │ (params, tags)│
├─────────────┼───────────────┤
│ Artifacts   │   Metrics     │
│ (models,    │ (accuracy,    │
│  files)     │  loss, etc.)  │
└─────────────┴───────────────┘

Build-Up - 7 Steps

1

Foundation

What is Experiment Tracking

Concept: Introduce the basic idea of recording machine learning experiments to keep track of what was tried and what worked.

When you train a machine learning model, you try different settings like learning rate or number of layers. Experiment tracking means saving these settings and the results so you can compare them later. Without it, you might forget which settings gave the best results.

Result

You understand why keeping records of experiments is important to avoid confusion and wasted effort.

Understanding the need for experiment tracking helps you appreciate why tools like MLflow exist and how they improve your workflow.

2

Foundation

MLflow Basics and Components

Concept: Learn the main parts of MLflow that help with experiment tracking: tracking server, UI, and storage.

MLflow has three main parts: the tracking server that saves experiment data, the UI where you can see and compare experiments, and storage where models and files are kept. You log parameters (settings), metrics (results), and artifacts (files) for each run.

Result

You know the basic structure of MLflow and what kinds of data it manages.

Knowing MLflow’s components helps you understand how it organizes and shows your experiments.

3

Intermediate

Logging Parameters and Metrics

🤔Before reading on: do you think parameters and metrics are saved automatically or do you need to tell MLflow explicitly? Commit to your answer.

Concept: Learn how to manually log parameters and metrics during model training using MLflow’s API.

In your training code, you use MLflow functions to log parameters like learning rate and metrics like accuracy. For example, mlflow.log_param('lr', 0.01) saves the learning rate, and mlflow.log_metric('accuracy', 0.85) saves the accuracy after training.

Result

You can record key details of each experiment run so they appear in MLflow’s UI for comparison.

Knowing that you must explicitly log parameters and metrics prevents missing important data and ensures your experiments are fully tracked.

4

Intermediate

Saving and Managing Artifacts

🤔Before reading on: do you think MLflow stores model files automatically or do you need to save them yourself? Commit to your answer.

Concept: Understand how to save model files and other outputs as artifacts in MLflow for later use or deployment.

Artifacts are files like trained model weights or plots. You save them using mlflow.log_artifact('model.pkl'). MLflow stores these files linked to the experiment run, so you can download or deploy the exact model later.

Result

You can keep all important files from your experiments organized and accessible.

Knowing how to save artifacts ensures your models and outputs are preserved, enabling reproducibility and deployment.

5

Intermediate

Using MLflow UI to Compare Runs

Concept: Learn how to use the MLflow web interface to view, filter, and compare experiment runs visually.

After logging experiments, open MLflow UI in your browser. You see a table of runs with parameters and metrics. You can sort by accuracy or filter by learning rate to find the best runs. This visual comparison helps pick the best model quickly.

Result

You can easily analyze and choose the best experiment without manual record-keeping.

Using the UI turns raw data into actionable insights, speeding up model selection.

6

Advanced

Organizing Experiments and Runs

🤔Before reading on: do you think MLflow automatically groups runs into experiments or do you need to create experiments explicitly? Commit to your answer.

Concept: Understand how to create and manage multiple experiments in MLflow to keep projects organized.

MLflow lets you create named experiments to group related runs. You create an experiment with mlflow.create_experiment('MyExperiment') and then log runs under it. This helps separate projects or model versions clearly.

Result

You keep your work organized and avoid mixing unrelated experiments.

Knowing how to organize experiments prevents confusion in larger projects with many runs.

7

Expert

Advanced Tracking: Autologging and Integration

🤔Before reading on: do you think MLflow can automatically log parameters and metrics without manual code? Commit to your answer.

Concept: Explore MLflow’s autologging features and integration with popular ML libraries to simplify tracking.

MLflow supports autologging for libraries like scikit-learn and TensorFlow. By calling mlflow.sklearn.autolog(), MLflow automatically records parameters, metrics, and models without extra code. This reduces errors and speeds up tracking. You can also integrate MLflow with cloud storage and CI/CD pipelines for production use.

Result

You can track experiments with minimal code changes and scale tracking in real projects.

Understanding autologging and integrations shows how MLflow fits into professional workflows and reduces manual effort.

Under the Hood

MLflow runs a tracking server that stores experiment data in a database or file system. When you log parameters, metrics, or artifacts, MLflow sends this data to the server via API calls. The server organizes data by experiments and runs, storing metadata and files separately. The UI queries this server to display experiment details. Artifacts are saved in a storage backend like local disk or cloud storage. This separation allows efficient querying and retrieval.

Why designed this way?

MLflow was designed to be flexible and language-agnostic, supporting many ML frameworks and storage backends. Separating metadata from artifacts allows fast search and scalable storage. The server-client model enables collaboration across teams. Alternatives like manual logging or spreadsheets were error-prone and unscalable, so MLflow provides a structured, automated solution.

┌───────────────┐       ┌───────────────┐       ┌───────────────┐
│  MLflow API   │──────▶│ Tracking      │──────▶│ Storage       │
│ (client code) │       │ Server        │       │ (DB + Files)  │
└───────────────┘       └───────────────┘       └───────────────┘
        │                      ▲                       ▲
        │                      │                       │
        ▼                      │                       │
┌───────────────┐              │                       │
│ MLflow UI     │◀────────────┘                       │
│ (browser)     │                                      │
└───────────────┘                                      │
                                                       │
                                               ┌───────────────┐
                                               │ Artifact      │
                                               │ Storage       │
                                               └───────────────┘

Myth Busters - 4 Common Misconceptions

Quick: Does MLflow automatically track all parameters and metrics without any code changes? Commit to yes or no.

Common Belief: MLflow automatically tracks everything about your model without extra code.

Reality: You must explicitly log parameters, metrics, and artifacts unless you use autologging features.

Why it matters: Assuming automatic tracking leads to missing important experiment details, making comparisons incomplete or impossible.

Quick: Is experiment tracking only useful for big teams, not individual learners? Commit to yes or no.

Common Belief: Experiment tracking is only necessary for large teams working on complex projects.

Reality: Even individual learners benefit greatly from tracking experiments to avoid confusion and speed learning.

Why it matters: Ignoring tracking early causes wasted time and frustration, slowing down progress.

Quick: Can MLflow replace all parts of the machine learning workflow? Commit to yes or no.

Common Belief: MLflow handles everything from data cleaning to deployment automatically.

Reality: MLflow focuses on experiment tracking and model management; other steps require separate tools.

Why it matters: Expecting MLflow to do everything leads to incomplete workflows and unmet needs.

Quick: Does saving artifacts mean your model is automatically ready for production? Commit to yes or no.

Common Belief: Once MLflow saves a model artifact, it is production-ready without further steps.

Reality: Saved models often need additional testing, packaging, and deployment steps before production use.

Why it matters: Misunderstanding this can cause premature deployment and unreliable systems.

Expert Zone

1

MLflow’s tracking server can be configured with different backends (SQL, file system) to balance speed and scalability depending on project size.

2

Autologging simplifies tracking but may miss custom metrics or parameters, so manual logging is still important for full control.

3

MLflow supports tagging runs with custom labels, enabling complex filtering and grouping beyond simple experiments.

When NOT to use

MLflow is not ideal if you need real-time experiment tracking with streaming data or extremely high-frequency updates. Alternatives like TensorBoard or custom logging may be better. Also, if your project requires integrated data versioning, tools like DVC complement MLflow.

Production Patterns

In production, MLflow is often integrated with CI/CD pipelines to automatically log experiments from training jobs. Teams use MLflow’s model registry to stage, approve, and deploy models systematically. It also integrates with cloud storage and Kubernetes for scalable model management.

Connections

Version Control Systems (Git)

Both track changes over time but Git tracks code changes while MLflow tracks experiment changes.

Understanding version control helps grasp why tracking experiments systematically is crucial for reproducibility and collaboration.

Scientific Method

Experiment tracking mirrors the scientific method’s need to record hypotheses, procedures, and results for validation.

Seeing MLflow as a digital lab notebook connects machine learning practice to fundamental scientific principles.

Project Management Tools

Both organize work into tasks and track progress; MLflow organizes experiments and results similarly.

Recognizing MLflow as a project management tool for ML experiments highlights its role in team coordination and workflow efficiency.

Common Pitfalls

#1 Not logging parameters and metrics explicitly.

Wrong approach:

mlflow.start_run()
# train model
# no calls to mlflow.log_param or mlflow.log_metric
mlflow.end_run()

Correct approach:

mlflow.start_run()
mlflow.log_param('learning_rate', 0.01)
mlflow.log_metric('accuracy', 0.85)
mlflow.end_run()

Root cause:

Assuming MLflow tracks everything automatically without explicit logging calls.

#2 Saving model files outside MLflow artifact system.

Wrong approach:

model.save('model.pkl') # saved locally but not logged to MLflow

Correct approach:

model.save('model.pkl')
mlflow.log_artifact('model.pkl')

Root cause:

Not understanding that MLflow needs to manage artifacts to link them with experiment runs.

#3 Mixing unrelated experiments in one MLflow experiment.

Wrong approach:

Using mlflow.start_run() without creating separate experiments for different projects.

Correct approach:

mlflow.create_experiment('ProjectA')
mlflow.set_experiment('ProjectA')
mlflow.start_run()

Root cause:

Not organizing experiments properly, leading to confusion and hard-to-find results.

Key Takeaways

Experiment tracking is essential to keep clear records of machine learning tests, making it easier to compare and improve models.

MLflow provides a structured way to log parameters, metrics, and artifacts, and visualize experiments through its UI.

Explicitly logging details is necessary unless you use autologging features; otherwise, important data can be lost.

Organizing experiments and runs properly prevents confusion and supports collaboration in larger projects.

Advanced MLflow features like autologging and model registry help integrate experiment tracking into professional workflows.