MLflow setup and basics in MLOps - Deep Dive

Overview - MLflow setup and basics

What is it?

MLflow is a tool that helps you manage machine learning projects easily. It tracks experiments, records results, and organizes models so you can reuse and share them. It works by letting you log data about your training runs and then view or compare them later. This makes machine learning work more organized and less confusing.

Why it matters

Without MLflow, managing machine learning experiments can become messy and error-prone. You might lose track of which model performed best or which settings you used. MLflow solves this by keeping everything in one place, making it easier to reproduce results and collaborate with others. This saves time and reduces mistakes in real projects.

Where it fits

Before learning MLflow, you should understand basic machine learning concepts and how to run training scripts. After MLflow basics, you can explore advanced model deployment, automated pipelines, and cloud-based experiment tracking. MLflow fits into the MLOps journey as the tool that organizes and tracks your machine learning work.

Mental Model

Core Idea

MLflow acts like a smart notebook that automatically records every detail of your machine learning experiments so you never lose track.

Think of it like...

Imagine you are baking different cakes trying new recipes. MLflow is like a kitchen journal where you write down each recipe, ingredients, baking time, and how the cake turned out, so you can always bake the best one again or share it with friends.

┌───────────────────────────────┐
│          MLflow Setup         │
├─────────────┬─────────────────┤
│ Components  │ Description     │
├─────────────┼─────────────────┤
│ Tracking    │ Logs experiments│
│ Server      │ Stores logs     │
│ Projects    │ Organizes code  │
│ Models      │ Manages models  │
└─────────────┴─────────────────┘

Build-Up - 7 Steps

1

Foundation

Installing MLflow and dependencies

Concept: Learn how to install MLflow and prepare your environment.

To start using MLflow, you first install it using Python's package manager. Run: pip install mlflow This command downloads MLflow and its required parts. You also need Python installed on your computer. After installation, you can check MLflow version by running: mlflow --version

Result

MLflow is installed and ready to use on your system.

Knowing how to install MLflow correctly is the first step to using it effectively and avoiding setup errors.

2

Foundation

Starting MLflow tracking server locally

Concept: Set up a local server to store experiment data.

MLflow uses a tracking server to save experiment info. You can start a local server by running: mlflow server --backend-store-uri sqlite:///mlflow.db --default-artifact-root ./mlruns --host 127.0.0.1 --port 5000 This command creates a local database and folder to save logs and artifacts.

Result

A local MLflow tracking server runs on your machine at http://127.0.0.1:5000.

Running a local server helps you keep all experiment data organized and accessible through a web interface.

3

Intermediate

Logging experiments with MLflow API

🤔Before reading on: do you think MLflow automatically tracks all your code changes or do you need to add logging commands? Commit to your answer.

Concept: Learn how to record parameters, metrics, and models during training using MLflow code commands.

In your training script, import MLflow and use its functions to log data: import mlflow with mlflow.start_run(): mlflow.log_param('learning_rate', 0.01) mlflow.log_metric('accuracy', 0.95) mlflow.log_artifact('model.pkl') This records the learning rate, accuracy, and saves the model file to MLflow.

Result

Experiment details are saved and visible in the MLflow UI for later review.

Explicit logging lets you control exactly what information is saved, making your experiments reproducible and comparable.

4

Intermediate

Using MLflow UI to compare runs

🤔Before reading on: do you think MLflow UI shows only the latest run or all past runs? Commit to your answer.

Concept: Explore the web interface to view and compare experiment results visually.

Open your browser and go to http://127.0.0.1:5000. You will see a list of experiment runs with parameters and metrics. You can select multiple runs to compare their performance side-by-side. This helps you find the best model settings quickly.

Result

You can visually analyze and compare all your experiment runs in one place.

A visual interface makes it easier to understand experiment outcomes and share results with teammates.

5

Intermediate

Organizing projects with MLflow Projects

Concept: Learn how to package your ML code and dependencies for easy sharing and reproducibility.

MLflow Projects use a simple file called MLproject to describe your project: name: MyProject conda_env: conda.yaml entry_points: main: parameters: learning_rate: {type: float, default: 0.01} command: python train.py --lr {learning_rate} This lets others run your project with the same environment and parameters.

Result

Your ML code is packaged with environment info, making it easy to run anywhere.

Packaging projects ensures consistent environments and reduces 'it works on my machine' problems.

6

Advanced

Managing models with MLflow Model Registry

🤔Before reading on: do you think MLflow Model Registry only stores model files or also tracks model versions and stages? Commit to your answer.

Concept: Use MLflow to register, version, and stage machine learning models for deployment.

After training, register your model: from mlflow.tracking import MlflowClient client = MlflowClient() model_uri = 'runs://model' client.create_registered_model('MyModel') client.create_model_version('MyModel', model_uri, '') You can then move models through stages like 'Staging' or 'Production' to manage deployment readiness.

Result

Models are tracked with versions and deployment stages, improving lifecycle management.

Model Registry adds control and safety to deploying ML models in real systems.

7

Expert

Scaling MLflow with remote servers and storage

🤔Before reading on: do you think MLflow tracking server can handle multiple users and large data by default? Commit to your answer.

Concept: Learn how to configure MLflow to use remote databases and cloud storage for team collaboration and scalability.

For teams, run MLflow server with a remote database like PostgreSQL: mlflow server --backend-store-uri postgresql://user:pass@host/dbname --default-artifact-root s3://mybucket/mlflow This setup stores metadata in a robust database and artifacts in cloud storage, allowing multiple users to track experiments safely.

Result

MLflow can support team workflows and large-scale projects with reliable storage and access.

Understanding scalable MLflow setups is key for professional MLOps in production environments.

Under the Hood

MLflow works by running a tracking server that stores experiment metadata in a database and artifacts like models or logs in a file system or cloud storage. When you call MLflow logging functions in your code, they send data to this server via an API. The server organizes data by experiments and runs, allowing retrieval and comparison. The UI queries this data to display results. MLflow Projects use environment files and entry points to recreate consistent runs. The Model Registry tracks model versions and stages in the database, enabling lifecycle management.

Why designed this way?

MLflow was designed to be modular and flexible, supporting many ML frameworks and storage backends. Using a server-client model separates experiment tracking from code execution, allowing remote access and collaboration. Storing metadata in databases ensures query efficiency, while artifact storage is decoupled for scalability. This design avoids locking users into specific tools and supports both local and cloud workflows.

┌───────────────┐       ┌───────────────┐       ┌───────────────┐
│ MLflow Client │──────▶│ Tracking      │──────▶│ Backend Store │
│ (Your Script) │       │ Server (API)  │       │ (Database)    │
└───────────────┘       └───────────────┘       └───────────────┘
         │                      │                      │
         │                      │                      │
         │                      ▼                      ▼
         │               ┌───────────────┐      ┌───────────────┐
         │               │ Artifact Store│      │ Model Registry│
         │               │ (File System/ │      │ (Metadata DB) │
         │               │  Cloud)       │      └───────────────┘
         ▼
┌─────────────────┐
│ MLflow UI       │
│ (Web Interface) │
└─────────────────┘

Myth Busters - 4 Common Misconceptions

Quick: Does MLflow automatically track every change in your code without any logging commands? Commit to yes or no.

Common Belief: MLflow automatically tracks all code changes and parameters without extra commands.

Reality: MLflow only tracks what you explicitly tell it to log using its API calls in your code.

Why it matters: Assuming automatic tracking leads to missing important experiment details and makes reproducing results impossible.

Quick: Can MLflow replace your entire machine learning pipeline automation? Commit to yes or no.

Common Belief: MLflow is a full pipeline automation tool that handles data processing, training, and deployment end-to-end.

Reality: MLflow focuses on experiment tracking, project packaging, and model management but does not automate data pipelines or deployment by itself.

Why it matters: Misusing MLflow as a pipeline tool can cause confusion and incomplete automation, requiring other tools for full MLOps.

Quick: Is it safe to use the default local MLflow server for team collaboration? Commit to yes or no.

Common Belief: The default local MLflow server setup is sufficient for multiple users working together.

Reality: The local server is single-user and not designed for concurrent access; production teams need remote servers with proper databases and storage.

Why it matters: Using local servers for teams risks data loss, conflicts, and poor scalability.

Quick: Does MLflow Model Registry only store model files without version control? Commit to yes or no.

Common Belief: Model Registry is just a storage place for model files without tracking versions or stages.

Reality: Model Registry tracks multiple versions of models and their lifecycle stages like staging or production.

Why it matters: Ignoring version control leads to deployment errors and difficulty managing model updates.

Expert Zone

1

MLflow's artifact storage can be configured separately from metadata storage, allowing flexible use of cloud buckets or local disks depending on project needs.

2

The MLflow Projects format supports multiple environment managers like Conda or Docker, enabling reproducible runs across diverse systems.

3

Model Registry integrates with CI/CD pipelines to automate model promotion and deployment, but requires careful permission and stage management.

When NOT to use

MLflow is not suitable when you need full pipeline orchestration or real-time model serving; in those cases, use tools like Kubeflow Pipelines or TensorFlow Serving. Also, for very large-scale experiment tracking, specialized platforms may be more efficient.

Production Patterns

Teams run MLflow tracking servers on cloud VMs with PostgreSQL and S3 storage for reliability. They integrate MLflow with CI pipelines to automatically log runs and register models. Model Registry stages control deployment approvals. Projects are packaged with Docker for consistent environments. The UI is used for experiment review and audit.

Connections

Version Control Systems (e.g., Git)

Both track changes and history of work artifacts over time.

Understanding version control helps grasp how MLflow tracks experiment versions and model changes systematically.

Continuous Integration/Continuous Deployment (CI/CD)

MLflow integrates with CI/CD pipelines to automate model testing and deployment.

Knowing CI/CD concepts clarifies how MLflow fits into automated workflows for reliable ML production.

Scientific Lab Notebooks

MLflow serves as a digital lab notebook for machine learning experiments.

Recognizing MLflow as a lab notebook highlights its role in organizing and documenting experiments for reproducibility.

Common Pitfalls

#1 Not starting the MLflow tracking server before logging experiments.

Wrong approach:

import mlflow
mlflow.log_param('lr', 0.01)
mlflow.log_metric('accuracy', 0.9)

Correct approach:

Start the server first:
mlflow server --backend-store-uri sqlite:///mlflow.db --default-artifact-root ./mlruns

Then in code:
import mlflow
with mlflow.start_run():
mlflow.log_param('lr', 0.01)
mlflow.log_metric('accuracy', 0.9)

Root cause:

Trying to log without a running server causes errors or lost data because MLflow has nowhere to store logs.

#2 Using local file paths for artifact storage in a multi-user environment.

Wrong approach:

mlflow server --backend-store-uri sqlite:///mlflow.db --default-artifact-root ./mlruns

Correct approach:

Use shared cloud storage:
mlflow server --backend-store-uri postgresql://user:pass@host/db --default-artifact-root s3://mybucket/mlflow

Root cause:

Local paths are not accessible to all users, causing missing artifacts and collaboration issues.

#3 Not explicitly calling mlflow.start_run() before logging parameters and metrics.

Wrong approach:

mlflow.log_param('batch_size', 32)
mlflow.log_metric('loss', 0.2)

Correct approach:

with mlflow.start_run():
mlflow.log_param('batch_size', 32)
mlflow.log_metric('loss', 0.2)

Root cause:

Logging outside a run context results in errors or logs not being saved properly.

Key Takeaways

MLflow organizes machine learning experiments by tracking parameters, metrics, and models in a central place.

You must explicitly log data in your code and run a tracking server to save experiment details.

The MLflow UI helps visualize and compare experiment runs, making analysis easier.

Model Registry manages model versions and deployment stages, improving production workflows.

Scaling MLflow for teams requires remote servers, databases, and cloud storage for reliability and collaboration.