Environment management with conda and pip in MLOps - Commands & Configuration

Introduction

Managing software environments helps keep your projects organized and avoids conflicts between different package versions. Conda and pip are tools that let you create and control these environments easily.

When you want to keep different projects isolated so their packages don't interfere with each other

When you need to install specific versions of Python or libraries for a project

When you want to share your environment setup with teammates or on another computer

When you want to add or update packages without breaking your existing setup

When you want to switch quickly between different project environments

Config File - environment.yml

environment.yml

name: my-ml-project
dependencies:
  - python=3.9
  - numpy=1.23.1
  - pandas=1.5.2
  - pip:
    - scikit-learn==1.1.3
    - matplotlib==3.6.2

This environment.yml file defines a conda environment named my-ml-project. It specifies Python 3.9 and some packages managed by conda like numpy and pandas. Under pip, it lists packages installed via pip, such as scikit-learn and matplotlib. This file lets you recreate the exact environment on any machine.

Commands

This command creates a new conda environment using the environment.yml file. It installs all listed packages and sets up Python version 3.9.

Terminal

conda env create -f environment.yml

Expected Output

Collecting package metadata (repodata.json): done

Solving environment: done



Downloading and Extracting Packages

...



Preparing transaction: done

Verifying transaction: done

Executing transaction: done



Environment 'my-ml-project' created successfully!

→

-f - Specifies the environment file to use for creation

This command switches your shell to use the newly created environment. Now, any Python or pip commands will use the packages from this environment.

Terminal

conda activate my-ml-project

Expected Output

(my-ml-project) $ 

This command shows all Python packages installed in the current environment via pip. It helps verify that pip packages like scikit-learn and matplotlib are installed.

Terminal

pip list

Expected Output

Package         Version

---------------- -------

matplotlib      3.6.2

pip             23.0.1

scikit-learn    1.1.3

setuptools      65.5.0

wheel           0.38.4

This command saves the current environment's full package list and versions to a file. You can share this file to recreate the environment exactly elsewhere.

Terminal

conda env export > environment_export.yml

Expected Output

No output (command runs silently)

This command exits the current conda environment and returns to the default system environment.

Terminal

conda deactivate

Expected Output

$ 

Key Concept

If you remember nothing else from this pattern, remember: conda manages environments and packages, while pip installs Python packages inside those environments.

Common Mistakes

Trying to install packages with pip outside an activated conda environment

Packages get installed globally or in the wrong environment, causing conflicts or missing packages in your project

Always activate the correct conda environment before running pip install

Editing the environment.yml file manually without proper indentation

YAML files are sensitive to indentation and format; errors cause environment creation to fail

Use a YAML editor or carefully maintain correct indentation and syntax

Not exporting the environment after installing new packages

The environment.yml file becomes outdated and others cannot recreate the exact setup

Run conda env export > environment_export.yml after changes to keep the file updated

Summary

Create a conda environment from an environment.yml file to manage Python and packages together.

Activate the environment to work inside it and use pip to install additional Python packages.

Export the environment to a file to share or recreate the setup exactly on other machines.

Practice

1. What is the main purpose of using conda create -n myenv python=3.8?

easy

A. To delete the environment named 'myenv' and install Python 3.8 globally

B. To update Python to version 3.8 in the current environment

C. To create a new isolated environment named 'myenv' with Python 3.8 installed

D. To install all packages listed in a file named 'myenv' with Python 3.8

Solution

1. Step 1: Understand the conda create command

   This command is used to create a new environment in conda, isolating packages and Python versions.

2. Step 2: Analyze the flags and arguments

   The -n myenv specifies the environment name, and python=3.8 sets the Python version inside it.

3. Final Answer:

   To create a new isolated environment named 'myenv' with Python 3.8 installed -> Option C

4. Quick Check:

   conda create -n myenv python=3.8 = D [OK]

Hint: Remember: 'conda create -n' makes new isolated envs [OK]

Common Mistakes:

• Confusing 'create' with 'install' or 'update'
• Thinking it affects the global Python installation
• Misunderstanding the '-n' flag as package name

2. Which of the following commands correctly activates a conda environment named dataenv?

easy

A. activate conda dataenv

B. conda activate dataenv

C. conda start dataenv

D. source deactivate dataenv

Solution

1. Step 1: Recall the syntax to activate conda environments

   The correct command to activate an environment is conda activate <env_name>.

2. Step 2: Check each option

   conda activate dataenv matches the correct syntax. Options B, C, and D use incorrect command order or wrong commands.

3. Final Answer:

   conda activate dataenv -> Option B

4. Quick Check:

   Activate env = conda activate env_name [OK]

Hint: Use 'conda activate env_name' to switch environments [OK]

Common Mistakes:

• Using 'activate conda' instead of 'conda activate'
• Confusing 'source deactivate' with activation
• Trying 'conda start' which is invalid

3. Given the following commands run in order:

conda create -n testenv python=3.9 -y
conda activate testenv
pip install numpy
pip list | grep numpy

What will be the output of the last command?

medium

A. numpy with its installed version number

B. No output because pip list does not work inside conda

C. Error: 'pip' command not found

D. List of all packages except numpy

Solution

1. Step 1: Understand environment creation and activation

   The environment 'testenv' is created with Python 3.9 and then activated, so all commands run inside it.

2. Step 2: Installing numpy with pip inside the active environment

   Running pip install numpy installs numpy in 'testenv'. The pip list | grep numpy command will show numpy and its version.

3. Final Answer:

   numpy with its installed version number -> Option A

4. Quick Check:

   pip install inside active env = numpy listed [OK]

Hint: pip installs packages in active conda env, visible with pip list [OK]

Common Mistakes:

• Thinking pip installs globally ignoring conda env
• Assuming pip commands fail inside conda
• Expecting no output from pip list

4. You run conda activate myenv but get the error: CommandNotFoundError: Your shell has not been properly configured to use 'conda activate'. What is the most likely fix?

medium

A. Reinstall Python globally on your system

B. Deactivate any active environment before activating 'myenv'

C. Use pip install conda to fix the error

D. Run conda init to configure your shell, then restart the terminal

Solution

1. Step 1: Understand the error cause

   This error means the shell does not know how to run conda activate because it lacks proper initialization.

2. Step 2: Apply the fix by initializing conda for the shell

   Running conda init sets up the shell scripts needed. Restarting the terminal applies changes.

3. Final Answer:

   Run conda init to configure your shell, then restart the terminal -> Option D

4. Quick Check:

   Shell config for conda = conda init + restart [OK]

Hint: Run 'conda init' once after install, then restart terminal [OK]

Common Mistakes:

• Trying to reinstall Python instead of fixing shell config
• Using pip to install conda which is incorrect
• Ignoring the need to restart terminal after init

5. You want to create a reproducible environment for a project using conda and pip. Which sequence of commands correctly creates an environment, installs packages from a requirements.txt file using pip, and exports the environment including pip packages?

hard

A. conda create -n projenv python=3.10 -y && conda activate projenv && pip install -r requirements.txt && conda env export > environment.yml

B. conda create -n projenv python=3.10 -y && pip install -r requirements.txt && conda activate projenv && conda env export > environment.yml

C. conda create -n projenv python=3.10 -y && conda activate projenv && pip install -r requirements.txt && conda env export --from-history > environment.yml

D. conda activate projenv && conda create -n projenv python=3.10 -y && pip install -r requirements.txt && conda env export > environment.yml

Solution

1. Step 1: Create and activate the environment before installing packages

   You must first create the environment, then activate it to install packages inside it.

2. Step 2: Install pip packages and export full environment

   After activation, install packages from requirements.txt using pip. Then export the full environment including pip packages with conda env export.

3. Final Answer:

   conda create -n projenv python=3.10 -y && conda activate projenv && pip install -r requirements.txt && conda env export > environment.yml -> Option A

4. Quick Check:

   Create, activate, pip install, export full env = C [OK]

Hint: Activate env before pip install; export full env to include pip packages [OK]

Common Mistakes:

• Installing pip packages before activating environment
• Using --from-history which excludes pip packages
• Activating environment after installing packages