Tracking datasets with DVC in MLOps - Commands & Configuration

Introduction

When working on data projects, it is hard to keep track of different versions of datasets. DVC helps by saving snapshots of your data files so you can easily go back or share them without copying large files manually.

When you want to save a version of your dataset before making changes.

When you need to share your dataset versions with teammates without sending big files.

When you want to reproduce a machine learning experiment with the exact same data.

When you want to track changes in your data alongside your code.

When you want to store datasets in remote storage but keep lightweight pointers in your project.

Commands

This command initializes DVC in your project folder by creating necessary configuration files and folders to start tracking data.

Terminal

dvc init

Expected Output

Initialized DVC repository.

You can now track data files with 'dvc add'.

This command tells DVC to track the dataset file located at data/dataset.csv. It creates a small pointer file and stores the actual data in DVC's cache.

Terminal

dvc add data/dataset.csv

Expected Output

Adding 'data/dataset.csv' to DVC tracking.

Saving information to 'data/dataset.csv.dvc'.

This command adds the DVC pointer file and updated .gitignore to Git so you can version control the dataset reference, not the data itself.

Terminal

git add data/dataset.csv.dvc .gitignore

Expected Output

No output (command runs silently)

This commits the changes to Git, saving the dataset pointer and ignore rules so your project history includes the dataset version.

Terminal

git commit -m "Track dataset with DVC"

Expected Output

[main abc1234] Track dataset with DVC

 2 files changed, 10 insertions(+)

This uploads the actual dataset file to the remote storage configured in DVC, so others can download it later without storing it in Git.

Terminal

dvc push

Expected Output

Uploading data/dataset.csv to remote storage.

Upload complete.

Key Concept

If you remember nothing else from this pattern, remember: DVC tracks data by saving small pointer files in Git and storing large data files separately.

Common Mistakes

Adding large data files directly to Git instead of using 'dvc add'.

Git is not designed to handle large files efficiently, which slows down your project and bloats the repository.

Use 'dvc add' to track large data files and commit only the small .dvc pointer files to Git.

Forgetting to run 'dvc push' after adding data.

Without pushing, the actual data is only stored locally and others cannot access it from remote storage.

Always run 'dvc push' to upload data files to the remote storage after adding or updating datasets.

Not committing the .dvc files to Git.

Without committing .dvc files, the dataset versions are not tracked in your project history.

Commit the .dvc pointer files and updated .gitignore to Git after running 'dvc add'.

Summary

Initialize DVC in your project with 'dvc init' to start tracking data.

Use 'dvc add' to track large dataset files and create pointer files.

Commit the pointer files to Git to version control dataset references.

Push the actual data to remote storage with 'dvc push' for sharing and backup.