Tracking datasets with DVC in MLOps - Deep Dive

Overview - Tracking datasets with DVC

What is it?

Tracking datasets with DVC means using a tool to keep versions of your data files, just like you do with code. DVC helps you save snapshots of datasets so you can go back to any version anytime. It works alongside Git but handles large files efficiently without storing them directly in Git. This makes managing data in machine learning projects easier and more reliable.

Why it matters

Without dataset tracking, teams struggle to reproduce results or understand which data version was used for a model. Mistakes happen when data changes without record, causing wasted time and wrong conclusions. DVC solves this by making dataset versions clear and easy to switch between, improving collaboration and trust in machine learning work. It prevents data loss and confusion, saving effort and boosting productivity.

Where it fits

Before learning DVC dataset tracking, you should understand basic Git version control and why versioning matters. After mastering dataset tracking, you can learn about DVC pipelines for automating ML workflows and advanced data management techniques like remote storage and data sharing.

Mental Model

Core Idea

DVC tracks datasets by storing small pointers in Git while keeping large data files in separate storage, enabling efficient version control of data alongside code.

Think of it like...

Tracking datasets with DVC is like keeping a photo album index in your notebook while the actual photos are stored in a big photo box. The notebook tells you which photo is where without carrying all photos around.

┌─────────────┐       ┌───────────────┐
│   Git Repo  │──────▶│ DVC Pointer   │
│ (code +    │       │ (small file   │
│  metadata) │       │  with hash)   │
└─────────────┘       └───────────────┘
                           │
                           ▼
                    ┌───────────────┐
                    │ Data Storage  │
                    │ (large files) │
                    └───────────────┘

Build-Up - 7 Steps

1

Foundation

What is DVC and why use it

Concept: Introduce DVC as a tool for data versioning and its role in ML projects.

DVC stands for Data Version Control. It helps track changes in datasets and models, similar to how Git tracks code. Unlike Git, DVC handles large files efficiently by storing them outside the Git repository. This keeps your project lightweight and organized.

Result

Learner understands the purpose of DVC and why normal Git is not enough for datasets.

Knowing why DVC exists clarifies the need for specialized tools in data-heavy projects.

2

Foundation

Basic DVC setup and initialization

Concept: Learn how to start using DVC in a project with simple commands.

To start tracking data with DVC, first install DVC. Then run 'dvc init' inside your Git project to add DVC support. This creates configuration files and folders that DVC uses to manage data versions.

Result

Project is ready to track datasets with DVC alongside Git.

Understanding the setup process shows how DVC integrates smoothly with existing Git workflows.

3

Intermediate

Adding datasets to DVC tracking

🤔Before reading on: do you think DVC copies your data into Git or stores it separately? Commit to your answer.

Concept: Learn how to add data files to DVC tracking and what happens behind the scenes.

Use 'dvc add ' to tell DVC to track a dataset. DVC creates a small pointer file with a hash of the data and moves the actual data to a special cache folder. The pointer file is committed to Git, not the large data itself.

Result

Data is tracked by DVC with a pointer in Git and the actual file stored separately.

Knowing that DVC separates pointers from data explains how it keeps Git repos small and efficient.

4

Intermediate

Committing and pushing data versions

🤔Before reading on: do you think pushing data with DVC is the same as pushing code with Git? Commit to your answer.

Concept: Understand how to save data versions locally and share them using remote storage.

After 'dvc add', commit the pointer file with Git. To share data, configure a remote storage (like cloud or network drive) with 'dvc remote add'. Then run 'dvc push' to upload data files to remote storage. Others can get data with 'dvc pull'.

Result

Data versions are saved locally and can be shared or retrieved from remote storage.

Understanding separate data push/pull steps clarifies how DVC manages large files beyond Git.

5

Intermediate

Switching between dataset versions

Concept: Learn how to move between different dataset versions using Git and DVC.

Use Git commands like 'git checkout ' to switch code versions. DVC pointer files change accordingly. Then run 'dvc checkout' to update your local data files to match the pointer. This lets you reproduce past experiments exactly.

Result

You can easily switch datasets to any saved version matching your code.

Knowing how code and data versions sync prevents confusion and ensures reproducibility.

6

Advanced

Using DVC with remote storage backends

🤔Before reading on: do you think DVC supports only one type of remote storage? Commit to your answer.

Concept: Explore how DVC supports multiple remote storage options for data files.

DVC can use cloud services like AWS S3, Google Drive, Azure Blob, SSH servers, or local network drives as remote storage. You configure these remotes with 'dvc remote add' and set one as default. This flexibility lets teams choose storage that fits their needs and budget.

Result

Data can be stored and shared securely on various remote platforms.

Understanding remote storage options helps adapt DVC to different team environments and scales.

7

Expert

How DVC handles data integrity and caching

🤔Before reading on: do you think DVC stores multiple copies of the same data file if unchanged? Commit to your answer.

Concept: Learn about DVC's internal caching and hashing to avoid data duplication and ensure integrity.

DVC uses content hashing to identify data files uniquely. When you add data, DVC stores it in a cache folder named by its hash. If the same file is added again, DVC reuses the cached copy instead of duplicating. This saves space and guarantees data integrity by verifying hashes.

Result

DVC efficiently manages storage and prevents accidental data corruption.

Knowing DVC's caching mechanism reveals how it optimizes storage and maintains trust in data versions.

Under the Hood

DVC works by creating small metafiles that contain hashes of data files. These metafiles are tracked by Git. The actual data files are stored in a separate cache directory managed by DVC. When data changes, DVC computes a new hash and stores the new version in the cache. Remote storage can be configured to sync this cache across machines. Commands like 'dvc checkout' sync the workspace data files to match the current Git commit's pointers.

Why designed this way?

DVC was designed to overcome Git's limitations with large files and binary data. Storing large files directly in Git slows down operations and bloats repositories. By separating pointers and data, DVC keeps Git fast and lightweight. Hashing ensures data integrity and deduplication. Supporting multiple remote storages allows flexibility for different team needs and infrastructure.

┌───────────────┐       ┌───────────────┐       ┌───────────────┐
│   Workspace   │──────▶│  DVC Pointer  │──────▶│   Git Repo    │
│ (data files)  │       │ (small .dvc)  │       │ (code + meta) │
└───────────────┘       └───────────────┘       └───────────────┘
         │                      │                      │
         ▼                      ▼                      ▼
┌─────────────────────────────────────────────────────────┐
│                      DVC Cache                          │
│ (stores data files named by hash, deduplicated)        │
└─────────────────────────────────────────────────────────┘
         │
         ▼
┌─────────────────────────────┐
│      Remote Storage          │
│ (cloud, network, local disk)│
└─────────────────────────────┘

Myth Busters - 4 Common Misconceptions

Quick: Does DVC store your actual data files inside the Git repository? Commit yes or no.

Common Belief: DVC stores the full dataset inside the Git repository just like code files.

Reality: DVC only stores small pointer files in Git; the actual data files are stored separately in a cache and optionally remote storage.

Why it matters: Believing data is in Git leads to confusion about repo size and causes frustration when large files slow down Git operations.

Quick: Can you track data changes with DVC without using Git? Commit yes or no.

Common Belief: DVC can track datasets independently without Git involvement.

Reality: DVC relies on Git to track pointer files and versions; it is designed to complement Git, not replace it.

Why it matters: Trying to use DVC without Git breaks the versioning workflow and causes loss of synchronization between code and data.

Quick: Does DVC automatically upload your data to remote storage when you add it? Commit yes or no.

Common Belief: DVC uploads data to remote storage automatically as soon as you add it.

Reality: You must explicitly run 'dvc push' to upload data to remote storage; adding data only updates local cache and pointers.

Why it matters: Assuming automatic upload can cause missing data on collaborators' machines and failed reproductions.

Quick: If you add the same data file twice, does DVC store two copies? Commit yes or no.

Common Belief: DVC stores a new copy of the data file every time you add it, even if unchanged.

Reality: DVC uses hashing to detect duplicates and reuses cached data, avoiding multiple copies.

Why it matters: Misunderstanding this leads to inefficient storage use and confusion about disk space.

Expert Zone

1

DVC's cache is content-addressable, meaning files are stored by their hash, enabling deduplication and integrity checks.

2

Pointer files (.dvc) are lightweight and human-readable, allowing manual inspection and troubleshooting of data versions.

3

DVC supports multiple remotes and can prioritize them, enabling complex workflows like fallback storage or multi-cloud setups.

When NOT to use

DVC is not ideal for real-time streaming data or extremely large datasets that change continuously; specialized data lakes or databases are better. For simple small datasets, plain Git or cloud storage may suffice without DVC overhead.

Production Patterns

Teams use DVC to version datasets alongside code in Git repositories, automate data fetching in CI/CD pipelines, and share data via cloud remotes. It integrates with ML pipelines to ensure experiments are reproducible with exact data versions.

Connections

Git Version Control

DVC builds on Git by extending version control to large data files using pointers.

Understanding Git's limitations with large files clarifies why DVC's pointer-and-cache model is necessary.

Content-Addressable Storage

DVC uses content-addressable storage internally to identify and deduplicate data files by hash.

Knowing content-addressable storage principles explains how DVC efficiently manages data integrity and storage.

Supply Chain Management

Tracking datasets with DVC is like managing inventory in supply chains, where each batch is tracked by ID and location.

Seeing dataset versions as inventory batches helps understand the importance of traceability and reproducibility.

Common Pitfalls

#1 Adding data files but forgetting to commit pointer files to Git.

Wrong approach:

dvc add data.csv
# No git add or commit afterwards

Correct approach:

dvc add data.csv
git add data.csv.dvc
git commit -m "Track data.csv with DVC"

Root cause:

Misunderstanding that DVC pointer files must be committed to Git to record data version changes.

#2 Assuming data is pushed to remote storage automatically after adding.

Wrong approach:

dvc add data.csv
git commit -m "Add data"
# No dvc push command

Correct approach:

dvc add data.csv
git commit -m "Add data"
dvc push

Root cause:

Not knowing that 'dvc push' is required to upload data files to remote storage.

#3 Switching Git branches but not running 'dvc checkout' to update data files.

Wrong approach:

git checkout old-branch
# No dvc checkout command

Correct approach:

git checkout old-branch
dvc checkout

Root cause:

Not realizing that DVC pointer files change with Git branches and local data must be synced manually.

Key Takeaways

DVC tracks datasets by storing small pointer files in Git and keeping large data files in a separate cache and remote storage.

This separation keeps Git repositories lightweight and enables efficient version control of large data files.

You must explicitly add, commit, and push data and pointer files to manage dataset versions properly.

Switching dataset versions requires syncing both Git pointers and local data files using DVC commands.

Understanding DVC's caching and remote storage mechanisms is key to using it effectively in real-world machine learning projects.