DVC (Data Version Control) basics in MLOps - Step-by-Step Execution

Process Flow - DVC (Data Version Control) basics

Initialize Git repo

↓

Initialize DVC in repo

↓

Add data files to DVC tracking

↓

Commit changes to Git

↓

Push data and code to remote storage

↓

Pull data and code from remote when needed

This flow shows how you start with a Git repo, add DVC to track data files, commit changes, and push or pull data versions.

Execution Sample

MLOps

git init

dvc init

dvc add data.csv

git add .
git commit -m "Add data with DVC"
dvc push

This code initializes Git and DVC, tracks a data file with DVC, commits changes, and pushes data to remote storage.

Process Table

Step	Command	Action	Result
1	git init	Create new Git repository	Empty Git repo created
2	dvc init	Initialize DVC in repo	.dvc folder and config created
3	dvc add data.csv	Track data.csv with DVC	data.csv.dvc file created, data tracked
4	git add .	Stage all changes for Git	All changes staged
5	git commit -m "Add data with DVC"	Commit changes to Git	Commit saved with message
6	dvc push	Upload data files to remote storage	Data files uploaded to remote
7		End of process	Data and code versioned and stored

💡 Process stops after data and code are versioned and pushed to remote storage

Status Tracker

Variable	Start	After Step 2	After Step 3	After Step 5	After Step 6
Git repo	None	Initialized	Initialized	Commit with DVC files	Commit with DVC files
DVC config	None	Created	Created	Created	Created
Tracked data	None	None	data.csv tracked	data.csv tracked	data.csv pushed to remote

Key Moments - 3 Insights

Why do we need both Git and DVC commands?

Git tracks code and small files, while DVC tracks large data files separately. Execution table rows 1-2 show Git and DVC initialization, and rows 3-6 show how both work together.

What does 'dvc add' actually do to the data file?

'dvc add' does not change the data file but creates a .dvc file that tracks its version. See execution table row 3 where data.csv.dvc is created.

Why do we run 'git commit' after 'dvc add'?

Because DVC creates tracking files that Git needs to save. Without committing, Git won't remember the data version info. This is shown in rows 4 and 5.

Visual Quiz - 3 Questions

Look at the execution table, what is the result after running 'dvc init'?

A Empty Git repository created

B data.csv.dvc file created

C .dvc folder and config created

D Data files uploaded to remote

💡 Hint

Check row 2 under Result column in the execution table

At which step is the data file actually tracked by DVC?

A Step 1

B Step 3

C Step 2

D Step 6

💡 Hint

Look at the Action column for 'dvc add data.csv' in the execution table

If you skip 'git commit' after 'dvc add', what will happen?

A Git won't save the DVC tracking files

B Data files won't be tracked by DVC

C Data files won't upload to remote

D DVC will fail to initialize

💡 Hint

Refer to key moment about why 'git commit' is needed after 'dvc add'

Concept Snapshot

DVC basics:
- Use 'git init' to start Git repo
- Use 'dvc init' to add DVC
- Use 'dvc add <file>' to track data files
- Commit changes with Git to save DVC tracking
- Use 'dvc push' to upload data to remote
- Use 'dvc pull' to retrieve data versions

Full Transcript

This lesson shows how to use DVC with Git to version control data files. First, you create a Git repository with 'git init'. Then, you add DVC support using 'dvc init'. Next, you track a data file using 'dvc add data.csv', which creates a tracking file but does not change the data itself. After that, you stage and commit all changes with Git commands 'git add .' and 'git commit'. Finally, you upload the data files to remote storage using 'dvc push'. This process helps keep your data and code versions in sync and easy to share or reproduce.

Practice

1. What is the main purpose of using dvc add in a project?

easy

A. To push code changes to a remote Git server

B. To initialize a new Git repository

C. To start tracking a data file or directory with DVC

D. To remove data files from the project

Solution

1. Step 1: Understand the role of dvc add

   dvc add is used to tell DVC to track a data file or directory, creating a pointer file in Git.

2. Step 2: Differentiate from other commands

   Commands like dvc init start DVC, while dvc push syncs data remotely. dvc add specifically tracks data.

3. Final Answer:

   To start tracking a data file or directory with DVC -> Option C

4. Quick Check:

   dvc add tracks data files [OK]

Hint: Remember: add means track data files with DVC [OK]

Common Mistakes:

• Confusing dvc add with dvc init
• Thinking dvc add pushes data remotely
• Assuming dvc add initializes Git

2. Which command correctly initializes DVC in an existing Git repository?

easy

A. dvc start

B. dvc init

C. git dvc init

D. dvc create

Solution

1. Step 1: Identify the DVC initialization command

   The correct command to initialize DVC in a Git repo is dvc init.

2. Step 2: Eliminate incorrect options

   dvc start and dvc create are not valid DVC commands. git dvc init is invalid syntax.

3. Final Answer:

   dvc init -> Option B

4. Quick Check:

   DVC init command = dvc init [OK]

Hint: Use dvc init to start DVC in your repo [OK]

Common Mistakes:

• Typing dvc start instead of dvc init
• Prefixing with git incorrectly
• Using non-existent commands like dvc create

3. Given the following commands run in order:

git init
 dvc init
 dvc add data.csv
 git add data.csv.dvc
 git commit -m "Add data"
 dvc push

What happens after dvc push is executed?

medium

A. The data file is deleted locally after upload

B. Only the data.csv.dvc pointer file is pushed to Git remote

C. The Git repository is cloned to remote storage

D. The actual data file data.csv is uploaded to remote storage

Solution

1. Step 1: Understand dvc push behavior

   dvc push uploads the actual large data files tracked by DVC to the configured remote storage, not just Git files.

2. Step 2: Differentiate Git and DVC storage roles

   Git stores small pointer files like data.csv.dvc, while DVC manages big data files separately in remote storage.

3. Final Answer:

   The actual data file data.csv is uploaded to remote storage -> Option D

4. Quick Check:

   dvc push uploads data files remotely [OK]

Hint: dvc push uploads big data files, not just pointers [OK]

Common Mistakes:

• Thinking dvc push only pushes Git files
• Confusing dvc push with git push
• Assuming data files are deleted after push

4. You ran dvc add dataset.csv but forgot to commit the generated dataset.csv.dvc file to Git. What problem will you face?

medium

A. DVC will not track the data file until the pointer file is committed

B. The data file will be deleted automatically

C. Git will track the data file instead of DVC

D. No problem; DVC tracks data without Git commits

Solution

1. Step 1: Understand the role of the .dvc pointer file

   The dataset.csv.dvc file is a small pointer tracked by Git that tells DVC about the data file version.

2. Step 2: Consequence of not committing the pointer file

   If you don't commit this pointer file, Git and collaborators won't know about the data version, so DVC tracking is incomplete.

3. Final Answer:

   DVC will not track the data file until the pointer file is committed -> Option A

4. Quick Check:

   Pointer file commit = DVC tracking active [OK]

Hint: Always commit .dvc pointer files after dvc add [OK]

Common Mistakes:

• Assuming data files are tracked without pointer commits
• Thinking data files get deleted automatically
• Believing Git tracks large data files directly

5. You have a large dataset tracked by DVC and a remote storage configured. Your teammate cloned the Git repo but the data files are missing locally. Which command should they run to get the data files?

hard

A. dvc pull

B. dvc add

C. git pull

D. git clone

Solution

1. Step 1: Understand what dvc pull does

   dvc pull downloads the actual data files from remote storage to the local machine based on the pointer files in Git.

2. Step 2: Differentiate from Git commands

   git pull updates code and pointer files but does not fetch large data files. dvc add tracks new data, and git clone clones the repo initially.

3. Final Answer:

   dvc pull -> Option A

4. Quick Check:

   Use dvc pull to fetch data files locally [OK]

Hint: Use dvc pull to download data after cloning repo [OK]

Common Mistakes:

• Running only git pull expecting data files
• Trying dvc add to get data files
• Confusing git clone with data download