Data pipelines with DVC in MLOps - Step-by-Step Execution

Process Flow - Data pipelines with DVC

Define stages in dvc.yaml

↓

Run dvc repro to execute pipeline

↓

DVC checks dependencies and outputs

↓

Execute commands for each stage

↓

Save outputs and update dvc.lock

↓

Track data and pipeline with git and dvc

↓

Repeat: modify data/code -> dvc repro -> track changes

This flow shows how DVC runs a data pipeline by defining stages, executing them, tracking outputs, and updating pipeline state.

Execution Sample

MLOps

stages:
  preprocess:
    cmd: python preprocess.py data/raw data/preprocessed
    deps:
      - data/raw
      - preprocess.py
    outs:
      - data/preprocessed

This dvc.yaml snippet defines a pipeline stage 'preprocess' that runs a Python script with input dependencies and output data.

Process Table

Step	Action	Stage	Dependencies Checked	Command Executed	Outputs Saved	dvc.lock Updated
1	Start pipeline run	-	-	-	-	-
2	Check 'preprocess' stage dependencies	preprocess	data/raw, preprocess.py	-	-	-
3	Run command	preprocess	-	python preprocess.py data/raw data/preprocessed	-	-
4	Save output data	preprocess	-	-	data/preprocessed	-
5	Update dvc.lock with new hashes	preprocess	-	-	-	Updated
6	Pipeline run complete	-	-	-	-	-

💡 All stages executed successfully, outputs saved, and dvc.lock updated to reflect current pipeline state.

Status Tracker

Variable	Start	After Step 2	After Step 3	After Step 4	After Step 5	Final
dependencies_checked	None	['data/raw', 'preprocess.py']	['data/raw', 'preprocess.py']	['data/raw', 'preprocess.py']	['data/raw', 'preprocess.py']	['data/raw', 'preprocess.py']
command_status	Not run	Not run	Running	Completed	Completed	Completed
outputs_saved	None	None	None	data/preprocessed	data/preprocessed	data/preprocessed
dvc_lock_status	Old	Old	Old	Old	Updated	Updated

Key Moments - 3 Insights

Why does DVC check dependencies before running a stage?

DVC checks dependencies (see Step 2 in execution_table) to decide if the stage needs to run. If dependencies haven't changed, DVC can skip running the stage to save time.

What happens if the output data already exists and dependencies are unchanged?

DVC will skip running the command and keep the existing outputs and dvc.lock unchanged, avoiding unnecessary work. This is shown by the dependency check and output save steps.

Why is dvc.lock updated after running a stage?

dvc.lock records exact hashes of dependencies and outputs after execution (Step 5). This helps DVC track changes and decide if future runs need to re-execute stages.

Visual Quiz - 3 Questions

Look at the execution_table, at which step does DVC run the actual command for the 'preprocess' stage?

A Step 2

B Step 4

C Step 3

D Step 5

💡 Hint

Check the 'Command Executed' column in execution_table rows.

According to variable_tracker, what is the status of 'dvc_lock_status' after Step 4?

A Updated

B Old

C Not created

D Deleted

💡 Hint

Look at the 'dvc_lock_status' row and the column 'After Step 4'.

If the dependencies change, what will DVC do differently in the execution flow?

A Run the stage command again

B Skip running the stage

C Delete outputs without running

D Ignore changes and keep old outputs

💡 Hint

Refer to the key moment about dependency checking and stage execution.

Concept Snapshot

DVC pipelines are defined in dvc.yaml with stages specifying commands, dependencies, and outputs.
Run 'dvc repro' to execute stages if dependencies changed.
DVC tracks outputs and updates dvc.lock with hashes.
This avoids rerunning unchanged stages, saving time.
Use git to version control pipeline files and data pointers.

Full Transcript

This visual execution shows how DVC manages data pipelines. First, you define stages in dvc.yaml with commands, dependencies, and outputs. When you run 'dvc repro', DVC checks if dependencies changed. If yes, it runs the stage command, saves outputs, and updates dvc.lock with new hashes. If no changes, it skips running to save time. This process helps track data and code changes efficiently in machine learning projects.