Data pipelines with DVC in MLOps - Practice Problems & Coding Challenges

Challenge - 5 Problems

💻 Command Output

intermediate

DVC Pipeline Stage Output

You run the command dvc run -n preprocess -d data/raw.csv -o data/processed.csv python preprocess.py. What will be the output of dvc pipeline show immediately after?

A preprocess

B No stages found

C preprocess -> train

D train

💡 Hint

Think about what stages are created after running dvc run.

The command creates a stage named 'preprocess'. Running 'dvc pipeline show' lists existing stages. Since only 'preprocess' is created, it shows just that.

❓ Configuration

intermediate

Correct DVC Stage Definition in dvc.yaml

Which of the following dvc.yaml stage definitions correctly specifies a stage named 'train' that depends on 'data/processed.csv' and runs 'python train.py' producing 'model.pkl'?

A

stages:
  train:
    cmd: python train.py
    deps:
      - data/processed.csv
    outs:
      - model.pkl

B

stages:
  train:
    cmd: python train.py
    outs:
      - data/processed.csv
    deps:
      - model.pkl

C

stages:
  train:
    cmd: python train.py
    deps:
      - model.pkl
    outs:
      - data/processed.csv

D

stages:
  train:
    cmd: python train.py
    deps:
      - model.pkl
    outs:
      - model.pkl

💡 Hint

Remember dependencies are inputs, outputs are results.

Option A correctly lists 'data/processed.csv' as a dependency and 'model.pkl' as an output. The command is correct.

❓ Troubleshoot

advanced

DVC Pipeline Reproduction Issue

You modified 'preprocess.py' but running dvc repro does not rerun the 'preprocess' stage. What is the most likely cause?

A The output file 'data/processed.csv' was deleted manually.

B The 'preprocess.py' file is not listed as a dependency in the dvc.yaml stage.

C The DVC cache is full and needs cleaning.

D The 'dvc.lock' file is missing.

💡 Hint

DVC tracks changes only in declared dependencies.

If 'preprocess.py' is not declared as a dependency, DVC does not detect changes in it and skips rerunning the stage.

🔀 Workflow

advanced

DVC Pipeline Stage Execution Order

Given a pipeline with stages: 'download' -> 'preprocess' -> 'train', which command will reproduce only the 'train' stage and all its dependencies?

A dvc repro

B dvc repro preprocess

C dvc repro download

D dvc repro train

💡 Hint

Reproducing a stage also reproduces its dependencies.

Running 'dvc repro train' will reproduce 'train' and all stages it depends on, including 'preprocess' and 'download'.

✅ Best Practice

expert

Best Practice for Large Data Files in DVC Pipelines

You have very large raw data files that rarely change but are needed for multiple pipeline runs. What is the best practice to manage these files with DVC?

A Commit the large raw data files directly to Git for easy access.

B Exclude the large raw data files from DVC and Git, and manually copy them before each run.

C Track the large raw data files with DVC and store them in remote storage, avoiding committing them to Git.

D Compress the large raw data files and commit the compressed files to Git.

💡 Hint

Think about versioning large files efficiently without bloating Git.

DVC is designed to track large files efficiently by storing them in remote storage and keeping lightweight pointers in Git. This avoids bloating Git repositories.

Practice

1. What is the main purpose of using dvc repro in a DVC pipeline?

easy

A. To delete all pipeline data and cache

B. To initialize a new DVC repository

C. To reproduce pipeline stages and update outputs if inputs changed

D. To manually edit pipeline stage commands

Solution

1. Step 1: Understand the role of dvc repro

   This command checks if any inputs or dependencies of pipeline stages have changed.

2. Step 2: Effect of running dvc repro

   If changes are detected, it reruns the affected stages to update outputs accordingly.

3. Final Answer:

   To reproduce pipeline stages and update outputs if inputs changed -> Option C

4. Quick Check:

   dvc repro updates pipeline outputs [OK]

Hint: Remember: repro means rerun changed pipeline parts [OK]

Common Mistakes:

• Confusing repro with initialization commands
• Thinking repro deletes data
• Assuming repro edits pipeline commands

2. Which of the following is the correct syntax to add a pipeline stage with DVC that runs python train.py and outputs model.pkl?

easy

A. dvc stage add -n train -o model.pkl python train.py

B. dvc add stage train -o model.pkl python train.py

C. dvc run -n train -o model.pkl python train.py

D. dvc stage add -n train -d train.py -o model.pkl python train.py

Solution

1. Step 1: Identify required flags for stage creation

   The dvc stage add command requires -n for name, -d for dependencies, and -o for outputs.

2. Step 2: Check which option includes all required flags correctly

   dvc stage add -n train -d train.py -o model.pkl python train.py uses -n train, -d train.py (dependency), and -o model.pkl with the command python train.py.

3. Final Answer:

   dvc stage add -n train -d train.py -o model.pkl python train.py -> Option D

4. Quick Check:

   Stage add needs name, dependency, output flags [OK]

Hint: Stage add needs -n (name), -d (deps), -o (outputs) [OK]

Common Mistakes:

• Omitting the dependency with -d
• Using deprecated dvc run instead of stage add
• Mixing order of flags incorrectly

3. Given this DVC pipeline stage definition in dvc.yaml:

stages:
  preprocess:
    cmd: python preprocess.py data/raw data/processed
    deps:
      - data/raw
      - preprocess.py
    outs:
      - data/processed

What happens when you run dvc repro after modifying data/raw?

medium

A. The preprocess stage reruns and updates data/processed

B. Nothing happens because only preprocess.py changes trigger rerun

C. The pipeline fails due to missing output specification

D. All pipeline stages rerun regardless of changes

Solution

1. Step 1: Identify dependencies of the preprocess stage

   The stage depends on data/raw and preprocess.py.

2. Step 2: Effect of changing data/raw on dvc repro

   Changing a dependency triggers rerun of that stage to update outputs.

3. Final Answer:

   The preprocess stage reruns and updates data/processed -> Option A

4. Quick Check:

   Changed input triggers stage rerun [OK]

Hint: Change in deps triggers rerun of that stage [OK]

Common Mistakes:

• Assuming no rerun if only data changes
• Thinking all stages rerun always
• Confusing outputs with dependencies

4. You run dvc repro but get an error: ERROR: failed to reproduce stage 'train': missing dependency 'data/train.csv'. What is the most likely cause?

medium

A. The file data/train.csv was deleted or moved after pipeline creation

B. The dvc.yaml file is missing the train stage

C. The dvc.lock file is corrupted

D. You forgot to run dvc init before dvc repro

Solution

1. Step 1: Understand the error message

   The error says a dependency file is missing, which means DVC cannot find data/train.csv.

2. Step 2: Common causes of missing dependency errors

   Usually, the file was deleted, renamed, or moved after the pipeline stage was created.

3. Final Answer:

   The file data/train.csv was deleted or moved after pipeline creation -> Option A

4. Quick Check:

   Missing dependency file causes repro error [OK]

Hint: Check if all dependency files exist before repro [OK]

Common Mistakes:

• Assuming dvc.yaml missing stage causes this error
• Blaming dvc.lock corruption without evidence
• Forgetting to initialize repo before repro

5. You want to create a DVC pipeline with two stages: extract that outputs data/raw.csv, and train that depends on data/raw.csv and outputs model.pkl. Which sequence of commands correctly sets up this pipeline?

hard

A. dvc stage add -n train -o model.pkl python train.py

dvc stage add -n extract -d data/raw.csv -o data/raw.csv python extract.py

B. dvc stage add -n extract -o data/raw.csv python extract.py

dvc stage add -n train -d data/raw.csv -o model.pkl python train.py

C. dvc run -n extract -o data/raw.csv python extract.py

dvc run -n train -d data/raw.csv -o model.pkl python train.py

D. dvc add data/raw.csv
dvc add model.pkl

Solution

1. Step 1: Define extract stage with output only

   Extract stage produces data/raw.csv so it needs -n extract and -o data/raw.csv with the command.

2. Step 2: Define train stage depending on extract output

   Train stage depends on data/raw.csv so it needs -d data/raw.csv, outputs model.pkl, and runs python train.py.

3. Step 3: Confirm correct order and commands

   dvc stage add -n extract -o data/raw.csv python extract.py dvc stage add -n train -d data/raw.csv -o model.pkl python train.py correctly adds extract first, then train with proper dependencies and outputs.

4. Final Answer:

   dvc stage add -n extract -o data/raw.csv python extract.py dvc stage add -n train -d data/raw.csv -o model.pkl python train.py -> Option B

5. Quick Check:

   Define stages with correct deps and outputs [OK]

Hint: Add extract stage first, then train with dependency on extract output [OK]

Common Mistakes:

• Adding train stage before extract output exists
• Using dvc add instead of stage add for pipeline steps
• Missing dependencies in train stage