Kubeflow Pipelines overview in MLOps - Commands & Configuration

Introduction

Kubeflow Pipelines help you automate and manage machine learning workflows. It solves the problem of running complex ML tasks step-by-step without manual work.

When you want to run a series of ML tasks like data preprocessing, training, and evaluation automatically.

When you need to track and compare different ML model versions easily.

When you want to share your ML workflow with teammates in a clear, repeatable way.

When you want to run ML workflows on Kubernetes without writing complex scripts.

When you want to visualize the progress and results of your ML experiments.

Commands

This command compiles your Python pipeline code into a YAML file that Kubeflow Pipelines can understand and run.

Terminal

dsl-compile --py my_pipeline.py --output my_pipeline.yaml

Expected Output

Successfully compiled pipeline to my_pipeline.yaml

→

--py - Specifies the Python file containing the pipeline definition

→

--output - Specifies the output YAML file name

This command submits the compiled pipeline YAML to Kubeflow Pipelines to start a new run named 'example-run'.

Terminal

kfp run submit --pipeline-file my_pipeline.yaml --run-name example-run

Expected Output

Run 'example-run' submitted successfully with run ID: 12345

→

--pipeline-file - Specifies the pipeline YAML file to run

→

--run-name - Names the pipeline run for easy identification

This command lists all the pipeline runs you have submitted, showing their status and run IDs.

Terminal

kfp run list

Expected Output

RUN ID   RUN NAME      STATUS    CREATED

12345    example-run   Running   2024-06-01 10:00:00

This command shows detailed information about the pipeline run with ID 12345, including steps and logs.

Terminal

kfp run describe 12345

Expected Output

Run ID: 12345

Name: example-run

Status: Running

Steps:

 - preprocess: Completed

 - train: Running

 - evaluate: Pending

Key Concept

If you remember nothing else from Kubeflow Pipelines, remember: it automates and tracks your ML workflows step-by-step on Kubernetes.

Code Example

MLOps

from kfp import dsl

@dsl.pipeline(name='Simple Pipeline')
def simple_pipeline():
    op1 = dsl.ContainerOp(
        name='echo',
        image='alpine',
        command=['echo', 'Hello Kubeflow Pipelines!']
    )

if __name__ == '__main__':
    import kfp.compiler as compiler
    compiler.Compiler().compile(simple_pipeline, 'simple_pipeline.yaml')
    print('Pipeline compiled to simple_pipeline.yaml')

Common Mistakes

Trying to run the Python pipeline file directly without compiling it to YAML first.

Kubeflow Pipelines requires the pipeline in YAML format to run, not raw Python code.

Always compile your Python pipeline code to YAML using 'dsl-compile' before submitting.

Submitting a pipeline run without specifying a unique run name.

Without a unique name, it is hard to identify and track your runs later.

Use the '--run-name' flag to give each run a clear, unique name.

Not checking the run status after submission.

You might miss errors or incomplete steps if you don't monitor the run.

Use 'kfp run list' and 'kfp run describe' to monitor your pipeline runs.

Summary

Compile your Python pipeline code into a YAML file using 'dsl-compile' or the Python SDK.

Submit the compiled pipeline YAML to Kubeflow Pipelines to start a run.

Use commands to list and describe runs to monitor progress and results.

Practice

1. What is the main purpose of Kubeflow Pipelines in machine learning workflows?

easy

A. To store large datasets for training

B. To create user interfaces for ML models

C. To automate and manage ML workflows with clear, reusable steps

D. To replace Kubernetes as a container platform

Solution

1. Step 1: Understand Kubeflow Pipelines' role

   Kubeflow Pipelines are designed to automate ML workflows by defining clear steps that can be reused and tracked.

2. Step 2: Compare options with this role

   Options describing UI creation, data storage, and replacing Kubernetes do not match this role.

3. Final Answer:

   To automate and manage ML workflows with clear, reusable steps -> Option C

4. Quick Check:

   Automation of ML workflows [OK]

Hint: Kubeflow Pipelines automate ML steps, not UI or storage [OK]

Common Mistakes:

• Confusing Kubeflow Pipelines with data storage tools
• Thinking Kubeflow replaces Kubernetes
• Assuming it builds user interfaces

2. Which of the following is the correct way to define a step in a Kubeflow Pipeline using Python?

easy

A. def step():
return dsl.ContainerOp(name='step', image='python:3.8')

B. def step():
return dsl.ContainerOp(image='python:3.8')

C. def step():
return dsl.ContainerOp(name='step')

D. def step():
return dsl.ContainerOp(name='step', image='python:3.8', command=['python', 'script.py'])

Solution

1. Step 1: Understand ContainerOp usage

   ContainerOp requires at least a name, image, and usually a command to run the step's container.

2. Step 2: Check each option

   def step(): return dsl.ContainerOp(name='step', image='python:3.8', command=['python', 'script.py']) correctly includes name, image, and command. The version with name and image misses command. The version with only image misses name. The version with only name misses image and command.

3. Final Answer:

   def step():\n return dsl.ContainerOp(name='step', image='python:3.8', command=['python', 'script.py']) -> Option D

4. Quick Check:

   ContainerOp needs name, image, and command [OK]

Hint: ContainerOp needs name, image, and command to run [OK]

Common Mistakes:

• Omitting the command argument
• Not specifying the container image
• Missing the step name

3. Given this Kubeflow Pipeline snippet, what will be the output when the pipeline runs?

from kfp import dsl

@dsl.pipeline(name='Sample Pipeline')
def sample_pipeline():
    step1 = dsl.ContainerOp(
        name='echo-step',
        image='alpine',
        command=['echo', 'Hello Kubeflow']
    )

medium

A. The pipeline prints 'Hello Kubeflow' in the step logs

B. The pipeline fails because 'echo' is not a valid command

C. The pipeline prints 'Hello Kubeflow' on the console where pipeline is defined

D. The pipeline does nothing because no output is defined

Solution

1. Step 1: Understand ContainerOp execution

   The ContainerOp runs a container with the alpine image and executes the command 'echo Hello Kubeflow'. This prints to the container's standard output logs.

2. Step 2: Identify where output appears

   The output appears in the step logs inside Kubeflow Pipelines UI, not on the local console or nowhere.

3. Final Answer:

   The pipeline prints 'Hello Kubeflow' in the step logs -> Option A

4. Quick Check:

   ContainerOp command output = step logs [OK]

Hint: Container output appears in step logs, not local console [OK]

Common Mistakes:

• Thinking output appears on local console
• Assuming 'echo' command is invalid in alpine
• Believing pipeline does nothing without explicit output

4. You wrote this Kubeflow Pipeline step but it fails to run:

def step():
    return dsl.ContainerOp(name='step', image='python:3.8')

What is the most likely cause of the failure?

medium

A. Missing the command argument to specify what to run inside the container

B. The image 'python:3.8' does not exist

C. The step name cannot be 'step'

D. ContainerOp requires a volume to run

Solution

1. Step 1: Check ContainerOp requirements

   ContainerOp needs a command to run inside the container; without it, the container starts and exits immediately.

2. Step 2: Validate other options

   Image 'python:3.8' exists on Docker Hub, step name can be any string, and volume is optional.

3. Final Answer:

   Missing the command argument to specify what to run inside the container -> Option A

4. Quick Check:

   ContainerOp needs command to run [OK]

Hint: Always specify command in ContainerOp to avoid immediate exit [OK]

Common Mistakes:

• Assuming image is missing or invalid
• Thinking step name is restricted
• Believing volume is mandatory

5. You want to create a Kubeflow Pipeline that runs two steps sequentially: first preprocess data, then train a model. Which code snippet correctly defines this dependency?

hard

A. step1 = dsl.ContainerOp(name='preprocess', image='python:3.8', command=['python', 'preprocess.py'])
step2 = dsl.ContainerOp(name='train', image='python:3.8', command=['python', 'train.py'])
step1.after(step2)

B. step1 = dsl.ContainerOp(name='preprocess', image='python:3.8', command=['python', 'preprocess.py'])
step2 = dsl.ContainerOp(name='train', image='python:3.8', command=['python', 'train.py'])
step2.after(step1)

C. step1 = dsl.ContainerOp(name='preprocess', image='python:3.8', command=['python', 'preprocess.py'])
step2 = dsl.ContainerOp(name='train', image='python:3.8', command=['python', 'train.py'])
step1.before(step2)

D. step1 = dsl.ContainerOp(name='preprocess', image='python:3.8', command=['python', 'preprocess.py'])
step2 = dsl.ContainerOp(name='train', image='python:3.8', command=['python', 'train.py'])

Solution

1. Step 1: Understand step dependencies in Kubeflow Pipelines

   To run step2 after step1, use step2.after(step1) to set the order.

2. Step 2: Analyze each option

   step1 = dsl.ContainerOp(name='preprocess', image='python:3.8', command=['python', 'preprocess.py']) step2 = dsl.ContainerOp(name='train', image='python:3.8', command=['python', 'train.py']) step2.after(step1) correctly sets step2 to run after step1. Using step1.after(step2) reverses the order. Using step1.before(step2) calls a non-existent method. No dependency causes parallel execution.

3. Final Answer:

   step2.after(step1) -> Option B

4. Quick Check:

   Use step2.after(step1) for sequential steps [OK]

Hint: Use step2.after(step1) to run steps sequentially [OK]

Common Mistakes:

• Reversing the order with after()
• Using before() which does not exist
• Not setting any dependency causing parallel runs