Kubeflow Pipelines overview in MLOps - Commands & Configuration

Introduction

Kubeflow Pipelines help you automate and manage machine learning workflows. It solves the problem of running complex ML tasks step-by-step without manual work.

When you want to run a series of ML tasks like data preprocessing, training, and evaluation automatically.

When you need to track and compare different ML model versions easily.

When you want to share your ML workflow with teammates in a clear, repeatable way.

When you want to run ML workflows on Kubernetes without writing complex scripts.

When you want to visualize the progress and results of your ML experiments.

Commands

This command compiles your Python pipeline code into a YAML file that Kubeflow Pipelines can understand and run.

Terminal

dsl-compile --py my_pipeline.py --output my_pipeline.yaml

Expected Output

Successfully compiled pipeline to my_pipeline.yaml

→

--py - Specifies the Python file containing the pipeline definition

→

--output - Specifies the output YAML file name

This command submits the compiled pipeline YAML to Kubeflow Pipelines to start a new run named 'example-run'.

Terminal

kfp run submit --pipeline-file my_pipeline.yaml --run-name example-run

Expected Output

Run 'example-run' submitted successfully with run ID: 12345

→

--pipeline-file - Specifies the pipeline YAML file to run

→

--run-name - Names the pipeline run for easy identification

This command lists all the pipeline runs you have submitted, showing their status and run IDs.

Terminal

kfp run list

Expected Output

RUN ID   RUN NAME      STATUS    CREATED

12345    example-run   Running   2024-06-01 10:00:00

This command shows detailed information about the pipeline run with ID 12345, including steps and logs.

Terminal

kfp run describe 12345

Expected Output

Run ID: 12345

Name: example-run

Status: Running

Steps:

 - preprocess: Completed

 - train: Running

 - evaluate: Pending

Key Concept

If you remember nothing else from Kubeflow Pipelines, remember: it automates and tracks your ML workflows step-by-step on Kubernetes.

Code Example

MLOps

from kfp import dsl

@dsl.pipeline(name='Simple Pipeline')
def simple_pipeline():
    op1 = dsl.ContainerOp(
        name='echo',
        image='alpine',
        command=['echo', 'Hello Kubeflow Pipelines!']
    )

if __name__ == '__main__':
    import kfp.compiler as compiler
    compiler.Compiler().compile(simple_pipeline, 'simple_pipeline.yaml')
    print('Pipeline compiled to simple_pipeline.yaml')

Common Mistakes

Trying to run the Python pipeline file directly without compiling it to YAML first.

Kubeflow Pipelines requires the pipeline in YAML format to run, not raw Python code.

Always compile your Python pipeline code to YAML using 'dsl-compile' before submitting.

Submitting a pipeline run without specifying a unique run name.

Without a unique name, it is hard to identify and track your runs later.

Use the '--run-name' flag to give each run a clear, unique name.

Not checking the run status after submission.

You might miss errors or incomplete steps if you don't monitor the run.

Use 'kfp run list' and 'kfp run describe' to monitor your pipeline runs.

Summary

Compile your Python pipeline code into a YAML file using 'dsl-compile' or the Python SDK.

Submit the compiled pipeline YAML to Kubeflow Pipelines to start a run.

Use commands to list and describe runs to monitor progress and results.