Kubeflow Pipelines overview in MLOps - Mini Project: Build & Apply

Kubeflow Pipelines Overview

📖 Scenario: You are working as a data scientist who wants to automate a simple machine learning workflow using Kubeflow Pipelines. This will help you run your steps in order and track results easily.

🎯 Goal: Build a basic Kubeflow Pipeline with three steps: data preparation, model training, and model evaluation. You will define each step as a Python function, create pipeline components, and then assemble them into a pipeline.

📋 What You'll Learn

Create Python functions for each pipeline step

Convert functions to Kubeflow pipeline components

Define a pipeline that connects the components in order

Compile and run the pipeline to see the output

💡 Why This Matters

🌍 Real World

Kubeflow Pipelines help automate machine learning workflows so data scientists can run experiments easily and track results.

💼 Career

Understanding Kubeflow Pipelines is important for MLOps engineers and data scientists working on production ML systems.

1

Create Python functions for pipeline steps

Write three Python functions named prepare_data, train_model, and evaluate_model. Each function should print a simple message: "Preparing data...", "Training model...", and "Evaluating model..." respectively.

MLOps

# Define the three functions below
# Your code here

Need a hint?

Define each function with def and use print() inside to show the step message.

2

Convert functions to Kubeflow pipeline components

Import kfp.components.create_component_from_func and create components named prepare_data_op, train_model_op, and evaluate_model_op from the functions prepare_data, train_model, and evaluate_model respectively.

MLOps

def prepare_data():
    print("Preparing data...")

def train_model():
    print("Training model...")

def evaluate_model():
    print("Evaluating model...")

# Import create_component_from_func and create components
# Your code here

Need a hint?

Use create_component_from_func(function_name) to convert each function into a pipeline component.

3

Define the Kubeflow pipeline connecting components

Import kfp.dsl.pipeline and define a pipeline function named ml_pipeline. Inside it, create tasks by calling prepare_data_op(), then train_model_op(), and finally evaluate_model_op(). Make sure train_model_op runs after prepare_data_op, and evaluate_model_op runs after train_model_op.

MLOps

from kfp.components import create_component_from_func
from kfp.dsl import pipeline

def prepare_data():
    print("Preparing data...")

def train_model():
    print("Training model...")

def evaluate_model():
    print("Evaluating model...")

prepare_data_op = create_component_from_func(prepare_data)
train_model_op = create_component_from_func(train_model)
evaluate_model_op = create_component_from_func(evaluate_model)

# Define the pipeline function ml_pipeline
# Your code here

Need a hint?

Use the @pipeline decorator and call components inside the pipeline function. Use task.after(other_task) to set order.

4

Compile and run the Kubeflow pipeline

Import kfp.Client and kfp.compiler. Compile the pipeline ml_pipeline to a file named ml_pipeline.yaml. Then create a kfp.Client instance and run the pipeline with the name ml-pipeline-run.

MLOps

from kfp import compiler, Client

# Compile the pipeline to ml_pipeline.yaml
# Create a Client instance
# Run the pipeline with name 'ml-pipeline-run'
# Your code here

Need a hint?

Use compiler.Compiler().compile() to create the YAML file. Then use Client() and create_run_from_pipeline_func() to run the pipeline.