How to Use Kubeflow on Kubernetes: Step-by-Step Guide

To use Kubeflow on Kubernetes, first install a Kubernetes cluster, then deploy Kubeflow using its official manifests or tools like kfctl. After deployment, you can create and run machine learning pipelines through Kubeflow's dashboard or CLI.

Syntax

Here is the basic syntax to deploy Kubeflow on Kubernetes using kfctl:

• kfctl apply -V -f <config-file-url>: Deploy Kubeflow using a config file URL.
• kubectl get pods -n kubeflow: Check the status of Kubeflow pods.
• kubectl port-forward svc/istio-ingressgateway -n istio-system 8080:80: Access Kubeflow dashboard locally.

Each command helps you set up, verify, and access Kubeflow on your Kubernetes cluster.

bash

export KF_NAME=my-kubeflow
export BASE_DIR=$HOME
export KF_DIR=${BASE_DIR}/${KF_NAME}
export CONFIG_URI="https://raw.githubusercontent.com/kubeflow/manifests/v1.6.0/kfdef/kfctl_k8s_istio.v1.6.0.yaml"

mkdir -p ${KF_DIR}
cd ${KF_DIR}
kfctl apply -V -f ${CONFIG_URI}

kubectl get pods -n kubeflow

kubectl port-forward svc/istio-ingressgateway -n istio-system 8080:80

Output

NAME                                         READY   STATUS    RESTARTS   AGE
centraldashboard-5d7f7f7d7f-8x9qv           1/1     Running   0          2m
metadata-grpc-deployment-7f9d6f7d7f-2x9qv    1/1     Running   0          2m
pipeline-ui-7f9d7f7d7f-9x9qv                 1/1     Running   0          2m
...

Forwarding from 127.0.0.1:8080 -> 80

Example

This example shows how to deploy Kubeflow on a local Kubernetes cluster (like Minikube) and access the dashboard:

bash

minikube start --memory=8192 --cpus=4

export KF_NAME=minikube-kubeflow
export BASE_DIR=$HOME
export KF_DIR=${BASE_DIR}/${KF_NAME}
export CONFIG_URI="https://raw.githubusercontent.com/kubeflow/manifests/v1.6.0/kfdef/kfctl_k8s_istio.v1.6.0.yaml"

mkdir -p ${KF_DIR}
cd ${KF_DIR}
kfctl apply -V -f ${CONFIG_URI}

kubectl get pods -n kubeflow

kubectl port-forward svc/istio-ingressgateway -n istio-system 8080:80

# Open http://localhost:8080 in your browser to use Kubeflow dashboard

Output

Starting local Kubernetes cluster...
Cluster started.

NAME                                         READY   STATUS    RESTARTS   AGE
centraldashboard-5d7f7f7d7f-8x9qv           1/1     Running   0          5m
metadata-grpc-deployment-7f9d6f7d7f-2x9qv    1/1     Running   0          5m
pipeline-ui-7f9d7f7d7f-9x9qv                 1/1     Running   0          5m
...

Forwarding from 127.0.0.1:8080 -> 80

Common Pitfalls

1. Insufficient resources: Kubeflow needs enough CPU, memory, and storage. Using small clusters can cause pods to fail.

2. Version mismatch: Using incompatible Kubernetes or Kubeflow versions can break deployment.

3. Missing namespace or context: Always specify the correct Kubernetes namespace (kubeflow) and cluster context.

4. Access issues: Forgetting to port-forward or configure ingress blocks dashboard access.

bash

kubectl get pods
# WRONG: This shows pods in default namespace, not kubeflow namespace

kubectl get pods -n kubeflow
# CORRECT: Shows Kubeflow pods status

Output

No resources found in default namespace.

NAME                                         READY   STATUS    RESTARTS   AGE
centraldashboard-5d7f7f7d7f-8x9qv           1/1     Running   0          10m
metadata-grpc-deployment-7f9d6f7d7f-2x9qv    1/1     Running   0          10m

Quick Reference

• Install Kubernetes: Use Minikube, kind, or cloud providers.
• Deploy Kubeflow: Use kfctl apply -f <config-url>.
• Check pods: kubectl get pods -n kubeflow.
• Access dashboard: kubectl port-forward svc/istio-ingressgateway -n istio-system 8080:80.
• Run pipelines: Use Kubeflow UI or CLI.

Key Takeaways

Deploy Kubeflow on Kubernetes using the official config files with kfctl.

Ensure your Kubernetes cluster has enough resources before deploying Kubeflow.

Always check Kubeflow pods in the kubeflow namespace to verify deployment.

Use port-forwarding to access the Kubeflow dashboard locally.

Match Kubeflow and Kubernetes versions to avoid compatibility issues.