Kafka on Kubernetes (Strimzi) - Commands & Configuration

Introduction

Running Kafka on Kubernetes can be complex because Kafka needs careful setup and management. Strimzi is a tool that makes it easy to deploy and manage Kafka clusters on Kubernetes automatically.

When you want to run Kafka alongside other apps in a Kubernetes cluster without manual setup.

When you need to scale Kafka brokers up or down easily using Kubernetes features.

When you want automatic Kafka cluster configuration and management inside Kubernetes.

When you want to upgrade Kafka versions safely with minimal downtime.

When you want to monitor Kafka cluster health using Kubernetes-native tools.

Config File - kafka-cluster.yaml

kafka-cluster.yaml

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: my-cluster
  namespace: kafka-namespace
spec:
  kafka:
    version: 3.4.0
    replicas: 3
    listeners:
      - name: plain
        port: 9092
        type: internal
        tls: false
    storage:
      type: persistent-claim
      size: 10Gi
      class: standard
      deleteClaim: false
  zookeeper:
    replicas: 3
    storage:
      type: persistent-claim
      size: 10Gi
      class: standard
      deleteClaim: false
  entityOperator:
    topicOperator: {}
    userOperator: {}

This file defines a Kafka cluster named my-cluster in the kafka-namespace namespace.

The kafka section sets 3 Kafka brokers with internal plain listeners on port 9092 and persistent storage of 10Gi each.

The zookeeper section sets 3 Zookeeper nodes with similar storage.

The entityOperator manages Kafka topics and users automatically.

Commands

Create a Kubernetes namespace to isolate Kafka resources.

Terminal

kubectl create namespace kafka-namespace

Expected Output

namespace/kafka-namespace created

Install Strimzi operator in the kafka-namespace to manage Kafka clusters.

Terminal

kubectl apply -f https://strimzi.io/install/latest?namespace=kafka-namespace -n kafka-namespace

Expected Output

customresourcedefinition.apiextensions.k8s.io/kafkas.kafka.strimzi.io created

customresourcedefinition.apiextensions.k8s.io/kafkatopics.kafka.strimzi.io created

customresourcedefinition.apiextensions.k8s.io/kafkausers.kafka.strimzi.io created

serviceaccount/strimzi-cluster-operator created

clusterrole.rbac.authorization.k8s.io/strimzi-cluster-operator-namespaced created

clusterrolebinding.rbac.authorization.k8s.io/strimzi-cluster-operator-namespaced created

deployment.apps/strimzi-cluster-operator created

→

-n - Specifies the namespace where Strimzi operator is installed

Create the Kafka cluster using the Strimzi Kafka custom resource definition.

Terminal

kubectl apply -f kafka-cluster.yaml -n kafka-namespace

Expected Output

kafka.kafka.strimzi.io/my-cluster created

Check that Kafka and Zookeeper pods are running and ready.

Terminal

kubectl get pods -n kafka-namespace

Expected Output

NAME                                   READY   STATUS    RESTARTS   AGE

my-cluster-kafka-0                     1/1     Running   0          2m

my-cluster-kafka-1                     1/1     Running   0          2m

my-cluster-kafka-2                     1/1     Running   0          2m

my-cluster-zookeeper-0                 1/1     Running   0          2m

my-cluster-zookeeper-1                 1/1     Running   0          2m

my-cluster-zookeeper-2                 1/1     Running   0          2m

strimzi-cluster-operator-5d7f9f6f7d-8x9zq 1/1 Running 0          5m

View detailed status and events of the Kafka cluster to verify it is healthy.

Terminal

kubectl describe kafka my-cluster -n kafka-namespace

Expected Output

Name:         my-cluster

Namespace:    kafka-namespace

Labels:       strimzi.io/cluster=my-cluster

Annotations:  <none>



Status:

  Conditions:

    Type:  Ready

    Status:  True

  Observed Generation: 1

  Replicas: 3

  Ready Replicas: 3

  Zookeeper Nodes Ready: 3

  Kafka Nodes Ready: 3

Events:

  Type    Reason  Age   From                      Message

  ----    ------  ----  ----                      -------

  Normal  Created  3m    kafka-operator            Created Kafka cluster

Key Concept

If you remember nothing else from this pattern, remember: Strimzi lets Kubernetes manage Kafka clusters easily by using custom resources and operators.

Common Mistakes

Not creating the Kubernetes namespace before installing Strimzi and Kafka resources.

The resources fail to deploy because the namespace does not exist.

Always create the namespace first with 'kubectl create namespace kafka-namespace'.

Applying the Kafka cluster YAML before the Strimzi operator is installed.

The Kafka custom resource is unknown to Kubernetes and the apply command fails.

Install the Strimzi operator first, then apply the Kafka cluster YAML.

Using storage class names that do not exist in the Kubernetes cluster.

Persistent volume claims fail to bind and Kafka pods stay pending.

Check available storage classes with 'kubectl get storageclass' and use a valid one.

Summary

Create a Kubernetes namespace to isolate Kafka resources.

Install the Strimzi operator to manage Kafka clusters on Kubernetes.

Apply a Kafka custom resource YAML to deploy a Kafka cluster with Zookeeper.

Verify Kafka and Zookeeper pods are running with 'kubectl get pods'.

Use 'kubectl describe kafka' to check Kafka cluster health and status.