Kafka on Kubernetes (Strimzi) - Commands & Configuration

Introduction

Running Kafka on Kubernetes can be complex because Kafka needs careful setup and management. Strimzi is a tool that makes it easy to deploy and manage Kafka clusters on Kubernetes automatically.

When you want to run Kafka alongside other apps in a Kubernetes cluster without manual setup.

When you need to scale Kafka brokers up or down easily using Kubernetes features.

When you want automatic Kafka cluster configuration and management inside Kubernetes.

When you want to upgrade Kafka versions safely with minimal downtime.

When you want to monitor Kafka cluster health using Kubernetes-native tools.

Config File - kafka-cluster.yaml

kafka-cluster.yaml

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: my-cluster
  namespace: kafka-namespace
spec:
  kafka:
    version: 3.4.0
    replicas: 3
    listeners:
      - name: plain
        port: 9092
        type: internal
        tls: false
    storage:
      type: persistent-claim
      size: 10Gi
      class: standard
      deleteClaim: false
  zookeeper:
    replicas: 3
    storage:
      type: persistent-claim
      size: 10Gi
      class: standard
      deleteClaim: false
  entityOperator:
    topicOperator: {}
    userOperator: {}

This file defines a Kafka cluster named my-cluster in the kafka-namespace namespace.

The kafka section sets 3 Kafka brokers with internal plain listeners on port 9092 and persistent storage of 10Gi each.

The zookeeper section sets 3 Zookeeper nodes with similar storage.

The entityOperator manages Kafka topics and users automatically.

Commands

Create a Kubernetes namespace to isolate Kafka resources.

Terminal

kubectl create namespace kafka-namespace

Expected Output

namespace/kafka-namespace created

Install Strimzi operator in the kafka-namespace to manage Kafka clusters.

Terminal

kubectl apply -f https://strimzi.io/install/latest?namespace=kafka-namespace -n kafka-namespace

Expected Output

customresourcedefinition.apiextensions.k8s.io/kafkas.kafka.strimzi.io created

customresourcedefinition.apiextensions.k8s.io/kafkatopics.kafka.strimzi.io created

customresourcedefinition.apiextensions.k8s.io/kafkausers.kafka.strimzi.io created

serviceaccount/strimzi-cluster-operator created

clusterrole.rbac.authorization.k8s.io/strimzi-cluster-operator-namespaced created

clusterrolebinding.rbac.authorization.k8s.io/strimzi-cluster-operator-namespaced created

deployment.apps/strimzi-cluster-operator created

→

-n - Specifies the namespace where Strimzi operator is installed

Create the Kafka cluster using the Strimzi Kafka custom resource definition.

Terminal

kubectl apply -f kafka-cluster.yaml -n kafka-namespace

Expected Output

kafka.kafka.strimzi.io/my-cluster created

Check that Kafka and Zookeeper pods are running and ready.

Terminal

kubectl get pods -n kafka-namespace

Expected Output

NAME                                   READY   STATUS    RESTARTS   AGE

my-cluster-kafka-0                     1/1     Running   0          2m

my-cluster-kafka-1                     1/1     Running   0          2m

my-cluster-kafka-2                     1/1     Running   0          2m

my-cluster-zookeeper-0                 1/1     Running   0          2m

my-cluster-zookeeper-1                 1/1     Running   0          2m

my-cluster-zookeeper-2                 1/1     Running   0          2m

strimzi-cluster-operator-5d7f9f6f7d-8x9zq 1/1 Running 0          5m

View detailed status and events of the Kafka cluster to verify it is healthy.

Terminal

kubectl describe kafka my-cluster -n kafka-namespace

Expected Output

Name:         my-cluster

Namespace:    kafka-namespace

Labels:       strimzi.io/cluster=my-cluster

Annotations:  <none>



Status:

  Conditions:

    Type:  Ready

    Status:  True

  Observed Generation: 1

  Replicas: 3

  Ready Replicas: 3

  Zookeeper Nodes Ready: 3

  Kafka Nodes Ready: 3

Events:

  Type    Reason  Age   From                      Message

  ----    ------  ----  ----                      -------

  Normal  Created  3m    kafka-operator            Created Kafka cluster

Key Concept

If you remember nothing else from this pattern, remember: Strimzi lets Kubernetes manage Kafka clusters easily by using custom resources and operators.

Common Mistakes

Not creating the Kubernetes namespace before installing Strimzi and Kafka resources.

The resources fail to deploy because the namespace does not exist.

Always create the namespace first with 'kubectl create namespace kafka-namespace'.

Applying the Kafka cluster YAML before the Strimzi operator is installed.

The Kafka custom resource is unknown to Kubernetes and the apply command fails.

Install the Strimzi operator first, then apply the Kafka cluster YAML.

Using storage class names that do not exist in the Kubernetes cluster.

Persistent volume claims fail to bind and Kafka pods stay pending.

Check available storage classes with 'kubectl get storageclass' and use a valid one.

Summary

Create a Kubernetes namespace to isolate Kafka resources.

Install the Strimzi operator to manage Kafka clusters on Kubernetes.

Apply a Kafka custom resource YAML to deploy a Kafka cluster with Zookeeper.

Verify Kafka and Zookeeper pods are running with 'kubectl get pods'.

Use 'kubectl describe kafka' to check Kafka cluster health and status.

Practice

1. What is the main purpose of Strimzi when running Kafka on Kubernetes?

easy

A. To simplify and automate Kafka cluster management on Kubernetes

B. To replace Kubernetes with a new container orchestration system

C. To provide a graphical user interface for Kafka topics

D. To convert Kafka messages into SQL queries automatically

Solution

1. Step 1: Understand Strimzi's role

   Strimzi is designed to help run Kafka on Kubernetes by managing Kafka clusters easily.

2. Step 2: Identify the correct purpose

   It automates pod management, storage, and recovery, simplifying Kafka operations on Kubernetes.

3. Final Answer:

   To simplify and automate Kafka cluster management on Kubernetes -> Option A

4. Quick Check:

   Strimzi = Kafka automation on Kubernetes [OK]

Hint: Strimzi automates Kafka on Kubernetes, not replaces or adds UI [OK]

Common Mistakes:

• Thinking Strimzi replaces Kubernetes
• Confusing Strimzi with Kafka UI tools
• Assuming Strimzi converts Kafka messages to SQL

2. Which of the following is the correct command to apply a Kafka cluster YAML file named kafka-cluster.yaml using Strimzi on Kubernetes?

easy

A. kubectl run kafka-cluster.yaml

B. strimzi create kafka-cluster.yaml

C. kubectl apply -f kafka-cluster.yaml

D. kafka apply kafka-cluster.yaml

Solution

1. Step 1: Recall how Kubernetes applies YAML files

   Kubernetes uses kubectl apply -f filename.yaml to create or update resources.

2. Step 2: Match the command to Strimzi usage

   Strimzi uses Kubernetes resources, so applying the Kafka cluster YAML is done with kubectl apply -f.

3. Final Answer:

   kubectl apply -f kafka-cluster.yaml -> Option C

4. Quick Check:

   Apply YAML with kubectl apply -f [OK]

Hint: Use kubectl apply -f for YAML files on Kubernetes [OK]

Common Mistakes:

• Using non-existent 'strimzi create' command
• Confusing kubectl run with apply
• Trying to use kafka CLI for Kubernetes YAML

3. Given the following YAML snippet for a Kafka cluster managed by Strimzi:

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: my-cluster
spec:
  kafka:
    replicas: 3
    storage:
      type: ephemeral
  zookeeper:
    replicas: 3
    storage:
      type: ephemeral

What will be the number of Kafka broker pods created in the Kubernetes cluster?

medium

A. 3

B. 0

C. 6

D. 1

Solution

1. Step 1: Identify Kafka replicas in YAML

   The replicas field under kafka is set to 3, meaning 3 Kafka broker pods.

2. Step 2: Understand pod creation

   Strimzi creates one pod per replica, so 3 Kafka broker pods will be created.

3. Final Answer:

   3 -> Option A

4. Quick Check:

   Kafka replicas = 3 pods [OK]

Hint: Kafka replicas number = Kafka pods count [OK]

Common Mistakes:

• Adding Kafka and Zookeeper replicas together
• Confusing ephemeral storage with pod count
• Assuming default pod count if replicas missing

4. You applied a Kafka cluster YAML with 3 replicas, but only 1 Kafka pod is running. Which of the following is the most likely cause?

medium

A. Kafka storage type is set to ephemeral

B. The YAML file has replicas: 1 under Kafka section

C. Zookeeper replicas are set to 1

D. Strimzi operator is not installed or running

Solution

1. Step 1: Check Kafka replicas in YAML

   If YAML specifies 3 replicas, pod count should be 3, so replicas: 1 is unlikely.

2. Step 2: Consider Strimzi operator role

   Strimzi operator manages Kafka pods. If it is missing or not running, pods won't be created properly.

3. Final Answer:

   Strimzi operator is not installed or running -> Option D

4. Quick Check:

   Operator missing = pods not managed [OK]

Hint: Check Strimzi operator status if pods don't match replicas [OK]

Common Mistakes:

• Blaming Zookeeper replicas for Kafka pod count
• Thinking ephemeral storage stops pod creation
• Assuming YAML replicas value is always correct

5. You want to update your Kafka cluster managed by Strimzi to use persistent storage instead of ephemeral. Which YAML snippet correctly changes the Kafka storage to persistent with 10Gi volume size?

hard

A. storage:
type: persistent-claim
size: 10Gi

B. storage:
type: persistent-claim
volumeClaimTemplate:
spec:
resources:
requests:
storage: 10Gi

C. storage:
type: persistent
size: 10Gi

D. storage:
persistentVolume:
size: 10Gi

Solution

1. Step 1: Understand Strimzi persistent storage syntax

   Strimzi uses persistent-claim type with a volumeClaimTemplate to define PVC size.

2. Step 2: Identify correct YAML structure

   storage: type: persistent-claim volumeClaimTemplate: spec: resources: requests: storage: 10Gi correctly uses volumeClaimTemplate.spec.resources.requests.storage: 10Gi to request 10Gi persistent volume.

3. Final Answer:

   storage: type: persistent-claim volumeClaimTemplate: spec: resources: requests: storage: 10Gi -> Option B

4. Quick Check:

   Persistent storage needs volumeClaimTemplate with size [OK]

Hint: Use volumeClaimTemplate for persistent storage size in Strimzi [OK]

Common Mistakes:

• Using 'size' directly under storage without volumeClaimTemplate
• Using wrong storage type like 'persistent'
• Confusing persistentVolume with persistent-claim