Kafka on Kubernetes (Strimzi) - Mini Project: Build & Apply

Kafka on Kubernetes with Strimzi

📖 Scenario: You are working in a team that wants to run Kafka on Kubernetes. To make this easier, you will use Strimzi, a tool that helps manage Kafka clusters on Kubernetes.This project will guide you step-by-step to create a Kafka cluster using Strimzi on Kubernetes.

🎯 Goal: Set up a Kafka cluster on Kubernetes using Strimzi by creating the necessary Kubernetes resources step-by-step.

📋 What You'll Learn

Have access to a Kubernetes cluster

kubectl command-line tool installed and configured

Strimzi Operator installed in the Kubernetes cluster

💡 Why This Matters

🌍 Real World

Running Kafka on Kubernetes is common in modern cloud-native applications to handle real-time data streams efficiently.

💼 Career

DevOps engineers and platform engineers often deploy and manage Kafka clusters on Kubernetes using tools like Strimzi.

1

Create a Kafka namespace

Create a Kubernetes namespace called kafka-project using the command kubectl create namespace kafka-project.

Kafka

# Run the command to create the namespace
# Your code here

Hint

Use kubectl create namespace kafka-project to create the namespace.

2

Deploy the Strimzi Kafka Operator

Apply the Strimzi Operator installation YAML in the kafka-project namespace using kubectl apply -f https://strimzi.io/install/latest?namespace=kafka-project -n kafka-project.

Kafka

kubectl create namespace kafka-project
# Apply the Strimzi Operator YAML below
# Your code here

Hint

Use kubectl apply -f with the Strimzi URL and specify the namespace with -n kafka-project.

3

Create a Kafka cluster custom resource

Create a YAML file named kafka-cluster.yaml with the exact content below to define a Kafka cluster with 1 Kafka broker and 1 Zookeeper node in the kafka-project namespace:

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: my-cluster
  namespace: kafka-project
spec:
  kafka:
    replicas: 1
    listeners:
      - name: plain
        port: 9092
        type: internal
        tls: false
    storage:
      type: ephemeral
  zookeeper:
    replicas: 1
    storage:
      type: ephemeral
  entityOperator:
    topicOperator: {}
    userOperator: {}

Then apply it with kubectl apply -f kafka-cluster.yaml.

Kafka

kubectl create namespace kafka-project
kubectl apply -f https://strimzi.io/install/latest?namespace=kafka-project -n kafka-project
# Create kafka-cluster.yaml with the given content and apply it
# Your code here

Hint

Create the YAML file exactly as shown and apply it with kubectl apply -f kafka-cluster.yaml.

4

Check Kafka cluster status

Run kubectl get kafka my-cluster -n kafka-project -o yaml to see the status of the Kafka cluster and confirm it is ready.

Kafka

kubectl create namespace kafka-project
kubectl apply -f https://strimzi.io/install/latest?namespace=kafka-project -n kafka-project

kubectl apply -f kafka-cluster.yaml
# Run the command to check Kafka cluster status
# Your code here

Hint

Use kubectl get kafka my-cluster -n kafka-project -o yaml to check the cluster status.

Practice

1. What is the main purpose of Strimzi when running Kafka on Kubernetes?

easy

A. To simplify and automate Kafka cluster management on Kubernetes

B. To replace Kubernetes with a new container orchestration system

C. To provide a graphical user interface for Kafka topics

D. To convert Kafka messages into SQL queries automatically

Solution

1. Step 1: Understand Strimzi's role

   Strimzi is designed to help run Kafka on Kubernetes by managing Kafka clusters easily.

2. Step 2: Identify the correct purpose

   It automates pod management, storage, and recovery, simplifying Kafka operations on Kubernetes.

3. Final Answer:

   To simplify and automate Kafka cluster management on Kubernetes -> Option A

4. Quick Check:

   Strimzi = Kafka automation on Kubernetes [OK]

Hint: Strimzi automates Kafka on Kubernetes, not replaces or adds UI [OK]

Common Mistakes:

• Thinking Strimzi replaces Kubernetes
• Confusing Strimzi with Kafka UI tools
• Assuming Strimzi converts Kafka messages to SQL

2. Which of the following is the correct command to apply a Kafka cluster YAML file named kafka-cluster.yaml using Strimzi on Kubernetes?

easy

A. kubectl run kafka-cluster.yaml

B. strimzi create kafka-cluster.yaml

C. kubectl apply -f kafka-cluster.yaml

D. kafka apply kafka-cluster.yaml

Solution

1. Step 1: Recall how Kubernetes applies YAML files

   Kubernetes uses kubectl apply -f filename.yaml to create or update resources.

2. Step 2: Match the command to Strimzi usage

   Strimzi uses Kubernetes resources, so applying the Kafka cluster YAML is done with kubectl apply -f.

3. Final Answer:

   kubectl apply -f kafka-cluster.yaml -> Option C

4. Quick Check:

   Apply YAML with kubectl apply -f [OK]

Hint: Use kubectl apply -f for YAML files on Kubernetes [OK]

Common Mistakes:

• Using non-existent 'strimzi create' command
• Confusing kubectl run with apply
• Trying to use kafka CLI for Kubernetes YAML

3. Given the following YAML snippet for a Kafka cluster managed by Strimzi:

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: my-cluster
spec:
  kafka:
    replicas: 3
    storage:
      type: ephemeral
  zookeeper:
    replicas: 3
    storage:
      type: ephemeral

What will be the number of Kafka broker pods created in the Kubernetes cluster?

medium

A. 3

B. 0

C. 6

D. 1

Solution

1. Step 1: Identify Kafka replicas in YAML

   The replicas field under kafka is set to 3, meaning 3 Kafka broker pods.

2. Step 2: Understand pod creation

   Strimzi creates one pod per replica, so 3 Kafka broker pods will be created.

3. Final Answer:

   3 -> Option A

4. Quick Check:

   Kafka replicas = 3 pods [OK]

Hint: Kafka replicas number = Kafka pods count [OK]

Common Mistakes:

• Adding Kafka and Zookeeper replicas together
• Confusing ephemeral storage with pod count
• Assuming default pod count if replicas missing

4. You applied a Kafka cluster YAML with 3 replicas, but only 1 Kafka pod is running. Which of the following is the most likely cause?

medium

A. Kafka storage type is set to ephemeral

B. The YAML file has replicas: 1 under Kafka section

C. Zookeeper replicas are set to 1

D. Strimzi operator is not installed or running

Solution

1. Step 1: Check Kafka replicas in YAML

   If YAML specifies 3 replicas, pod count should be 3, so replicas: 1 is unlikely.

2. Step 2: Consider Strimzi operator role

   Strimzi operator manages Kafka pods. If it is missing or not running, pods won't be created properly.

3. Final Answer:

   Strimzi operator is not installed or running -> Option D

4. Quick Check:

   Operator missing = pods not managed [OK]

Hint: Check Strimzi operator status if pods don't match replicas [OK]

Common Mistakes:

• Blaming Zookeeper replicas for Kafka pod count
• Thinking ephemeral storage stops pod creation
• Assuming YAML replicas value is always correct

5. You want to update your Kafka cluster managed by Strimzi to use persistent storage instead of ephemeral. Which YAML snippet correctly changes the Kafka storage to persistent with 10Gi volume size?

hard

A. storage:
type: persistent-claim
size: 10Gi

B. storage:
type: persistent-claim
volumeClaimTemplate:
spec:
resources:
requests:
storage: 10Gi

C. storage:
type: persistent
size: 10Gi

D. storage:
persistentVolume:
size: 10Gi

Solution

1. Step 1: Understand Strimzi persistent storage syntax

   Strimzi uses persistent-claim type with a volumeClaimTemplate to define PVC size.

2. Step 2: Identify correct YAML structure

   storage: type: persistent-claim volumeClaimTemplate: spec: resources: requests: storage: 10Gi correctly uses volumeClaimTemplate.spec.resources.requests.storage: 10Gi to request 10Gi persistent volume.

3. Final Answer:

   storage: type: persistent-claim volumeClaimTemplate: spec: resources: requests: storage: 10Gi -> Option B

4. Quick Check:

   Persistent storage needs volumeClaimTemplate with size [OK]

Hint: Use volumeClaimTemplate for persistent storage size in Strimzi [OK]

Common Mistakes:

• Using 'size' directly under storage without volumeClaimTemplate
• Using wrong storage type like 'persistent'
• Confusing persistentVolume with persistent-claim