Kafka on Kubernetes (Strimzi) - Step-by-Step Execution

Process Flow - Kafka on Kubernetes (Strimzi)

Create Kubernetes Namespace

↓

Apply Strimzi Operator YAML

↓

Operator Installs Kafka Cluster CRDs

↓

Create Kafka Cluster Custom Resource

↓

Operator Deploys Kafka Pods & Services

↓

Kafka Cluster Runs inside Kubernetes

↓

Clients Connect to Kafka via Services

This flow shows how Strimzi operator manages Kafka on Kubernetes: install operator, create Kafka resource, operator deploys Kafka pods, then clients connect.

Execution Sample

Kafka

kubectl create namespace kafka
kubectl apply -f strimzi-cluster-operator.yaml -n kafka
kubectl apply -f kafka-cluster.yaml -n kafka
kubectl get pods -n kafka

Commands to create namespace, install Strimzi operator, deploy Kafka cluster, and check Kafka pods running.

Process Table

Step	Command	Action	Result/Output
1	kubectl create namespace kafka	Create namespace 'kafka'	namespace/kafka created
2	kubectl apply -f strimzi-cluster-operator.yaml -n kafka	Install Strimzi operator	deployment.apps/strimzi-cluster-operator created
3	Operator installs Kafka CRDs	Custom Resource Definitions added	kafkas.kafka.strimzi.io CRD created
4	kubectl apply -f kafka-cluster.yaml -n kafka	Create Kafka cluster resource	kafka.kafka.strimzi.io/my-cluster created
5	Operator deploys Kafka pods	Kafka pods starting	pod/my-cluster-kafka-0 Running pod/my-cluster-kafka-1 Running pod/my-cluster-kafka-2 Running
6	kubectl get pods -n kafka	Check Kafka pods status	NAME READY STATUS RESTARTS AGE my-cluster-kafka-0 1/1 Running 0 2m my-cluster-kafka-1 1/1 Running 0 2m my-cluster-kafka-2 1/1 Running 0 2m
7	Kafka cluster ready	Kafka cluster is running	Kafka cluster pods are all Running and Ready

💡 Kafka pods are Running and Ready, cluster is ready for clients

Status Tracker

Resource	Initial State	After Step 2	After Step 4	After Step 5	Final State
Namespace 'kafka'	Not present	Created	Created	Created	Created
Strimzi Operator	Not installed	Installed	Installed	Installed	Installed
Kafka CRDs	Not present	Installed	Installed	Installed	Installed
Kafka Cluster Resource	Not present	Not present	Created	Created	Created
Kafka Pods	Not present	Not present	Not present	Starting	Running

Key Moments - 3 Insights

Why do we need to create a namespace before installing Strimzi?

The namespace isolates Kafka resources. Step 1 creates it so Strimzi operator and Kafka pods run inside it, as shown in execution_table row 1.

What happens if we apply the Kafka cluster resource before installing Strimzi operator?

The Kafka CRDs won't exist yet, so Kubernetes rejects the Kafka cluster resource. Step 3 shows CRDs must be installed first.

Why do we check pod status after deploying Kafka cluster?

To confirm Kafka pods are running and ready for use. Execution_table rows 5 and 6 show pods transitioning to Running state.

Visual Quiz - 3 Questions

Look at the execution_table at step 3. What does the operator install?

A Kafka Custom Resource Definitions (CRDs)

B Kafka client tools

C Kafka pods

D Kubernetes namespace

💡 Hint

Check the 'Action' and 'Result/Output' columns at step 3 in execution_table.

At which step do Kafka pods start running?

A Step 2

B Step 4

C Step 5

D Step 7

💡 Hint

Look at the 'Action' and 'Result/Output' columns for pod status in execution_table.

If the namespace 'kafka' was not created, what would happen at step 2?

A Operator installs successfully in default namespace

B kubectl apply fails because namespace does not exist

C Kafka cluster deploys anyway

D Pods start running in default namespace

💡 Hint

Refer to variable_tracker for namespace state and execution_table step 1 and 2.

Concept Snapshot

Kafka on Kubernetes with Strimzi:
- Create a Kubernetes namespace for Kafka
- Install Strimzi operator YAML in that namespace
- Operator adds Kafka CRDs to Kubernetes
- Apply Kafka cluster custom resource YAML
- Operator deploys Kafka pods and services
- Check pods are Running before using Kafka
- Clients connect via Kubernetes services

Full Transcript

This visual execution shows how to run Kafka on Kubernetes using Strimzi operator. First, create a namespace called 'kafka' to isolate resources. Then apply the Strimzi operator YAML in that namespace. The operator installs Kafka Custom Resource Definitions (CRDs) into Kubernetes. Next, apply a Kafka cluster custom resource YAML to request a Kafka cluster. The operator sees this and deploys Kafka pods and services inside the namespace. Finally, check the pods are running and ready. Once ready, clients can connect to Kafka through Kubernetes services. This step-by-step flow helps beginners see how Kubernetes and Strimzi work together to run Kafka.

Practice

1. What is the main purpose of Strimzi when running Kafka on Kubernetes?

easy

A. To simplify and automate Kafka cluster management on Kubernetes

B. To replace Kubernetes with a new container orchestration system

C. To provide a graphical user interface for Kafka topics

D. To convert Kafka messages into SQL queries automatically

Solution

1. Step 1: Understand Strimzi's role

   Strimzi is designed to help run Kafka on Kubernetes by managing Kafka clusters easily.

2. Step 2: Identify the correct purpose

   It automates pod management, storage, and recovery, simplifying Kafka operations on Kubernetes.

3. Final Answer:

   To simplify and automate Kafka cluster management on Kubernetes -> Option A

4. Quick Check:

   Strimzi = Kafka automation on Kubernetes [OK]

Hint: Strimzi automates Kafka on Kubernetes, not replaces or adds UI [OK]

Common Mistakes:

• Thinking Strimzi replaces Kubernetes
• Confusing Strimzi with Kafka UI tools
• Assuming Strimzi converts Kafka messages to SQL

2. Which of the following is the correct command to apply a Kafka cluster YAML file named kafka-cluster.yaml using Strimzi on Kubernetes?

easy

A. kubectl run kafka-cluster.yaml

B. strimzi create kafka-cluster.yaml

C. kubectl apply -f kafka-cluster.yaml

D. kafka apply kafka-cluster.yaml

Solution

1. Step 1: Recall how Kubernetes applies YAML files

   Kubernetes uses kubectl apply -f filename.yaml to create or update resources.

2. Step 2: Match the command to Strimzi usage

   Strimzi uses Kubernetes resources, so applying the Kafka cluster YAML is done with kubectl apply -f.

3. Final Answer:

   kubectl apply -f kafka-cluster.yaml -> Option C

4. Quick Check:

   Apply YAML with kubectl apply -f [OK]

Hint: Use kubectl apply -f for YAML files on Kubernetes [OK]

Common Mistakes:

• Using non-existent 'strimzi create' command
• Confusing kubectl run with apply
• Trying to use kafka CLI for Kubernetes YAML

3. Given the following YAML snippet for a Kafka cluster managed by Strimzi:

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: my-cluster
spec:
  kafka:
    replicas: 3
    storage:
      type: ephemeral
  zookeeper:
    replicas: 3
    storage:
      type: ephemeral

What will be the number of Kafka broker pods created in the Kubernetes cluster?

medium

A. 3

B. 0

C. 6

D. 1

Solution

1. Step 1: Identify Kafka replicas in YAML

   The replicas field under kafka is set to 3, meaning 3 Kafka broker pods.

2. Step 2: Understand pod creation

   Strimzi creates one pod per replica, so 3 Kafka broker pods will be created.

3. Final Answer:

   3 -> Option A

4. Quick Check:

   Kafka replicas = 3 pods [OK]

Hint: Kafka replicas number = Kafka pods count [OK]

Common Mistakes:

• Adding Kafka and Zookeeper replicas together
• Confusing ephemeral storage with pod count
• Assuming default pod count if replicas missing

4. You applied a Kafka cluster YAML with 3 replicas, but only 1 Kafka pod is running. Which of the following is the most likely cause?

medium

A. Kafka storage type is set to ephemeral

B. The YAML file has replicas: 1 under Kafka section

C. Zookeeper replicas are set to 1

D. Strimzi operator is not installed or running

Solution

1. Step 1: Check Kafka replicas in YAML

   If YAML specifies 3 replicas, pod count should be 3, so replicas: 1 is unlikely.

2. Step 2: Consider Strimzi operator role

   Strimzi operator manages Kafka pods. If it is missing or not running, pods won't be created properly.

3. Final Answer:

   Strimzi operator is not installed or running -> Option D

4. Quick Check:

   Operator missing = pods not managed [OK]

Hint: Check Strimzi operator status if pods don't match replicas [OK]

Common Mistakes:

• Blaming Zookeeper replicas for Kafka pod count
• Thinking ephemeral storage stops pod creation
• Assuming YAML replicas value is always correct

5. You want to update your Kafka cluster managed by Strimzi to use persistent storage instead of ephemeral. Which YAML snippet correctly changes the Kafka storage to persistent with 10Gi volume size?

hard

A. storage:
type: persistent-claim
size: 10Gi

B. storage:
type: persistent-claim
volumeClaimTemplate:
spec:
resources:
requests:
storage: 10Gi

C. storage:
type: persistent
size: 10Gi

D. storage:
persistentVolume:
size: 10Gi

Solution

1. Step 1: Understand Strimzi persistent storage syntax

   Strimzi uses persistent-claim type with a volumeClaimTemplate to define PVC size.

2. Step 2: Identify correct YAML structure

   storage: type: persistent-claim volumeClaimTemplate: spec: resources: requests: storage: 10Gi correctly uses volumeClaimTemplate.spec.resources.requests.storage: 10Gi to request 10Gi persistent volume.

3. Final Answer:

   storage: type: persistent-claim volumeClaimTemplate: spec: resources: requests: storage: 10Gi -> Option B

4. Quick Check:

   Persistent storage needs volumeClaimTemplate with size [OK]

Hint: Use volumeClaimTemplate for persistent storage size in Strimzi [OK]

Common Mistakes:

• Using 'size' directly under storage without volumeClaimTemplate
• Using wrong storage type like 'persistent'
• Confusing persistentVolume with persistent-claim