Why Kafka on Kubernetes (Strimzi)? - Purpose & Use Cases

The Big Idea

What if you could run Kafka clusters without wrestling with complex setup and endless fixes?

The Scenario

Imagine you need to run Kafka, a system that handles lots of messages, on many computers. You try to set it up by hand on each machine, installing software, configuring settings, and making sure everything talks to each other.

The Problem

Doing this manually is slow and confusing. One small mistake can break the whole system. If a machine fails, you have to fix it yourself. Scaling up means repeating the same hard steps again and again.

The Solution

Strimzi helps by automating Kafka setup on Kubernetes. It manages Kafka clusters for you, handles failures, and makes scaling easy. You just tell it what you want, and it does the hard work behind the scenes.

Before vs After

✗ Before

Install Kafka on each server manually
Configure each broker by hand
Monitor and fix issues yourself

✓ After

Apply Strimzi Kafka custom resource on Kubernetes
Let Strimzi operator create and manage Kafka cluster
Scale or recover automatically

What It Enables

With Strimzi on Kubernetes, you can run reliable, scalable Kafka clusters effortlessly, freeing you to focus on your apps, not infrastructure.

Real Life Example

A company wants to process millions of messages daily. Using Strimzi, they deploy Kafka on Kubernetes with one command. When traffic grows, they scale Kafka easily without downtime or manual fixes.

Key Takeaways

Manual Kafka setup is complex and error-prone.

Strimzi automates Kafka management on Kubernetes.

This makes Kafka reliable, scalable, and easier to maintain.

Practice

1. What is the main purpose of Strimzi when running Kafka on Kubernetes?

easy

A. To simplify and automate Kafka cluster management on Kubernetes

B. To replace Kubernetes with a new container orchestration system

C. To provide a graphical user interface for Kafka topics

D. To convert Kafka messages into SQL queries automatically

Solution

1. Step 1: Understand Strimzi's role

   Strimzi is designed to help run Kafka on Kubernetes by managing Kafka clusters easily.

2. Step 2: Identify the correct purpose

   It automates pod management, storage, and recovery, simplifying Kafka operations on Kubernetes.

3. Final Answer:

   To simplify and automate Kafka cluster management on Kubernetes -> Option A

4. Quick Check:

   Strimzi = Kafka automation on Kubernetes [OK]

Hint: Strimzi automates Kafka on Kubernetes, not replaces or adds UI [OK]

Common Mistakes:

• Thinking Strimzi replaces Kubernetes
• Confusing Strimzi with Kafka UI tools
• Assuming Strimzi converts Kafka messages to SQL

2. Which of the following is the correct command to apply a Kafka cluster YAML file named kafka-cluster.yaml using Strimzi on Kubernetes?

easy

A. kubectl run kafka-cluster.yaml

B. strimzi create kafka-cluster.yaml

C. kubectl apply -f kafka-cluster.yaml

D. kafka apply kafka-cluster.yaml

Solution

1. Step 1: Recall how Kubernetes applies YAML files

   Kubernetes uses kubectl apply -f filename.yaml to create or update resources.

2. Step 2: Match the command to Strimzi usage

   Strimzi uses Kubernetes resources, so applying the Kafka cluster YAML is done with kubectl apply -f.

3. Final Answer:

   kubectl apply -f kafka-cluster.yaml -> Option C

4. Quick Check:

   Apply YAML with kubectl apply -f [OK]

Hint: Use kubectl apply -f for YAML files on Kubernetes [OK]

Common Mistakes:

• Using non-existent 'strimzi create' command
• Confusing kubectl run with apply
• Trying to use kafka CLI for Kubernetes YAML

3. Given the following YAML snippet for a Kafka cluster managed by Strimzi:

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: my-cluster
spec:
  kafka:
    replicas: 3
    storage:
      type: ephemeral
  zookeeper:
    replicas: 3
    storage:
      type: ephemeral

What will be the number of Kafka broker pods created in the Kubernetes cluster?

medium

A. 3

B. 0

C. 6

D. 1

Solution

1. Step 1: Identify Kafka replicas in YAML

   The replicas field under kafka is set to 3, meaning 3 Kafka broker pods.

2. Step 2: Understand pod creation

   Strimzi creates one pod per replica, so 3 Kafka broker pods will be created.

3. Final Answer:

   3 -> Option A

4. Quick Check:

   Kafka replicas = 3 pods [OK]

Hint: Kafka replicas number = Kafka pods count [OK]

Common Mistakes:

• Adding Kafka and Zookeeper replicas together
• Confusing ephemeral storage with pod count
• Assuming default pod count if replicas missing

4. You applied a Kafka cluster YAML with 3 replicas, but only 1 Kafka pod is running. Which of the following is the most likely cause?

medium

A. Kafka storage type is set to ephemeral

B. The YAML file has replicas: 1 under Kafka section

C. Zookeeper replicas are set to 1

D. Strimzi operator is not installed or running

Solution

1. Step 1: Check Kafka replicas in YAML

   If YAML specifies 3 replicas, pod count should be 3, so replicas: 1 is unlikely.

2. Step 2: Consider Strimzi operator role

   Strimzi operator manages Kafka pods. If it is missing or not running, pods won't be created properly.

3. Final Answer:

   Strimzi operator is not installed or running -> Option D

4. Quick Check:

   Operator missing = pods not managed [OK]

Hint: Check Strimzi operator status if pods don't match replicas [OK]

Common Mistakes:

• Blaming Zookeeper replicas for Kafka pod count
• Thinking ephemeral storage stops pod creation
• Assuming YAML replicas value is always correct

5. You want to update your Kafka cluster managed by Strimzi to use persistent storage instead of ephemeral. Which YAML snippet correctly changes the Kafka storage to persistent with 10Gi volume size?

hard

A. storage:
type: persistent-claim
size: 10Gi

B. storage:
type: persistent-claim
volumeClaimTemplate:
spec:
resources:
requests:
storage: 10Gi

C. storage:
type: persistent
size: 10Gi

D. storage:
persistentVolume:
size: 10Gi

Solution

1. Step 1: Understand Strimzi persistent storage syntax

   Strimzi uses persistent-claim type with a volumeClaimTemplate to define PVC size.

2. Step 2: Identify correct YAML structure

   storage: type: persistent-claim volumeClaimTemplate: spec: resources: requests: storage: 10Gi correctly uses volumeClaimTemplate.spec.resources.requests.storage: 10Gi to request 10Gi persistent volume.

3. Final Answer:

   storage: type: persistent-claim volumeClaimTemplate: spec: resources: requests: storage: 10Gi -> Option B

4. Quick Check:

   Persistent storage needs volumeClaimTemplate with size [OK]

Hint: Use volumeClaimTemplate for persistent storage size in Strimzi [OK]

Common Mistakes:

• Using 'size' directly under storage without volumeClaimTemplate
• Using wrong storage type like 'persistent'
• Confusing persistentVolume with persistent-claim