Kafka on Kubernetes (Strimzi) - Time & Space Complexity

Time Complexity: Kafka on Kubernetes (Strimzi)

O(n * m)

Understanding Time Complexity

When running Kafka on Kubernetes using Strimzi, it's important to understand how the system handles tasks as the number of Kafka topics or partitions grows.

We want to know how the time to manage Kafka resources changes as we add more topics or partitions.

Scenario Under Consideration

Analyze the time complexity of the following Kafka resource reconciliation loop in Strimzi.

for (Topic topic : kafkaCluster.getTopics()) {
  reconcileTopic(topic);
  for (Partition partition : topic.getPartitions()) {
    assignPartition(partition);
  }
}
    

This code loops through all Kafka topics and then through all partitions of each topic to assign them properly in the Kubernetes cluster.

Identify Repeating Operations

Look at the loops that repeat work:

• Primary operation: Looping through all topics and then all partitions inside each topic.
• How many times: The outer loop runs once per topic, and the inner loop runs once per partition per topic.

How Execution Grows With Input

As the number of topics and partitions grows, the work grows too.

Input Size (topics * partitions)	Approx. Operations
10 topics * 5 partitions	50 operations
100 topics * 5 partitions	500 operations
1000 topics * 5 partitions	5000 operations

Pattern observation: The total work grows roughly in proportion to the number of topics times the number of partitions.

Final Time Complexity

Time Complexity: O(n * m)

This means the time to complete the task grows in direct proportion to the number of topics (n) multiplied by the number of partitions per topic (m).

Common Mistake

[X] Wrong: "The time to assign partitions grows only with the number of topics, not partitions."

[OK] Correct: Each partition needs separate handling, so the total work depends on both topics and partitions, not just topics alone.

Interview Connect

Understanding how Kafka scales on Kubernetes helps you explain how systems handle growing workloads, a key skill in DevOps roles.

Self-Check

"What if we changed the code to assign partitions in parallel instead of sequentially? How would the time complexity change?"

Practice

1. What is the main purpose of Strimzi when running Kafka on Kubernetes?

easy

A. To simplify and automate Kafka cluster management on Kubernetes

B. To replace Kubernetes with a new container orchestration system

C. To provide a graphical user interface for Kafka topics

D. To convert Kafka messages into SQL queries automatically

Solution

1. Step 1: Understand Strimzi's role

   Strimzi is designed to help run Kafka on Kubernetes by managing Kafka clusters easily.

2. Step 2: Identify the correct purpose

   It automates pod management, storage, and recovery, simplifying Kafka operations on Kubernetes.

3. Final Answer:

   To simplify and automate Kafka cluster management on Kubernetes -> Option A

4. Quick Check:

   Strimzi = Kafka automation on Kubernetes [OK]

Hint: Strimzi automates Kafka on Kubernetes, not replaces or adds UI [OK]

Common Mistakes:

• Thinking Strimzi replaces Kubernetes
• Confusing Strimzi with Kafka UI tools
• Assuming Strimzi converts Kafka messages to SQL

2. Which of the following is the correct command to apply a Kafka cluster YAML file named kafka-cluster.yaml using Strimzi on Kubernetes?

easy

A. kubectl run kafka-cluster.yaml

B. strimzi create kafka-cluster.yaml

C. kubectl apply -f kafka-cluster.yaml

D. kafka apply kafka-cluster.yaml

Solution

1. Step 1: Recall how Kubernetes applies YAML files

   Kubernetes uses kubectl apply -f filename.yaml to create or update resources.

2. Step 2: Match the command to Strimzi usage

   Strimzi uses Kubernetes resources, so applying the Kafka cluster YAML is done with kubectl apply -f.

3. Final Answer:

   kubectl apply -f kafka-cluster.yaml -> Option C

4. Quick Check:

   Apply YAML with kubectl apply -f [OK]

Hint: Use kubectl apply -f for YAML files on Kubernetes [OK]

Common Mistakes:

• Using non-existent 'strimzi create' command
• Confusing kubectl run with apply
• Trying to use kafka CLI for Kubernetes YAML

3. Given the following YAML snippet for a Kafka cluster managed by Strimzi:

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: my-cluster
spec:
  kafka:
    replicas: 3
    storage:
      type: ephemeral
  zookeeper:
    replicas: 3
    storage:
      type: ephemeral

What will be the number of Kafka broker pods created in the Kubernetes cluster?

medium

A. 3

B. 0

C. 6

D. 1

Solution

1. Step 1: Identify Kafka replicas in YAML

   The replicas field under kafka is set to 3, meaning 3 Kafka broker pods.

2. Step 2: Understand pod creation

   Strimzi creates one pod per replica, so 3 Kafka broker pods will be created.

3. Final Answer:

   3 -> Option A

4. Quick Check:

   Kafka replicas = 3 pods [OK]

Hint: Kafka replicas number = Kafka pods count [OK]

Common Mistakes:

• Adding Kafka and Zookeeper replicas together
• Confusing ephemeral storage with pod count
• Assuming default pod count if replicas missing

4. You applied a Kafka cluster YAML with 3 replicas, but only 1 Kafka pod is running. Which of the following is the most likely cause?

medium

A. Kafka storage type is set to ephemeral

B. The YAML file has replicas: 1 under Kafka section

C. Zookeeper replicas are set to 1

D. Strimzi operator is not installed or running

Solution

1. Step 1: Check Kafka replicas in YAML

   If YAML specifies 3 replicas, pod count should be 3, so replicas: 1 is unlikely.

2. Step 2: Consider Strimzi operator role

   Strimzi operator manages Kafka pods. If it is missing or not running, pods won't be created properly.

3. Final Answer:

   Strimzi operator is not installed or running -> Option D

4. Quick Check:

   Operator missing = pods not managed [OK]

Hint: Check Strimzi operator status if pods don't match replicas [OK]

Common Mistakes:

• Blaming Zookeeper replicas for Kafka pod count
• Thinking ephemeral storage stops pod creation
• Assuming YAML replicas value is always correct

5. You want to update your Kafka cluster managed by Strimzi to use persistent storage instead of ephemeral. Which YAML snippet correctly changes the Kafka storage to persistent with 10Gi volume size?

hard

A. storage:
type: persistent-claim
size: 10Gi

B. storage:
type: persistent-claim
volumeClaimTemplate:
spec:
resources:
requests:
storage: 10Gi

C. storage:
type: persistent
size: 10Gi

D. storage:
persistentVolume:
size: 10Gi

Solution

1. Step 1: Understand Strimzi persistent storage syntax

   Strimzi uses persistent-claim type with a volumeClaimTemplate to define PVC size.

2. Step 2: Identify correct YAML structure

   storage: type: persistent-claim volumeClaimTemplate: spec: resources: requests: storage: 10Gi correctly uses volumeClaimTemplate.spec.resources.requests.storage: 10Gi to request 10Gi persistent volume.

3. Final Answer:

   storage: type: persistent-claim volumeClaimTemplate: spec: resources: requests: storage: 10Gi -> Option B

4. Quick Check:

   Persistent storage needs volumeClaimTemplate with size [OK]

Hint: Use volumeClaimTemplate for persistent storage size in Strimzi [OK]

Common Mistakes:

• Using 'size' directly under storage without volumeClaimTemplate
• Using wrong storage type like 'persistent'
• Confusing persistentVolume with persistent-claim