Helm charts for Kafka - Commands & Configuration

Introduction

Installing and managing Kafka on Kubernetes can be complex. Helm charts simplify this by packaging Kafka configurations and deployment steps into reusable templates, making setup and updates easier.

When you want to deploy Kafka quickly on a Kubernetes cluster without writing complex YAML files.

When you need to manage Kafka configurations consistently across multiple environments.

When you want to upgrade Kafka versions or configurations with minimal manual changes.

When you want to share your Kafka deployment setup with your team or community.

When you want to automate Kafka deployment as part of your CI/CD pipeline.

Config File - values.yaml

values.yaml

replicaCount: 3
image:
  repository: bitnami/kafka
  tag: 3.5.1
  pullPolicy: IfNotPresent
service:
  type: ClusterIP
  port: 9092
zookeeper:
  enabled: true
  replicaCount: 3
  image:
    repository: bitnami/zookeeper
    tag: 3.8.1
    pullPolicy: IfNotPresent
resources:
  limits:
    cpu: 500m
    memory: 1Gi
  requests:
    cpu: 250m
    memory: 512Mi
persistence:
  enabled: true
  size: 8Gi
  storageClass: standard

This values.yaml file configures the Kafka Helm chart:

• replicaCount: Number of Kafka broker pods.
• image: Docker image details for Kafka.
• service: Defines Kafka service type and port.
• zookeeper: Enables embedded Zookeeper with its own replicas and image.
• resources: CPU and memory limits and requests for Kafka pods.
• persistence: Enables persistent storage with size and storage class.

This file customizes the Kafka deployment on Kubernetes using Helm.

Commands

Add the Bitnami Helm chart repository which contains the Kafka chart.

Terminal

helm repo add bitnami https://charts.bitnami.com/bitnami

Expected Output

"bitnami" has been added to your repositories

Update your local Helm chart repository cache to get the latest charts.

Terminal

helm repo update

Expected Output

Hang tight while we grab the latest from your chart repositories...

...Successfully got an update from the "bitnami" chart repository

Update Complete.

Install Kafka using the Bitnami Helm chart with your custom configuration from values.yaml.

Terminal

helm install my-kafka bitnami/kafka -f values.yaml

Expected Output

NAME: my-kafka

LAST DEPLOYED: Sat Jun 10 12:00:00 2023

NAMESPACE: default

STATUS: deployed

REVISION: 1

TEST SUITE: None

→

-f - Specify a custom values file to override default chart settings

Check the status of Kafka pods to ensure they are running after installation.

Terminal

kubectl get pods -l app.kubernetes.io/name=kafka

Expected Output

NAME                     READY   STATUS    RESTARTS   AGE

my-kafka-0               1/1     Running   0          1m

my-kafka-1               1/1     Running   0          1m

my-kafka-2               1/1     Running   0          1m

→

-l - Filter pods by label selector

Key Concept

If you remember nothing else from this pattern, remember: Helm charts package complex Kafka deployment details into simple commands and configuration files for easy Kubernetes management.

Common Mistakes

Not adding the Helm repository before trying to install the Kafka chart.

Helm cannot find the Kafka chart without the repository added, causing installation failure.

Always run 'helm repo add bitnami https://charts.bitnami.com/bitnami' before installing the chart.

Forgetting to update the Helm repo cache after adding a new repository.

Helm may use outdated chart information, leading to errors or installing old versions.

Run 'helm repo update' after adding or before installing charts to get the latest versions.

Installing Kafka without specifying a values file when custom settings are needed.

Kafka will deploy with default settings that may not fit your resource or persistence needs.

Use the '-f values.yaml' flag to apply your custom configuration during installation.

Summary

Add the Bitnami Helm repository to access Kafka charts.

Update Helm repositories to get the latest chart versions.

Install Kafka using Helm with a custom values.yaml file for configuration.

Verify Kafka pods are running using kubectl with label filtering.

Practice

1. What is the main purpose of using Helm charts for Kafka deployment on Kubernetes?

easy

A. To automate and simplify Kafka deployment and management

B. To write Kafka application code

C. To monitor Kafka topics manually

D. To replace Kafka brokers with databases

Solution

1. Step 1: Understand Helm charts role

   Helm charts package Kubernetes resources to automate deployment and management.

2. Step 2: Apply to Kafka context

   Using Helm charts for Kafka automates its deployment and configuration on Kubernetes clusters.

3. Final Answer:

   To automate and simplify Kafka deployment and management -> Option A

4. Quick Check:

   Helm automates Kafka deployment = B [OK]

Hint: Helm automates deployment tasks for Kafka on Kubernetes [OK]

Common Mistakes:

• Confusing Helm with Kafka application coding
• Thinking Helm replaces Kafka brokers
• Assuming Helm is for manual monitoring

2. Which Helm command syntax correctly sets the Kafka broker count to 3 during installation?

easy

A. helm install kafka bitnami/kafka --set brokerCount=3

B. helm install kafka bitnami/kafka --set brokers=3

C. helm install kafka bitnami/kafka --set replicaCount=3

D. helm install kafka bitnami/kafka --set kafka.broker=3

Solution

1. Step 1: Identify correct Helm parameter for replicas

   Common Helm charts use replicaCount to set number of pods/brokers.

2. Step 2: Match parameter with Kafka Helm chart

   Bitnami Kafka chart uses replicaCount to define broker count.

3. Final Answer:

   helm install kafka bitnami/kafka --set replicaCount=3 -> Option C

4. Quick Check:

   replicaCount sets broker pods = A [OK]

Hint: Use replicaCount to set broker number in Helm [OK]

Common Mistakes:

• Using incorrect parameter names like brokers or brokerCount
• Misplacing dot notation in --set
• Confusing Helm install syntax

3. Given this Helm command:

helm install kafka bitnami/kafka --set replicaCount=2,logRetentionHours=168

What will be the effect on the Kafka deployment?

medium

A. Deploy 2 Kafka brokers with logs retained for 168 hours

B. Deploy 168 Kafka brokers with 2 hours log retention

C. Deploy 2 Kafka brokers with default log retention

D. Deployment will fail due to syntax error

Solution

1. Step 1: Analyze replicaCount setting

   replicaCount=2 means 2 Kafka broker pods will be deployed.

2. Step 2: Analyze logRetentionHours setting

   logRetentionHours=168 sets log retention to 168 hours (7 days).

3. Final Answer:

   Deploy 2 Kafka brokers with logs retained for 168 hours -> Option A

4. Quick Check:

   replicaCount=2 and logRetentionHours=168 = D [OK]

Hint: Check each --set key=value carefully for effect [OK]

Common Mistakes:

• Mixing up values of replicaCount and logRetentionHours
• Assuming syntax error due to comma
• Ignoring log retention setting

4. You run this Helm upgrade command:

helm upgrade kafka bitnami/kafka --set replicaCount=3

But the Kafka deployment still has 2 brokers. What is the likely issue?

medium

A. The Helm chart version is incompatible

B. The Helm release name is incorrect

C. Kafka does not support scaling brokers via Helm

D. The replicaCount value was not applied due to missing --reuse-values flag

Solution

1. Step 1: Understand Helm upgrade behavior

   Helm upgrade requires --reuse-values to reuse previous values while applying new --set overrides.

2. Step 2: Identify why replicaCount didn't change

   Without --reuse-values, the new value may not apply correctly in combination with prior settings.

3. Final Answer:

   The replicaCount value was not applied due to missing --reuse-values flag -> Option D

4. Quick Check:

   Missing --reuse-values can block value changes = A [OK]

Hint: Use --reuse-values to keep and update Helm settings on upgrade [OK]

Common Mistakes:

• Assuming Helm release name is wrong
• Believing Kafka can't scale brokers
• Ignoring Helm chart version compatibility

5. You want to deploy Kafka with 3 brokers and customize the storage size to 50Gi using Helm. Which command correctly achieves this?

hard

A. helm install kafka bitnami/kafka --set replicaCount=3,storage.size=50Gi

B. helm install kafka bitnami/kafka --set replicaCount=3 --set persistence.size=50Gi

C. helm install kafka bitnami/kafka --set brokers=3 --set storage.size=50Gi

D. helm install kafka bitnami/kafka --set replicaCount=3 --set persistence.storage=50Gi

Solution

1. Step 1: Set broker count correctly

   replicaCount=3 sets 3 Kafka brokers.

2. Step 2: Set storage size correctly

   Bitnami Kafka chart uses persistence.size to define storage size per broker.

3. Step 3: Use correct Helm syntax

   Separate --set flags or comma-separated values are valid, but helm install kafka bitnami/kafka --set replicaCount=3 --set persistence.size=50Gi uses clear separate flags.

4. Final Answer:

   helm install kafka bitnami/kafka --set replicaCount=3 --set persistence.size=50Gi -> Option B

5. Quick Check:

   replicaCount and persistence.size set correctly = C [OK]

Hint: Use replicaCount and persistence.size keys for brokers and storage [OK]

Common Mistakes:

• Using wrong keys like brokers or storage.size
• Combining keys incorrectly in one --set
• Using persistence.storage instead of persistence.size