ZooKeeper role (and KRaft replacement) in Kafka - Mini Project: Build & Apply

ZooKeeper Role and KRaft Replacement in Kafka

📖 Scenario: You are managing a Kafka cluster that currently uses ZooKeeper for metadata management. Kafka is moving towards using KRaft mode, which replaces ZooKeeper. You want to understand how to configure and verify the Kafka cluster roles in both ZooKeeper and KRaft modes.

🎯 Goal: Learn how to set up Kafka metadata management with ZooKeeper and then switch to KRaft mode, understanding the role of each component and verifying the cluster state.

📋 What You'll Learn

Create a Kafka configuration file with ZooKeeper connection

Add a configuration variable to enable KRaft mode

Write a command to check the Kafka cluster metadata in ZooKeeper mode

Write a command to check the Kafka cluster metadata in KRaft mode

💡 Why This Matters

🌍 Real World

Kafka clusters currently use ZooKeeper for metadata, but newer versions use KRaft mode to simplify architecture and improve scalability.

💼 Career

Understanding Kafka metadata management is essential for Kafka administrators and DevOps engineers managing streaming data infrastructure.

1

Create Kafka configuration with ZooKeeper connection

Create a file named server.properties with the following exact content:
zookeeper.connect=localhost:2181
This sets up Kafka to use ZooKeeper for metadata management.

Kafka

# Create a file named server.properties with ZooKeeper connection
# Your code here

Hint

Use the exact line zookeeper.connect=localhost:2181 in the configuration file.

2

Add KRaft mode configuration

Add these exact lines to server.properties to enable KRaft mode:
process.roles=controller,broker
node.id=1
controller.quorum.voters=1@localhost:9093
This configures Kafka to run in KRaft mode without ZooKeeper.

Kafka

zookeeper.connect=localhost:2181
# Add KRaft mode configuration lines below
# Your code here

Hint

Include all three lines exactly as shown to enable KRaft mode.

3

Check Kafka cluster metadata in ZooKeeper mode

Write the exact command to describe Kafka topics using ZooKeeper mode:
kafka-topics.sh --describe --zookeeper localhost:2181
This command shows the metadata managed by ZooKeeper.

Kafka

# Write the command to describe topics in ZooKeeper mode
# Your code here

Hint

Use the --zookeeper localhost:2181 option to query ZooKeeper metadata.

4

Check Kafka cluster metadata in KRaft mode

Write the exact command to describe Kafka topics using KRaft mode:
kafka-topics.sh --describe --bootstrap-server localhost:9092
This command shows the metadata managed by KRaft controllers.

Kafka

# Write the command to describe topics in KRaft mode
# Your code here

Hint

Use the --bootstrap-server localhost:9092 option to query KRaft metadata.

Practice

1. What is the main role of ZooKeeper in a Kafka cluster?

easy

A. To store Kafka messages permanently

B. To replace Kafka brokers in the cluster

C. To serve as a Kafka client for producers

D. To manage cluster coordination and metadata

Solution

1. Step 1: Understand ZooKeeper's purpose

   ZooKeeper is used to manage coordination tasks like leader election and metadata storage in Kafka clusters.

2. Step 2: Identify what ZooKeeper does not do

   ZooKeeper does not store Kafka messages or act as a client or broker replacement.

3. Final Answer:

   To manage cluster coordination and metadata -> Option D

4. Quick Check:

   ZooKeeper = cluster coordination [OK]

Hint: ZooKeeper manages coordination, not message storage [OK]

Common Mistakes:

• Thinking ZooKeeper stores Kafka messages
• Confusing ZooKeeper with Kafka brokers
• Assuming ZooKeeper acts as a client

2. Which of the following is the correct way to enable KRaft mode in Kafka's configuration?

easy

A. set kafka.broker.only=true

B. set kafka.zk.enabled=true

C. set process.roles=controller,broker

D. set zookeeper.connect=localhost:2181

Solution

1. Step 1: Identify KRaft configuration property

   KRaft mode requires setting the process.roles property to include controller and broker roles.

2. Step 2: Eliminate incorrect options

   ZooKeeper-related settings like kafka.zk.enabled=true and zookeeper.connect=localhost:2181 are not for KRaft. kafka.broker.only=true is not a valid Kafka config.

3. Final Answer:

   set process.roles=controller,broker -> Option C

4. Quick Check:

   process.roles = controller,broker [OK]

Hint: KRaft uses process.roles, not zookeeper.connect [OK]

Common Mistakes:

• Confusing ZooKeeper settings with KRaft
• Using deprecated or invalid config keys
• Assuming KRaft disables brokers

3. Given the following Kafka broker configuration snippet in KRaft mode:

process.roles=broker,controller
node.id=1
controller.quorum.voters=1@localhost:9093

What is the role of this broker?

medium

A. It acts as both broker and controller

B. It acts only as a controller, not a broker

C. It acts neither as broker nor controller

D. It acts only as a broker, not a controller

Solution

1. Step 1: Analyze process.roles value

   The value 'broker,controller' means this node acts as both broker and controller.

2. Step 2: Confirm node.id and quorum voters

   node.id=1 and controller.quorum.voters=1@localhost:9093 confirm this node is part of the controller quorum.

3. Final Answer:

   It acts as both broker and controller -> Option A

4. Quick Check:

   process.roles includes both roles [OK]

Hint: process.roles lists all roles this node performs [OK]

Common Mistakes:

• Ignoring multiple roles in process.roles
• Confusing node.id with role type
• Assuming controller.quorum.voters disables broker role

4. You try to start a Kafka cluster in KRaft mode but get an error about missing controller quorum voters. Which configuration is missing or incorrect?

medium

A. Incorrect 'process.roles' without 'broker'

B. Missing or incorrect 'controller.quorum.voters' setting

C. Missing 'zookeeper.connect' setting

D. Missing 'log.dirs' setting

Solution

1. Step 1: Understand error context

   KRaft mode requires 'controller.quorum.voters' to define controller nodes.

2. Step 2: Identify missing or wrong config

   If this setting is missing or wrong, Kafka cannot form the controller quorum, causing startup errors.

3. Final Answer:

   Missing or incorrect 'controller.quorum.voters' setting -> Option B

4. Quick Check:

   controller.quorum.voters required in KRaft [OK]

Hint: KRaft needs controller.quorum.voters set [OK]

Common Mistakes:

• Confusing ZooKeeper settings with KRaft
• Ignoring controller quorum config
• Assuming log.dirs causes quorum errors

5. You want to migrate a Kafka cluster from ZooKeeper mode to KRaft mode. Which of the following steps is essential for a successful migration?

hard

A. Configure 'process.roles' and 'controller.quorum.voters' and migrate metadata to KRaft

B. Remove all brokers and restart ZooKeeper only

C. Disable KRaft and keep using ZooKeeper indefinitely

D. Set 'zookeeper.connect' to empty string and restart brokers

Solution

1. Step 1: Understand migration requirements

   Moving to KRaft requires configuring roles and controller quorum voters and migrating metadata from ZooKeeper.

2. Step 2: Eliminate incorrect options

   The other options do not perform a proper migration to KRaft or incorrectly suggest keeping or modifying ZooKeeper.

3. Final Answer:

   Configure 'process.roles' and 'controller.quorum.voters' and migrate metadata to KRaft -> Option A

4. Quick Check:

   Migration needs roles, quorum, and metadata move [OK]

Hint: Set roles, quorum, migrate metadata for KRaft migration [OK]

Common Mistakes:

• Skipping metadata migration step
• Trying to run ZooKeeper and KRaft simultaneously
• Misconfiguring controller quorum voters