ZooKeeper role (and KRaft replacement) in Kafka - Commands & Configuration

Introduction

Kafka uses ZooKeeper to manage cluster metadata and coordinate brokers. KRaft mode replaces ZooKeeper by embedding this coordination inside Kafka itself, simplifying setup and improving reliability.

When you want to run a Kafka cluster with external coordination using ZooKeeper.

When you want to simplify Kafka cluster management by using KRaft mode without ZooKeeper.

When setting up a new Kafka cluster and deciding between traditional ZooKeeper or KRaft mode.

When upgrading Kafka and considering migration from ZooKeeper to KRaft mode.

When troubleshooting Kafka cluster metadata or broker coordination issues.

Config File - kafka.properties

kafka.properties

process.roles=broker,controller
node.id=1
controller.quorum.voters=1@localhost:9093
listeners=PLAINTEXT://localhost:9092,CONTROLLER://localhost:9093
inter.broker.listener.name=PLAINTEXT
log.dirs=/tmp/kraft-logs
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connect=

process.roles: Defines the roles this Kafka node will play, including broker and controller for KRaft mode.

node.id: Unique identifier for this Kafka node.

controller.quorum.voters: Defines the controller nodes and their IDs and addresses for quorum.

listeners: Network addresses for client and controller communication.

inter.broker.listener.name: Listener used for broker-to-broker communication.

log.dirs: Directory where Kafka stores logs.

zookeeper.connect: Empty because KRaft mode does not use ZooKeeper.

Commands

Starts the Kafka broker in KRaft mode using the configuration file that defines roles and listeners.

Terminal

bin/kafka-server-start.sh config/kafka.properties

Expected Output

[2024-06-01 12:00:00,000] INFO Kafka version: 3.5.0 (org.apache.kafka.common.utils.AppInfoParser)

[2024-06-01 12:00:00,100] INFO Kafka startTimeMs: 1685611200000 (org.apache.kafka.common.utils.AppInfoParser)

[2024-06-01 12:00:01,000] INFO Started Kafka server (kafka.server.KafkaServer) (kafka.server.KafkaServer)

[2024-06-01 12:00:01,500] INFO Controller quorum started with voters: 1@localhost:9093 (kafka.controller.QuorumController)

Creates a new topic named 'example-topic' with 3 partitions and replication factor 1 on the Kafka cluster running in KRaft mode.

Terminal

bin/kafka-topics.sh --bootstrap-server localhost:9092 --create --topic example-topic --partitions 3 --replication-factor 1

Expected Output

Created topic example-topic.

→

--bootstrap-server - Specifies the Kafka broker address to connect to.

→

--create - Creates a new topic.

→

--partitions - Sets the number of partitions for the topic.

→

--replication-factor - Sets the replication factor for the topic.

Lists all topics currently available in the Kafka cluster to verify the topic creation.

Terminal

bin/kafka-topics.sh --bootstrap-server localhost:9092 --list

Expected Output

example-topic

→

--bootstrap-server - Specifies the Kafka broker address to connect to.

→

--list - Lists all topics.

Key Concept

If you remember nothing else from this pattern, remember: KRaft mode removes the need for ZooKeeper by embedding cluster metadata management inside Kafka itself.

Common Mistakes

Trying to start Kafka in KRaft mode without setting process.roles to include 'controller'.

Kafka won't start the controller quorum, causing cluster coordination failure.

Always include 'controller' in process.roles in the configuration file when using KRaft mode.

Leaving zookeeper.connect configured with a ZooKeeper address when using KRaft mode.

Kafka will attempt to connect to ZooKeeper, causing startup errors or confusion since KRaft mode does not use ZooKeeper.

Set zookeeper.connect to empty or remove it entirely in KRaft mode configurations.

Using old Kafka commands or scripts that expect ZooKeeper connection parameters.

Commands may fail or behave unexpectedly because KRaft mode uses bootstrap-server instead of zookeeper.connect.

Use commands with --bootstrap-server flag and avoid ZooKeeper-specific flags when working with KRaft mode.

Summary

Configure Kafka with process.roles including 'controller' and 'broker' to enable KRaft mode.

Start Kafka using the configuration file that defines KRaft settings, removing the need for ZooKeeper.

Use Kafka CLI commands with --bootstrap-server to manage topics and verify cluster status.

Practice

1. What is the main role of ZooKeeper in a Kafka cluster?

easy

A. To store Kafka messages permanently

B. To replace Kafka brokers in the cluster

C. To serve as a Kafka client for producers

D. To manage cluster coordination and metadata

Solution

1. Step 1: Understand ZooKeeper's purpose

   ZooKeeper is used to manage coordination tasks like leader election and metadata storage in Kafka clusters.

2. Step 2: Identify what ZooKeeper does not do

   ZooKeeper does not store Kafka messages or act as a client or broker replacement.

3. Final Answer:

   To manage cluster coordination and metadata -> Option D

4. Quick Check:

   ZooKeeper = cluster coordination [OK]

Hint: ZooKeeper manages coordination, not message storage [OK]

Common Mistakes:

• Thinking ZooKeeper stores Kafka messages
• Confusing ZooKeeper with Kafka brokers
• Assuming ZooKeeper acts as a client

2. Which of the following is the correct way to enable KRaft mode in Kafka's configuration?

easy

A. set kafka.broker.only=true

B. set kafka.zk.enabled=true

C. set process.roles=controller,broker

D. set zookeeper.connect=localhost:2181

Solution

1. Step 1: Identify KRaft configuration property

   KRaft mode requires setting the process.roles property to include controller and broker roles.

2. Step 2: Eliminate incorrect options

   ZooKeeper-related settings like kafka.zk.enabled=true and zookeeper.connect=localhost:2181 are not for KRaft. kafka.broker.only=true is not a valid Kafka config.

3. Final Answer:

   set process.roles=controller,broker -> Option C

4. Quick Check:

   process.roles = controller,broker [OK]

Hint: KRaft uses process.roles, not zookeeper.connect [OK]

Common Mistakes:

• Confusing ZooKeeper settings with KRaft
• Using deprecated or invalid config keys
• Assuming KRaft disables brokers

3. Given the following Kafka broker configuration snippet in KRaft mode:

process.roles=broker,controller
node.id=1
controller.quorum.voters=1@localhost:9093

What is the role of this broker?

medium

A. It acts as both broker and controller

B. It acts only as a controller, not a broker

C. It acts neither as broker nor controller

D. It acts only as a broker, not a controller

Solution

1. Step 1: Analyze process.roles value

   The value 'broker,controller' means this node acts as both broker and controller.

2. Step 2: Confirm node.id and quorum voters

   node.id=1 and controller.quorum.voters=1@localhost:9093 confirm this node is part of the controller quorum.

3. Final Answer:

   It acts as both broker and controller -> Option A

4. Quick Check:

   process.roles includes both roles [OK]

Hint: process.roles lists all roles this node performs [OK]

Common Mistakes:

• Ignoring multiple roles in process.roles
• Confusing node.id with role type
• Assuming controller.quorum.voters disables broker role

4. You try to start a Kafka cluster in KRaft mode but get an error about missing controller quorum voters. Which configuration is missing or incorrect?

medium

A. Incorrect 'process.roles' without 'broker'

B. Missing or incorrect 'controller.quorum.voters' setting

C. Missing 'zookeeper.connect' setting

D. Missing 'log.dirs' setting

Solution

1. Step 1: Understand error context

   KRaft mode requires 'controller.quorum.voters' to define controller nodes.

2. Step 2: Identify missing or wrong config

   If this setting is missing or wrong, Kafka cannot form the controller quorum, causing startup errors.

3. Final Answer:

   Missing or incorrect 'controller.quorum.voters' setting -> Option B

4. Quick Check:

   controller.quorum.voters required in KRaft [OK]

Hint: KRaft needs controller.quorum.voters set [OK]

Common Mistakes:

• Confusing ZooKeeper settings with KRaft
• Ignoring controller quorum config
• Assuming log.dirs causes quorum errors

5. You want to migrate a Kafka cluster from ZooKeeper mode to KRaft mode. Which of the following steps is essential for a successful migration?

hard

A. Configure 'process.roles' and 'controller.quorum.voters' and migrate metadata to KRaft

B. Remove all brokers and restart ZooKeeper only

C. Disable KRaft and keep using ZooKeeper indefinitely

D. Set 'zookeeper.connect' to empty string and restart brokers

Solution

1. Step 1: Understand migration requirements

   Moving to KRaft requires configuring roles and controller quorum voters and migrating metadata from ZooKeeper.

2. Step 2: Eliminate incorrect options

   The other options do not perform a proper migration to KRaft or incorrectly suggest keeping or modifying ZooKeeper.

3. Final Answer:

   Configure 'process.roles' and 'controller.quorum.voters' and migrate metadata to KRaft -> Option A

4. Quick Check:

   Migration needs roles, quorum, and metadata move [OK]

Hint: Set roles, quorum, migrate metadata for KRaft migration [OK]

Common Mistakes:

• Skipping metadata migration step
• Trying to run ZooKeeper and KRaft simultaneously
• Misconfiguring controller quorum voters