ZooKeeper role (and KRaft replacement) in Kafka - Commands & Configuration

Introduction

Kafka uses ZooKeeper to manage cluster metadata and coordinate brokers. KRaft mode replaces ZooKeeper by embedding this coordination inside Kafka itself, simplifying setup and improving reliability.

When you want to run a Kafka cluster with external coordination using ZooKeeper.

When you want to simplify Kafka cluster management by using KRaft mode without ZooKeeper.

When setting up a new Kafka cluster and deciding between traditional ZooKeeper or KRaft mode.

When upgrading Kafka and considering migration from ZooKeeper to KRaft mode.

When troubleshooting Kafka cluster metadata or broker coordination issues.

Config File - kafka.properties

kafka.properties

process.roles=broker,controller
node.id=1
controller.quorum.voters=1@localhost:9093
listeners=PLAINTEXT://localhost:9092,CONTROLLER://localhost:9093
inter.broker.listener.name=PLAINTEXT
log.dirs=/tmp/kraft-logs
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connect=

process.roles: Defines the roles this Kafka node will play, including broker and controller for KRaft mode.

node.id: Unique identifier for this Kafka node.

controller.quorum.voters: Defines the controller nodes and their IDs and addresses for quorum.

listeners: Network addresses for client and controller communication.

inter.broker.listener.name: Listener used for broker-to-broker communication.

log.dirs: Directory where Kafka stores logs.

zookeeper.connect: Empty because KRaft mode does not use ZooKeeper.

Commands

Starts the Kafka broker in KRaft mode using the configuration file that defines roles and listeners.

Terminal

bin/kafka-server-start.sh config/kafka.properties

Expected Output

[2024-06-01 12:00:00,000] INFO Kafka version: 3.5.0 (org.apache.kafka.common.utils.AppInfoParser)

[2024-06-01 12:00:00,100] INFO Kafka startTimeMs: 1685611200000 (org.apache.kafka.common.utils.AppInfoParser)

[2024-06-01 12:00:01,000] INFO Started Kafka server (kafka.server.KafkaServer) (kafka.server.KafkaServer)

[2024-06-01 12:00:01,500] INFO Controller quorum started with voters: 1@localhost:9093 (kafka.controller.QuorumController)

Creates a new topic named 'example-topic' with 3 partitions and replication factor 1 on the Kafka cluster running in KRaft mode.

Terminal

bin/kafka-topics.sh --bootstrap-server localhost:9092 --create --topic example-topic --partitions 3 --replication-factor 1

Expected Output

Created topic example-topic.

→

--bootstrap-server - Specifies the Kafka broker address to connect to.

→

--create - Creates a new topic.

→

--partitions - Sets the number of partitions for the topic.

→

--replication-factor - Sets the replication factor for the topic.

Lists all topics currently available in the Kafka cluster to verify the topic creation.

Terminal

bin/kafka-topics.sh --bootstrap-server localhost:9092 --list

Expected Output

example-topic

→

--bootstrap-server - Specifies the Kafka broker address to connect to.

→

--list - Lists all topics.

Key Concept

If you remember nothing else from this pattern, remember: KRaft mode removes the need for ZooKeeper by embedding cluster metadata management inside Kafka itself.

Common Mistakes

Trying to start Kafka in KRaft mode without setting process.roles to include 'controller'.

Kafka won't start the controller quorum, causing cluster coordination failure.

Always include 'controller' in process.roles in the configuration file when using KRaft mode.

Leaving zookeeper.connect configured with a ZooKeeper address when using KRaft mode.

Kafka will attempt to connect to ZooKeeper, causing startup errors or confusion since KRaft mode does not use ZooKeeper.

Set zookeeper.connect to empty or remove it entirely in KRaft mode configurations.

Using old Kafka commands or scripts that expect ZooKeeper connection parameters.

Commands may fail or behave unexpectedly because KRaft mode uses bootstrap-server instead of zookeeper.connect.

Use commands with --bootstrap-server flag and avoid ZooKeeper-specific flags when working with KRaft mode.

Summary

Configure Kafka with process.roles including 'controller' and 'broker' to enable KRaft mode.

Start Kafka using the configuration file that defines KRaft settings, removing the need for ZooKeeper.

Use Kafka CLI commands with --bootstrap-server to manage topics and verify cluster status.