ZooKeeper role (and KRaft replacement) in Kafka - Step-by-Step Execution

Process Flow - ZooKeeper role (and KRaft replacement)

Kafka Broker starts

↓

Connect to ZooKeeper?

No→Use KRaft mode

|Yes↓

Register Broker in ZooKeeper

↓

ZooKeeper manages cluster metadata

↓

Brokers coordinate via ZooKeeper

↓

Client requests handled

↓

Cluster state updated in ZooKeeper

↓

If KRaft mode

↓

Kafka manages metadata internally

↓

Cluster operates without ZooKeeper

Kafka brokers either connect to ZooKeeper to manage cluster metadata or use KRaft mode where Kafka manages metadata internally without ZooKeeper.

Execution Sample

Kafka

1. Broker starts
2. Connect to ZooKeeper
3. Register broker
4. ZooKeeper manages metadata
5. Brokers coordinate
6. Client requests processed

This sequence shows how Kafka brokers interact with ZooKeeper to manage cluster metadata and coordinate client requests.

Process Table

Step	Action	ZooKeeper Interaction	Metadata State	Cluster Coordination
1	Broker starts	No	No metadata	No coordination
2	Connect to ZooKeeper	Connects	Metadata access begins	Preparation for coordination
3	Register broker	Registers broker info	Metadata updated with broker	Broker joins cluster
4	ZooKeeper manages metadata	Maintains cluster state	Metadata consistent	Brokers aware of cluster
5	Brokers coordinate	Use ZooKeeper info	Metadata stable	Cluster coordination active
6	Client requests processed	ZooKeeper used indirectly	Metadata used for routing	Requests handled correctly
7	If KRaft mode	No ZooKeeper connection	Kafka manages metadata	Cluster coordination internal
8	Cluster operates	ZooKeeper replaced	Metadata consistent internally	Cluster fully coordinated
9	Exit	ZooKeeper role ends or replaced	Metadata managed	Cluster stable

💡 Execution stops when cluster is fully coordinated either via ZooKeeper or KRaft mode.

Status Tracker

Variable	Start	After Step 2	After Step 3	After Step 4	After Step 5	After Step 6	After Step 7	After Step 8	Final
ZooKeeper Connection	No	Yes	Yes	Yes	Yes	Yes	No (KRaft)	No (KRaft)	No (KRaft)
Metadata State	None	Access begins	Broker registered	Consistent	Stable	Used for routing	Kafka internal	Consistent internal	Consistent internal
Cluster Coordination	None	Preparing	Broker joined	Active	Active	Requests handled	Internal coordination	Fully coordinated	Stable

Key Moments - 3 Insights

Why does Kafka need ZooKeeper before KRaft?

Before KRaft, Kafka uses ZooKeeper to store and manage cluster metadata and broker coordination, as shown in steps 2 to 6 in the execution_table.

What changes in cluster coordination when switching to KRaft?

In KRaft mode (steps 7 and 8), Kafka manages metadata internally without ZooKeeper, so cluster coordination happens inside Kafka itself.

Does ZooKeeper remain connected during KRaft mode?

No, as shown in variable_tracker, ZooKeeper connection is dropped after step 6 and replaced by Kafka's internal metadata management.

Visual Quiz - 3 Questions

Look at the execution_table, at which step does the broker register with ZooKeeper?

A Step 2

B Step 5

C Step 3

D Step 7

💡 Hint

Check the 'Action' and 'ZooKeeper Interaction' columns in execution_table row for Step 3.

According to variable_tracker, when does Kafka stop connecting to ZooKeeper?

A After Step 6

B After Step 4

C After Step 8

D At Start

💡 Hint

Look at the 'ZooKeeper Connection' row and see when it changes from 'Yes' to 'No'.

If Kafka did not register the broker in ZooKeeper, what would happen to cluster coordination?

A Coordination would be active

B Coordination would fail or be incomplete

C Metadata would be consistent

D Client requests would be processed normally

💡 Hint

Refer to execution_table steps 3 and 4 where registration affects metadata and coordination.

Concept Snapshot

Kafka brokers use ZooKeeper to manage cluster metadata and coordination.
ZooKeeper stores broker info and cluster state.
KRaft mode replaces ZooKeeper by managing metadata internally.
Cluster coordination depends on metadata availability.
Switching to KRaft removes ZooKeeper dependency.

Full Transcript

This visual execution shows how Kafka brokers start and connect to ZooKeeper to register themselves and manage cluster metadata. ZooKeeper keeps the cluster state consistent and helps brokers coordinate. Client requests are routed using this metadata. When Kafka switches to KRaft mode, it no longer connects to ZooKeeper. Instead, Kafka manages metadata internally, coordinating the cluster without ZooKeeper. Variables like ZooKeeper connection, metadata state, and cluster coordination change step-by-step, showing the transition from ZooKeeper to KRaft. Key moments clarify why ZooKeeper was needed before and how KRaft replaces it. The quiz tests understanding of broker registration, ZooKeeper connection timing, and the importance of registration for coordination.

Practice

1. What is the main role of ZooKeeper in a Kafka cluster?

easy

A. To store Kafka messages permanently

B. To replace Kafka brokers in the cluster

C. To serve as a Kafka client for producers

D. To manage cluster coordination and metadata

Solution

1. Step 1: Understand ZooKeeper's purpose

   ZooKeeper is used to manage coordination tasks like leader election and metadata storage in Kafka clusters.

2. Step 2: Identify what ZooKeeper does not do

   ZooKeeper does not store Kafka messages or act as a client or broker replacement.

3. Final Answer:

   To manage cluster coordination and metadata -> Option D

4. Quick Check:

   ZooKeeper = cluster coordination [OK]

Hint: ZooKeeper manages coordination, not message storage [OK]

Common Mistakes:

• Thinking ZooKeeper stores Kafka messages
• Confusing ZooKeeper with Kafka brokers
• Assuming ZooKeeper acts as a client

2. Which of the following is the correct way to enable KRaft mode in Kafka's configuration?

easy

A. set kafka.broker.only=true

B. set kafka.zk.enabled=true

C. set process.roles=controller,broker

D. set zookeeper.connect=localhost:2181

Solution

1. Step 1: Identify KRaft configuration property

   KRaft mode requires setting the process.roles property to include controller and broker roles.

2. Step 2: Eliminate incorrect options

   ZooKeeper-related settings like kafka.zk.enabled=true and zookeeper.connect=localhost:2181 are not for KRaft. kafka.broker.only=true is not a valid Kafka config.

3. Final Answer:

   set process.roles=controller,broker -> Option C

4. Quick Check:

   process.roles = controller,broker [OK]

Hint: KRaft uses process.roles, not zookeeper.connect [OK]

Common Mistakes:

• Confusing ZooKeeper settings with KRaft
• Using deprecated or invalid config keys
• Assuming KRaft disables brokers

3. Given the following Kafka broker configuration snippet in KRaft mode:

process.roles=broker,controller
node.id=1
controller.quorum.voters=1@localhost:9093

What is the role of this broker?

medium

A. It acts as both broker and controller

B. It acts only as a controller, not a broker

C. It acts neither as broker nor controller

D. It acts only as a broker, not a controller

Solution

1. Step 1: Analyze process.roles value

   The value 'broker,controller' means this node acts as both broker and controller.

2. Step 2: Confirm node.id and quorum voters

   node.id=1 and controller.quorum.voters=1@localhost:9093 confirm this node is part of the controller quorum.

3. Final Answer:

   It acts as both broker and controller -> Option A

4. Quick Check:

   process.roles includes both roles [OK]

Hint: process.roles lists all roles this node performs [OK]

Common Mistakes:

• Ignoring multiple roles in process.roles
• Confusing node.id with role type
• Assuming controller.quorum.voters disables broker role

4. You try to start a Kafka cluster in KRaft mode but get an error about missing controller quorum voters. Which configuration is missing or incorrect?

medium

A. Incorrect 'process.roles' without 'broker'

B. Missing or incorrect 'controller.quorum.voters' setting

C. Missing 'zookeeper.connect' setting

D. Missing 'log.dirs' setting

Solution

1. Step 1: Understand error context

   KRaft mode requires 'controller.quorum.voters' to define controller nodes.

2. Step 2: Identify missing or wrong config

   If this setting is missing or wrong, Kafka cannot form the controller quorum, causing startup errors.

3. Final Answer:

   Missing or incorrect 'controller.quorum.voters' setting -> Option B

4. Quick Check:

   controller.quorum.voters required in KRaft [OK]

Hint: KRaft needs controller.quorum.voters set [OK]

Common Mistakes:

• Confusing ZooKeeper settings with KRaft
• Ignoring controller quorum config
• Assuming log.dirs causes quorum errors

5. You want to migrate a Kafka cluster from ZooKeeper mode to KRaft mode. Which of the following steps is essential for a successful migration?

hard

A. Configure 'process.roles' and 'controller.quorum.voters' and migrate metadata to KRaft

B. Remove all brokers and restart ZooKeeper only

C. Disable KRaft and keep using ZooKeeper indefinitely

D. Set 'zookeeper.connect' to empty string and restart brokers

Solution

1. Step 1: Understand migration requirements

   Moving to KRaft requires configuring roles and controller quorum voters and migrating metadata from ZooKeeper.

2. Step 2: Eliminate incorrect options

   The other options do not perform a proper migration to KRaft or incorrectly suggest keeping or modifying ZooKeeper.

3. Final Answer:

   Configure 'process.roles' and 'controller.quorum.voters' and migrate metadata to KRaft -> Option A

4. Quick Check:

   Migration needs roles, quorum, and metadata move [OK]

Hint: Set roles, quorum, migrate metadata for KRaft migration [OK]

Common Mistakes:

• Skipping metadata migration step
• Trying to run ZooKeeper and KRaft simultaneously
• Misconfiguring controller quorum voters