ZooKeeper role (and KRaft replacement) in Kafka - Step-by-Step Execution

Process Flow - ZooKeeper role (and KRaft replacement)

Kafka Broker starts

↓

Connect to ZooKeeper?

No→Use KRaft mode

|Yes↓

Register Broker in ZooKeeper

↓

ZooKeeper manages cluster metadata

↓

Brokers coordinate via ZooKeeper

↓

Client requests handled

↓

Cluster state updated in ZooKeeper

↓

If KRaft mode

↓

Kafka manages metadata internally

↓

Cluster operates without ZooKeeper

Kafka brokers either connect to ZooKeeper to manage cluster metadata or use KRaft mode where Kafka manages metadata internally without ZooKeeper.

Execution Sample

Kafka

1. Broker starts
2. Connect to ZooKeeper
3. Register broker
4. ZooKeeper manages metadata
5. Brokers coordinate
6. Client requests processed

This sequence shows how Kafka brokers interact with ZooKeeper to manage cluster metadata and coordinate client requests.

Process Table

Step	Action	ZooKeeper Interaction	Metadata State	Cluster Coordination
1	Broker starts	No	No metadata	No coordination
2	Connect to ZooKeeper	Connects	Metadata access begins	Preparation for coordination
3	Register broker	Registers broker info	Metadata updated with broker	Broker joins cluster
4	ZooKeeper manages metadata	Maintains cluster state	Metadata consistent	Brokers aware of cluster
5	Brokers coordinate	Use ZooKeeper info	Metadata stable	Cluster coordination active
6	Client requests processed	ZooKeeper used indirectly	Metadata used for routing	Requests handled correctly
7	If KRaft mode	No ZooKeeper connection	Kafka manages metadata	Cluster coordination internal
8	Cluster operates	ZooKeeper replaced	Metadata consistent internally	Cluster fully coordinated
9	Exit	ZooKeeper role ends or replaced	Metadata managed	Cluster stable

💡 Execution stops when cluster is fully coordinated either via ZooKeeper or KRaft mode.

Status Tracker

Variable	Start	After Step 2	After Step 3	After Step 4	After Step 5	After Step 6	After Step 7	After Step 8	Final
ZooKeeper Connection	No	Yes	Yes	Yes	Yes	Yes	No (KRaft)	No (KRaft)	No (KRaft)
Metadata State	None	Access begins	Broker registered	Consistent	Stable	Used for routing	Kafka internal	Consistent internal	Consistent internal
Cluster Coordination	None	Preparing	Broker joined	Active	Active	Requests handled	Internal coordination	Fully coordinated	Stable

Key Moments - 3 Insights

Why does Kafka need ZooKeeper before KRaft?

Before KRaft, Kafka uses ZooKeeper to store and manage cluster metadata and broker coordination, as shown in steps 2 to 6 in the execution_table.

What changes in cluster coordination when switching to KRaft?

In KRaft mode (steps 7 and 8), Kafka manages metadata internally without ZooKeeper, so cluster coordination happens inside Kafka itself.

Does ZooKeeper remain connected during KRaft mode?

No, as shown in variable_tracker, ZooKeeper connection is dropped after step 6 and replaced by Kafka's internal metadata management.

Visual Quiz - 3 Questions

Look at the execution_table, at which step does the broker register with ZooKeeper?

A Step 2

B Step 5

C Step 3

D Step 7

💡 Hint

Check the 'Action' and 'ZooKeeper Interaction' columns in execution_table row for Step 3.

According to variable_tracker, when does Kafka stop connecting to ZooKeeper?

A After Step 6

B After Step 4

C After Step 8

D At Start

💡 Hint

Look at the 'ZooKeeper Connection' row and see when it changes from 'Yes' to 'No'.

If Kafka did not register the broker in ZooKeeper, what would happen to cluster coordination?

A Coordination would be active

B Coordination would fail or be incomplete

C Metadata would be consistent

D Client requests would be processed normally

💡 Hint

Refer to execution_table steps 3 and 4 where registration affects metadata and coordination.

Concept Snapshot

Kafka brokers use ZooKeeper to manage cluster metadata and coordination.
ZooKeeper stores broker info and cluster state.
KRaft mode replaces ZooKeeper by managing metadata internally.
Cluster coordination depends on metadata availability.
Switching to KRaft removes ZooKeeper dependency.

Full Transcript

This visual execution shows how Kafka brokers start and connect to ZooKeeper to register themselves and manage cluster metadata. ZooKeeper keeps the cluster state consistent and helps brokers coordinate. Client requests are routed using this metadata. When Kafka switches to KRaft mode, it no longer connects to ZooKeeper. Instead, Kafka manages metadata internally, coordinating the cluster without ZooKeeper. Variables like ZooKeeper connection, metadata state, and cluster coordination change step-by-step, showing the transition from ZooKeeper to KRaft. Key moments clarify why ZooKeeper was needed before and how KRaft replaces it. The quiz tests understanding of broker registration, ZooKeeper connection timing, and the importance of registration for coordination.