ZooKeeper role (and KRaft replacement) in Kafka - Cheat Sheet & Quick Revision

Recall & Review

beginner

What is the primary role of ZooKeeper in Apache Kafka?

ZooKeeper manages the Kafka cluster metadata, including broker information, topic configurations, and leader election for partitions. It helps keep the cluster coordinated and consistent.

beginner

How does ZooKeeper help with leader election in Kafka?

ZooKeeper keeps track of which broker is the leader for each partition. When a leader fails, ZooKeeper helps elect a new leader to maintain availability and consistency.

intermediate

What is KRaft in the context of Kafka?

KRaft (Kafka Raft) is Kafka's built-in consensus mechanism designed to replace ZooKeeper. It manages metadata internally using the Raft protocol, simplifying Kafka's architecture.

intermediate

Name one advantage of KRaft over ZooKeeper in Kafka.

KRaft removes the need for an external ZooKeeper service, reducing operational complexity and improving scalability by managing metadata within Kafka brokers themselves.

advanced

What happens to Kafka's metadata management when switching from ZooKeeper to KRaft?

Kafka brokers themselves form a quorum using the Raft protocol to manage metadata, eliminating the external dependency on ZooKeeper and enabling a more streamlined cluster management.

What does ZooKeeper primarily manage in a Kafka cluster?

A Cluster metadata and leader election

B Message storage

C Consumer offset tracking only

D Network communication

ZooKeeper manages cluster metadata and helps with leader election in Kafka.

What protocol does KRaft use to manage Kafka metadata internally?

A Paxos

B Raft

C Two-phase commit

D Gossip

KRaft uses the Raft consensus protocol to manage metadata inside Kafka brokers.

Which of the following is a benefit of using KRaft over ZooKeeper?

A Removes external dependency on ZooKeeper

B Increases operational complexity

C Requires an external service

D Slower metadata updates

KRaft removes the need for an external ZooKeeper service, simplifying Kafka's operation.

In Kafka with ZooKeeper, what triggers a new leader election?

A Message production

B New topic creation

C Consumer joining

D Broker failure

When a broker fails, ZooKeeper helps elect a new leader for the affected partitions.

Which Kafka component is responsible for metadata management in KRaft mode?

A ZooKeeper

B Kafka producers

C Kafka brokers forming a quorum

D Kafka consumers

In KRaft mode, Kafka brokers themselves form a quorum to manage metadata.

Explain the role of ZooKeeper in Kafka and why Kafka is moving towards KRaft.

Think about how Kafka manages metadata and what problems KRaft solves.

Describe how KRaft changes the architecture of Kafka compared to using ZooKeeper.

Focus on the shift from external to internal metadata management.

Practice

1. What is the main role of ZooKeeper in a Kafka cluster?

easy

A. To store Kafka messages permanently

B. To replace Kafka brokers in the cluster

C. To serve as a Kafka client for producers

D. To manage cluster coordination and metadata

Solution

1. Step 1: Understand ZooKeeper's purpose

   ZooKeeper is used to manage coordination tasks like leader election and metadata storage in Kafka clusters.

2. Step 2: Identify what ZooKeeper does not do

   ZooKeeper does not store Kafka messages or act as a client or broker replacement.

3. Final Answer:

   To manage cluster coordination and metadata -> Option D

4. Quick Check:

   ZooKeeper = cluster coordination [OK]

Hint: ZooKeeper manages coordination, not message storage [OK]

Common Mistakes:

• Thinking ZooKeeper stores Kafka messages
• Confusing ZooKeeper with Kafka brokers
• Assuming ZooKeeper acts as a client

2. Which of the following is the correct way to enable KRaft mode in Kafka's configuration?

easy

A. set kafka.broker.only=true

B. set kafka.zk.enabled=true

C. set process.roles=controller,broker

D. set zookeeper.connect=localhost:2181

Solution

1. Step 1: Identify KRaft configuration property

   KRaft mode requires setting the process.roles property to include controller and broker roles.

2. Step 2: Eliminate incorrect options

   ZooKeeper-related settings like kafka.zk.enabled=true and zookeeper.connect=localhost:2181 are not for KRaft. kafka.broker.only=true is not a valid Kafka config.

3. Final Answer:

   set process.roles=controller,broker -> Option C

4. Quick Check:

   process.roles = controller,broker [OK]

Hint: KRaft uses process.roles, not zookeeper.connect [OK]

Common Mistakes:

• Confusing ZooKeeper settings with KRaft
• Using deprecated or invalid config keys
• Assuming KRaft disables brokers

3. Given the following Kafka broker configuration snippet in KRaft mode:

process.roles=broker,controller
node.id=1
controller.quorum.voters=1@localhost:9093

What is the role of this broker?

medium

A. It acts as both broker and controller

B. It acts only as a controller, not a broker

C. It acts neither as broker nor controller

D. It acts only as a broker, not a controller

Solution

1. Step 1: Analyze process.roles value

   The value 'broker,controller' means this node acts as both broker and controller.

2. Step 2: Confirm node.id and quorum voters

   node.id=1 and controller.quorum.voters=1@localhost:9093 confirm this node is part of the controller quorum.

3. Final Answer:

   It acts as both broker and controller -> Option A

4. Quick Check:

   process.roles includes both roles [OK]

Hint: process.roles lists all roles this node performs [OK]

Common Mistakes:

• Ignoring multiple roles in process.roles
• Confusing node.id with role type
• Assuming controller.quorum.voters disables broker role

4. You try to start a Kafka cluster in KRaft mode but get an error about missing controller quorum voters. Which configuration is missing or incorrect?

medium

A. Incorrect 'process.roles' without 'broker'

B. Missing or incorrect 'controller.quorum.voters' setting

C. Missing 'zookeeper.connect' setting

D. Missing 'log.dirs' setting

Solution

1. Step 1: Understand error context

   KRaft mode requires 'controller.quorum.voters' to define controller nodes.

2. Step 2: Identify missing or wrong config

   If this setting is missing or wrong, Kafka cannot form the controller quorum, causing startup errors.

3. Final Answer:

   Missing or incorrect 'controller.quorum.voters' setting -> Option B

4. Quick Check:

   controller.quorum.voters required in KRaft [OK]

Hint: KRaft needs controller.quorum.voters set [OK]

Common Mistakes:

• Confusing ZooKeeper settings with KRaft
• Ignoring controller quorum config
• Assuming log.dirs causes quorum errors

5. You want to migrate a Kafka cluster from ZooKeeper mode to KRaft mode. Which of the following steps is essential for a successful migration?

hard

A. Configure 'process.roles' and 'controller.quorum.voters' and migrate metadata to KRaft

B. Remove all brokers and restart ZooKeeper only

C. Disable KRaft and keep using ZooKeeper indefinitely

D. Set 'zookeeper.connect' to empty string and restart brokers

Solution

1. Step 1: Understand migration requirements

   Moving to KRaft requires configuring roles and controller quorum voters and migrating metadata from ZooKeeper.

2. Step 2: Eliminate incorrect options

   The other options do not perform a proper migration to KRaft or incorrectly suggest keeping or modifying ZooKeeper.

3. Final Answer:

   Configure 'process.roles' and 'controller.quorum.voters' and migrate metadata to KRaft -> Option A

4. Quick Check:

   Migration needs roles, quorum, and metadata move [OK]

Hint: Set roles, quorum, migrate metadata for KRaft migration [OK]

Common Mistakes:

• Skipping metadata migration step
• Trying to run ZooKeeper and KRaft simultaneously
• Misconfiguring controller quorum voters