Kafka with Zookeeper vs KRaft: Key Differences and When to Use Each

Kafka traditionally uses Zookeeper to manage cluster metadata and coordination, while the newer KRaft mode removes this dependency by integrating metadata management directly into Kafka brokers. KRaft simplifies setup and improves scalability by eliminating the external Zookeeper service.

Quick Comparison

This table summarizes the main differences between Kafka with Zookeeper and Kafka with KRaft mode.

Factor	Kafka with Zookeeper	Kafka with KRaft
Metadata Management	External Zookeeper ensemble	Built-in Kafka quorum controller
Setup Complexity	Requires separate Zookeeper cluster	No external dependency, simpler setup
Scalability	Limited by Zookeeper performance	Improved scalability with Kafka-native controller
Fault Tolerance	Depends on Zookeeper ensemble health	Controller quorum handles failover internally
Maturity	Battle-tested and stable	Newer, stable since Kafka 3.3+
Operational Overhead	More components to monitor and maintain	Fewer components, easier maintenance

Key Differences

Kafka with Zookeeper uses an external Zookeeper service to store metadata like topic configurations, partition assignments, and broker information. This means you must run and maintain a separate Zookeeper cluster alongside Kafka brokers. Zookeeper handles leader election and cluster coordination, but adds operational complexity and potential bottlenecks.

KRaft mode, introduced as a replacement starting Kafka 2.8 and stable since 3.3, integrates metadata management directly into Kafka brokers using a quorum controller. This removes the need for Zookeeper, simplifying deployment and reducing latency in metadata operations. The Kafka brokers themselves form a controller quorum that manages cluster state and failover.

While Kafka with Zookeeper is mature and widely used, KRaft offers better scalability and easier operations by eliminating an external dependency. However, migrating existing clusters requires careful planning since KRaft uses a different metadata storage mechanism.

Code Comparison

Here is a simple example of starting Kafka with Zookeeper mode using a basic configuration.

properties

zookeeper.connect=localhost:2181
broker.id=1
listeners=PLAINTEXT://:9092
log.dirs=/tmp/kafka-logs

Output

Kafka broker starts and connects to Zookeeper at localhost:2181 to register itself and manage metadata.

KRaft Equivalent

Here is the equivalent configuration snippet to start Kafka in KRaft mode without Zookeeper.

properties

process.roles=broker,controller
node.id=1
controller.quorum.voters=1@localhost:9093
listeners=PLAINTEXT://:9092,CONTROLLER://:9093
log.dirs=/tmp/kafka-logs

Output

Kafka broker starts as both broker and controller, managing metadata internally without Zookeeper.

When to Use Which

Choose Kafka with Zookeeper if you have an existing stable cluster or require mature tooling and ecosystem support. It is battle-tested and widely supported in production.

Choose KRaft mode for new Kafka deployments to benefit from simpler setup, fewer components, and better scalability. It reduces operational overhead and is the future direction of Kafka metadata management.

For migration, plan carefully as switching from Zookeeper to KRaft requires metadata migration and downtime.

Key Takeaways

Kafka with Zookeeper requires an external service for metadata, adding complexity.

KRaft mode integrates metadata management into Kafka brokers, simplifying operations.

KRaft offers better scalability and easier maintenance for new Kafka clusters.

Use Zookeeper mode for existing stable clusters; use KRaft for new deployments.

Migrating from Zookeeper to KRaft needs careful planning and downtime.