Why ZooKeeper role (and KRaft replacement) in Kafka? - Purpose & Use Cases

The Big Idea

What if your entire system could coordinate itself perfectly without messy notes or confusion?

The Scenario

Imagine you run a busy zoo where many animals need care and feeding. You try to keep track of who fed which animal and when by writing notes on paper. But as the zoo grows, papers get lost, and different keepers get confused about who did what.

The Problem

Using paper notes or separate lists to coordinate tasks is slow and error-prone. Keepers might feed animals twice or miss feeding altogether. It's hard to keep everyone in sync, especially when many people work at once.

The Solution

ZooKeeper acts like a smart, shared notebook that all keepers can access safely. It keeps track of tasks and states in one place, so everyone knows what's done and what's next. KRaft replaces ZooKeeper by building this coordination directly into Kafka, making the system simpler and faster.

Before vs After

✗ Before

keeper1_notes = ['fed lion at 9am']
keeper2_notes = ['fed tiger at 10am']  # No shared view, easy to miss or duplicate

✓ After

zookeeper.store('fed_lion', '9am')
zookeeper.store('fed_tiger', '10am')  # Central place everyone reads and writes

What It Enables

It enables reliable coordination and state management across many servers, so Kafka runs smoothly without confusion or data loss.

Real Life Example

In a large online store, many servers handle orders. ZooKeeper or KRaft ensures all servers agree on who processes which order, preventing mistakes like double shipments or lost orders.

Key Takeaways

Manual coordination is slow and error-prone in distributed systems.

ZooKeeper provides a central, reliable way to manage shared state and coordination.

KRaft replaces ZooKeeper by integrating this coordination inside Kafka for better simplicity and performance.