Why State stores in Kafka? - Purpose & Use Cases

The Big Idea

What if you could track live data instantly without slow database calls?

The Scenario

Imagine you are building a system that processes streams of data, like tracking user clicks or orders in real time. Without state stores, you have to manually keep track of all past events in external databases or files, constantly fetching and updating data as new events arrive.

The Problem

This manual approach is slow and error-prone because every time you want to know the current count or status, you must query an external database, which adds delay and complexity. Also, managing consistency and updates yourself can lead to bugs and lost data.

The Solution

State stores in Kafka let you keep track of data directly inside your stream processing application. They provide fast, local storage that updates automatically as new events come in, so you can quickly access and update the current state without extra database calls.

Before vs After

✗ Before

count = db.query('SELECT count FROM clicks WHERE user_id=123')
count += 1
db.update('UPDATE clicks SET count=? WHERE user_id=123', count)

✓ After

store = context.getStateStore('clickCounts')
count = store.get(user_id) or 0
store.put(user_id, count + 1)

What It Enables

State stores enable real-time, efficient, and reliable tracking of data changes directly within your streaming application, making complex event processing smooth and scalable.

Real Life Example

For example, an online shop can use state stores to keep a live count of items in users' shopping carts as they add or remove products, without needing to query a database every time.

Key Takeaways

Manual tracking of streaming data is slow and complex.

State stores keep data locally and update it automatically.

This makes real-time data processing faster and more reliable.