State stores in Kafka - Mini Project: Build & Apply

Working with State Stores in Kafka Streams

📖 Scenario: You are building a simple Kafka Streams application that counts the number of times each word appears in a stream of text messages. To keep track of the counts, you will use a state store.

🎯 Goal: Create a Kafka Streams application that uses a state store to count word occurrences and prints the counts.

📋 What You'll Learn

Create a Kafka Streams topology with a source topic named input-topic

Create a state store named word-count-store to keep track of word counts

Use a processor to update counts in the state store

Print the word counts from the state store

💡 Why This Matters

🌍 Real World

State stores in Kafka Streams let you keep track of data across messages, like counting words or tracking user sessions.

💼 Career

Understanding state stores is important for building real-time streaming applications that need to remember information between events.

1

Create the Kafka Streams topology with a source

Create a StreamsBuilder object called builder and define a source stream from the topic input-topic using builder.stream("input-topic") assigned to a variable called sourceStream.

Kafka

from kafka import KafkaProducer

# Create a StreamsBuilder and source stream
# Your code here

Hint

Use StreamsBuilder() to create the builder and builder.stream("input-topic") to get the source stream.

2

Add a state store configuration

Create a state store named word-count-store using Stores.keyValueStoreBuilder with Stores.inMemoryKeyValueStore("word-count-store") and Serdes.String() for both key and value serdes. Assign it to a variable called wordCountStore.

Kafka

from kafka.streams import Stores, Serdes

# Create a state store called wordCountStore
# Your code here

Hint

Use Stores.keyValueStoreBuilder with an in-memory store and string serdes for keys and values.

3

Attach the state store and update counts in a processor

Attach the wordCountStore to the topology using builder.addStateStore(wordCountStore). Then, use sourceStream.process() with a processor class called WordCountProcessor that updates counts in the state store. Assume WordCountProcessor is defined to read words and update counts.

Kafka

# Attach the state store and add the processor
# Your code here

Hint

Use builder.addStateStore() to add the store and sourceStream.process() with the processor class and state store list.

4

Print the word counts from the state store

Write code to access the word-count-store from the Kafka Streams context and print all word counts stored. Use store.all() to iterate over all entries and print each word and its count.

Kafka

from kafka.streams.state import ReadOnlyKeyValueStore

# Print all word counts from the state store
# Your code here

Hint

Use store.all() to get all entries and print each key and value.

Practice

1. What is the main purpose of a state store in Kafka Streams?

easy

A. To monitor Kafka cluster health

B. To send messages to Kafka topics

C. To configure Kafka brokers

D. To keep data between processing of different messages

Solution

1. Step 1: Understand state store role and compare options

   A state store holds data locally so the app can remember information across messages. Sending messages, configuring brokers, and monitoring health are unrelated to state stores.

2. Final Answer:

   To keep data between processing of different messages -> Option D

3. Quick Check:

   State store = remembers data between messages [OK]

Hint: State stores remember data across messages [OK]

Common Mistakes:

• Confusing state store with message sending
• Thinking state store configures Kafka brokers
• Assuming state store monitors cluster health

2. Which of the following is the correct way to add a persistent key-value state store named myStore to a StreamsBuilder in Kafka Streams?

easy

A. builder.addStateStore(Stores.keyValueStoreBuilder(Stores.persistentKeyValueStore("myStore"), Serdes.String(), Serdes.String()))

B. builder.addStore("myStore", persistentKeyValueStore())

C. builder.createStateStore("myStore", KeyValueStore.class)

D. builder.addKeyValueStore("myStore", Serdes.String(), Serdes.String())

Solution

1. Step 1: Recall correct syntax for adding state store and check options

   The correct method uses addStateStore with a store builder from Stores.keyValueStoreBuilder and Stores.persistentKeyValueStore. Only builder.addStateStore(Stores.keyValueStoreBuilder(Stores.persistentKeyValueStore("myStore"), Serdes.String(), Serdes.String())) matches the correct method signature and usage.

2. Final Answer:

   builder.addStateStore(Stores.keyValueStoreBuilder(Stores.persistentKeyValueStore("myStore"), Serdes.String(), Serdes.String())) -> Option A

3. Quick Check:

   Use addStateStore with keyValueStoreBuilder [OK]

Hint: Use addStateStore with keyValueStoreBuilder [OK]

Common Mistakes:

• Using incorrect method names like addStore or createStateStore
• Missing Serdes parameters for key and value
• Confusing persistent and in-memory store creation

3. Given the following Kafka Streams processor code snippet, what will be the output stored in the state store after processing the input key-value pairs ("key1", 1) and ("key1", 2)?

KeyValueStore<String, Integer> store = context.getStateStore("countStore");
Integer current = store.get("key1");
if (current == null) current = 0;
store.put("key1", current + value);

medium

A. The state store will have key "key1" with value 1

B. The state store will have key "key1" with value 2

C. The state store will have key "key1" with value 3

D. The state store will be empty

Solution

1. Step 1: Trace code execution for both inputs

   First ("key1", 1): store.get("key1") returns null, current = 0, store.put("key1", 0 + 1) = 1. Second ("key1", 2): store.get("key1") = 1, current = 1, store.put("key1", 1 + 2) = 3.

2. Final Answer:

   The state store will have key "key1" with value 3 -> Option C

3. Quick Check:

   Sum values for key1 = 3 [OK]

Hint: Add new value to existing store value for same key [OK]

Common Mistakes:

• Assuming store resets between messages
• Not handling null initial value
• Overwriting instead of adding values

4. Identify the error in this Kafka Streams processor code that tries to update a state store named myStore:

KeyValueStore<String, Long> store = context.getStateStore("myStore");
Long count = store.get(key);
count += 1L;
store.put(key, count);

medium

A. The getStateStore method is called incorrectly

B. The variable count may be null causing a NullPointerException

C. The put method should be called before incrementing count

D. The store should be cast to ReadOnlyKeyValueStore

Solution

1. Step 1: Check store.get(key) handling and increment operation

   If the key does not exist, store.get(key) returns null, so count is null. Trying to do count += 1L when count is null causes a NullPointerException.

2. Final Answer:

   The variable count may be null causing a NullPointerException -> Option B

3. Quick Check:

   Null check needed before incrementing count [OK]

Hint: Always check for null before incrementing store value [OK]

Common Mistakes:

• Ignoring null values from store.get()
• Incorrect method usage for state store access
• Wrong store type casting

5. You want to build a Kafka Streams application that counts the number of times each word appears in a stream of sentences. Which approach correctly uses a state store to maintain the counts?

hard

A. Use a persistent key-value state store to keep counts, update counts on each word, and query the store for results

B. Use an in-memory state store without updating counts, just print words as they arrive

C. Use a state store only to store the last processed sentence, ignoring counts

D. Use a state store to buffer all sentences without processing counts

Solution

1. Step 1: Understand word count requirement and evaluate options

   Counting words requires storing and updating counts per word, so a key-value store is needed. Use a persistent key-value state store to keep counts, update counts on each word, and query the store for results fits the requirement.

2. Final Answer:

   Use a persistent key-value state store to keep counts, update counts on each word, and query the store for results -> Option A

3. Quick Check:

   Persistent store + update counts = correct word count approach [OK]

Hint: Use persistent store and update counts per word [OK]

Common Mistakes:

• Using in-memory store without updating counts
• Storing sentences instead of counts
• Buffering data without processing