KStream and KTable concepts in Kafka - Commands & Configuration

Introduction

When working with data streams, you often need to process continuous flows of data or keep track of the latest state. KStream and KTable help you handle these two different needs in Kafka by letting you work with streams of events or tables of current data.

When you want to process each event in a stream as it arrives, like tracking user clicks in real time.

When you need to keep the latest state of data, such as the current balance of a bank account.

When you want to join a stream of events with a table of reference data, like enriching orders with product details.

When you want to aggregate data over time, like counting the number of sales per product.

When you want to update or delete records based on new incoming data.

Commands

This command creates a Kafka topic named 'user-clicks' with 3 partitions to hold streaming event data.

Terminal

kafka-topics --create --topic user-clicks --bootstrap-server localhost:9092 --partitions 3 --replication-factor 1

Expected Output

Created topic user-clicks.

→

--topic - Specifies the name of the Kafka topic to create

→

--partitions - Sets the number of partitions for parallel processing

→

--replication-factor - Sets how many copies of the data are kept for fault tolerance

This command starts a producer to send events to the 'user-clicks' topic, simulating a stream of user click events.

Terminal

kafka-console-producer --topic user-clicks --bootstrap-server localhost:9092

Expected Output

No output (command runs silently)

→

--topic - Specifies the topic to send messages to

→

--bootstrap-server - Specifies the Kafka server address

This command reads all events from the 'user-clicks' topic from the start, showing the stream of events as they arrive.

Terminal

kafka-console-consumer --topic user-clicks --bootstrap-server localhost:9092 --from-beginning

Expected Output

user1,click,home

user2,click,product

user1,click,cart

→

--from-beginning - Reads all messages from the start of the topic

This command runs a Kafka Streams application that reads the 'user-clicks' stream, processes it as a KStream, and outputs results to 'user-clicks-count'.

Terminal

kafka-streams-application --application-id user-clicks-app --bootstrap-server localhost:9092 --input-topic user-clicks --output-topic user-clicks-count

Expected Output

Kafka Streams application started with application.id=user-clicks-app

→

--application-id - Unique ID for the Kafka Streams application

→

--input-topic - Topic to read the input stream from

→

--output-topic - Topic to write the processed output to

Key Concept

If you remember nothing else from this pattern, remember: KStream processes each event as it arrives, while KTable represents the latest state of data like a table.

Common Mistakes

Using KStream when you need to track the latest state of data

KStream treats data as a continuous event stream and does not keep the latest value, so you lose the current state.

Use KTable to represent and query the latest state of each key.

Trying to join two KStreams without considering event timing

Joining KStreams requires careful handling of event time windows; otherwise, you get incomplete or incorrect results.

Use windowed joins or join a KStream with a KTable for simpler state enrichment.

Not creating the Kafka topic before producing or consuming

Kafka commands fail if the topic does not exist or is misconfigured.

Always create topics with the right partitions and replication before use.

Summary

Create Kafka topics to hold streams or tables of data.

Use KStream to process each event in a data stream as it arrives.

Use KTable to keep and query the latest state of data.

Run Kafka Streams applications to process and transform streams and tables.

Consume and produce data to Kafka topics to see the flow of events.

Practice

1. Which statement best describes the difference between KStream and KTable in Kafka Streams?

easy

A. KTable is used for event processing, and KStream is used for batch processing.

B. KStream processes continuous event streams, while KTable maintains the latest state of data.

C. KStream stores data permanently, while KTable only processes events temporarily.

D. KStream and KTable are identical and can be used interchangeably.

Solution

1. Step 1: Understand KStream purpose

   KStream is designed to handle continuous streams of events as they happen.

2. Step 2: Understand KTable purpose

   KTable keeps track of the latest state of data, updating as new information arrives.

3. Final Answer:

   KStream processes continuous event streams, while KTable maintains the latest state of data. -> Option B

4. Quick Check:

   KStream = events, KTable = state [OK]

Hint: Remember: Stream = events, Table = current state [OK]

Common Mistakes:

• Thinking KStream stores data permanently
• Confusing KTable as batch processor
• Assuming KStream and KTable are the same

2. Which of the following is the correct way to create a KTable from a Kafka topic named users in Java?

easy

A. builder.createKTable("users");

B. builder.stream("users").toTable();

C. builder.table("users");

D. builder.getTable("users");

Solution

1. Step 1: Recall KTable creation syntax

   In Kafka Streams Java API, KTable is created using builder.table(topicName).

2. Step 2: Check options for correct method

   Only builder.table("users") matches the correct syntax.

3. Final Answer:

   builder.table("users") -> Option C

4. Quick Check:

   KTable created with builder.table(topic) [OK]

Hint: Use builder.table() to create KTable from topic [OK]

Common Mistakes:

• Using builder.stream() then toTable() which is invalid
• Trying non-existent methods like createKTable()
• Confusing KStream and KTable creation methods

3. Given the following Kafka Streams code snippet, what will be the output printed?

var stream = builder.stream("orders");
var table = builder.table("orders");
stream.foreach((k, v) -> System.out.println("Stream: " + k + ":" + v));
table.toStream().foreach((k, v) -> System.out.println("Table: " + k + ":" + v));

Assuming the topic orders has these records in order: (1, "A"), (1, "B"), (2, "C").

medium

A. Stream prints all three records; Table prints only latest for each key: (1, "B") and (2, "C")

B. Stream prints only latest record per key; Table prints all records

C. Both Stream and Table print all three records

D. Stream prints no records; Table prints all three records

Solution

1. Step 1: Understand KStream output

   KStream processes every event as it arrives, so it prints all three records: Stream: 1:A, Stream: 1:B, Stream: 2:C.

2. Step 2: Understand KTable.toStream() output

   KTable.toStream() emits the latest update per key. It will print updates for key 1 twice (first "A", then "B") and key 2 once ("C"). However, in practice, KTable.toStream() emits only the latest update per key, so the final output for the table stream will be (1, "B") and (2, "C").

3. Final Answer:

   Stream prints all three records; Table prints only latest for each key: (1, "B") and (2, "C") -> Option A

4. Quick Check:

   KStream emits all events; KTable.toStream() emits latest per key [OK]

Hint: KStream prints all events; KTable.toStream() prints latest per key [OK]

Common Mistakes:

• Thinking KTable.toStream() prints all updates
• Thinking KStream shows only latest per key
• Confusing output order between stream and table

4. You wrote this code to convert a KStream to a KTable:

var stream = builder.stream("events");
var table = stream.toTable();

But it throws a compilation error. What is the likely cause?

medium

A. toTable() method does not exist on KStream interface.

B. You cannot convert a KStream to a KTable at all.

C. You must specify a key serde when converting to KTable.

D. You need to call toTable() with a materialized state store.

Solution

1. Step 1: Check KStream toTable() method requirements

   In Kafka Streams, toTable() requires a Materialized parameter to define state store details.

2. Step 2: Identify missing parameter causing error

   Not providing Materialized causes compilation error because Kafka Streams needs to know how to store the table.

3. Final Answer:

   You need to call toTable() with a materialized state store. -> Option D

4. Quick Check:

   toTable() needs Materialized param [OK]

Hint: toTable() requires Materialized argument [OK]

Common Mistakes:

• Assuming toTable() exists without parameters
• Thinking conversion is impossible
• Ignoring state store configuration

5. You want to build a Kafka Streams application that tracks the latest price of products and also processes every price update event for analytics. Which combination should you use?

hard

A. Use a KTable to keep latest prices and a KStream to process all price update events.

B. Use only a KStream for both latest prices and event processing.

C. Use only a KTable because it handles both events and state.

D. Use a KStream for latest prices and a KTable for event processing.

Solution

1. Step 1: Identify need for latest state tracking

   Tracking latest prices requires a KTable because it maintains the current state per product.

2. Step 2: Identify need for processing all events

   Processing every price update event for analytics requires a KStream to handle all incoming events.

3. Step 3: Combine both correctly

   Use KTable for latest prices and KStream for event processing simultaneously.

4. Final Answer:

   Use a KTable to keep latest prices and a KStream to process all price update events. -> Option A

5. Quick Check:

   KTable = state, KStream = all events [OK]

Hint: Use KTable for state, KStream for event analytics [OK]

Common Mistakes:

• Using only KStream loses latest state
• Using only KTable misses event details
• Swapping roles of KStream and KTable