KStream and KTable concepts in Kafka - Mini Project: Build & Apply

Understanding KStream and KTable Concepts

📖 Scenario: You are working with Kafka Streams to process real-time data from a store's sales system. You want to understand how to use KStream and KTable to handle sales events and maintain the latest product inventory.

🎯 Goal: Build a simple Kafka Streams application that creates a KStream for sales events and a KTable for product inventory, then join them to enrich sales with current inventory data.

📋 What You'll Learn

Create a KStream named salesStream with sample sales data

Create a KTable named inventoryTable with product inventory data

Join salesStream and inventoryTable on product ID

Print the joined results showing sales enriched with inventory info

💡 Why This Matters

🌍 Real World

Retail companies use Kafka Streams to process sales and inventory data in real time to make quick decisions about stock and promotions.

💼 Career

Understanding KStream and KTable is essential for roles in data engineering and real-time analytics using Kafka.

1

Create the salesStream with sample sales data

Create a KStream<String, Integer> called salesStream with these entries: ("product1", 3), ("product2", 5), and ("product3", 2).

Kafka

# Create salesStream with sample sales data
# Your code here

Hint

Use builder.stream() to create the KStream from the topic "sales-topic".

2

Create the inventoryTable with product inventory data

Create a KTable<String, Integer> called inventoryTable with these entries: ("product1", 10), ("product2", 20), and ("product3", 15).

Kafka

KStream salesStream = builder.stream("sales-topic");
// For this exercise, assume salesStream contains: ("product1", 3), ("product2", 5), ("product3", 2)

# Create inventoryTable with product inventory data
# Your code here

Hint

Use builder.table() to create the KTable from the topic "inventory-topic".

3

Join salesStream with inventoryTable

Use salesStream.join(inventoryTable, (saleQty, inventoryQty) -> saleQty + inventoryQty) to create a joined stream called enrichedSales that adds sale quantity and inventory quantity.

Kafka

KStream salesStream = builder.stream("sales-topic");
// For this exercise, assume salesStream contains: ("product1", 3), ("product2", 5), ("product3", 2)

KTable inventoryTable = builder.table("inventory-topic");
// For this exercise, assume inventoryTable contains: ("product1", 10), ("product2", 20), ("product3", 15)

# Join salesStream with inventoryTable to create enrichedSales
# Your code here

Hint

Use the join method on salesStream with inventoryTable and a lambda that adds the two quantities.

4

Print the joined results

Use enrichedSales.foreach((product, totalQty) -> System.out.println(product + ": " + totalQty)) to print each product and its total quantity.

Kafka

KStream salesStream = builder.stream("sales-topic");
// For this exercise, assume salesStream contains: ("product1", 3), ("product2", 5), ("product3", 2)

KTable inventoryTable = builder.table("inventory-topic");
// For this exercise, assume inventoryTable contains: ("product1", 10), ("product2", 20), ("product3", 15)

KStream enrichedSales = salesStream.join(inventoryTable, (saleQty, inventoryQty) -> saleQty + inventoryQty);

# Print the enrichedSales stream
# Your code here

Hint

Use foreach on enrichedSales to print each product and its total quantity.

Practice

1. Which statement best describes the difference between KStream and KTable in Kafka Streams?

easy

A. KTable is used for event processing, and KStream is used for batch processing.

B. KStream processes continuous event streams, while KTable maintains the latest state of data.

C. KStream stores data permanently, while KTable only processes events temporarily.

D. KStream and KTable are identical and can be used interchangeably.

Solution

1. Step 1: Understand KStream purpose

   KStream is designed to handle continuous streams of events as they happen.

2. Step 2: Understand KTable purpose

   KTable keeps track of the latest state of data, updating as new information arrives.

3. Final Answer:

   KStream processes continuous event streams, while KTable maintains the latest state of data. -> Option B

4. Quick Check:

   KStream = events, KTable = state [OK]

Hint: Remember: Stream = events, Table = current state [OK]

Common Mistakes:

• Thinking KStream stores data permanently
• Confusing KTable as batch processor
• Assuming KStream and KTable are the same

2. Which of the following is the correct way to create a KTable from a Kafka topic named users in Java?

easy

A. builder.createKTable("users");

B. builder.stream("users").toTable();

C. builder.table("users");

D. builder.getTable("users");

Solution

1. Step 1: Recall KTable creation syntax

   In Kafka Streams Java API, KTable is created using builder.table(topicName).

2. Step 2: Check options for correct method

   Only builder.table("users") matches the correct syntax.

3. Final Answer:

   builder.table("users") -> Option C

4. Quick Check:

   KTable created with builder.table(topic) [OK]

Hint: Use builder.table() to create KTable from topic [OK]

Common Mistakes:

• Using builder.stream() then toTable() which is invalid
• Trying non-existent methods like createKTable()
• Confusing KStream and KTable creation methods

3. Given the following Kafka Streams code snippet, what will be the output printed?

var stream = builder.stream("orders");
var table = builder.table("orders");
stream.foreach((k, v) -> System.out.println("Stream: " + k + ":" + v));
table.toStream().foreach((k, v) -> System.out.println("Table: " + k + ":" + v));

Assuming the topic orders has these records in order: (1, "A"), (1, "B"), (2, "C").

medium

A. Stream prints all three records; Table prints only latest for each key: (1, "B") and (2, "C")

B. Stream prints only latest record per key; Table prints all records

C. Both Stream and Table print all three records

D. Stream prints no records; Table prints all three records

Solution

1. Step 1: Understand KStream output

   KStream processes every event as it arrives, so it prints all three records: Stream: 1:A, Stream: 1:B, Stream: 2:C.

2. Step 2: Understand KTable.toStream() output

   KTable.toStream() emits the latest update per key. It will print updates for key 1 twice (first "A", then "B") and key 2 once ("C"). However, in practice, KTable.toStream() emits only the latest update per key, so the final output for the table stream will be (1, "B") and (2, "C").

3. Final Answer:

   Stream prints all three records; Table prints only latest for each key: (1, "B") and (2, "C") -> Option A

4. Quick Check:

   KStream emits all events; KTable.toStream() emits latest per key [OK]

Hint: KStream prints all events; KTable.toStream() prints latest per key [OK]

Common Mistakes:

• Thinking KTable.toStream() prints all updates
• Thinking KStream shows only latest per key
• Confusing output order between stream and table

4. You wrote this code to convert a KStream to a KTable:

var stream = builder.stream("events");
var table = stream.toTable();

But it throws a compilation error. What is the likely cause?

medium

A. toTable() method does not exist on KStream interface.

B. You cannot convert a KStream to a KTable at all.

C. You must specify a key serde when converting to KTable.

D. You need to call toTable() with a materialized state store.

Solution

1. Step 1: Check KStream toTable() method requirements

   In Kafka Streams, toTable() requires a Materialized parameter to define state store details.

2. Step 2: Identify missing parameter causing error

   Not providing Materialized causes compilation error because Kafka Streams needs to know how to store the table.

3. Final Answer:

   You need to call toTable() with a materialized state store. -> Option D

4. Quick Check:

   toTable() needs Materialized param [OK]

Hint: toTable() requires Materialized argument [OK]

Common Mistakes:

• Assuming toTable() exists without parameters
• Thinking conversion is impossible
• Ignoring state store configuration

5. You want to build a Kafka Streams application that tracks the latest price of products and also processes every price update event for analytics. Which combination should you use?

hard

A. Use a KTable to keep latest prices and a KStream to process all price update events.

B. Use only a KStream for both latest prices and event processing.

C. Use only a KTable because it handles both events and state.

D. Use a KStream for latest prices and a KTable for event processing.

Solution

1. Step 1: Identify need for latest state tracking

   Tracking latest prices requires a KTable because it maintains the current state per product.

2. Step 2: Identify need for processing all events

   Processing every price update event for analytics requires a KStream to handle all incoming events.

3. Step 3: Combine both correctly

   Use KTable for latest prices and KStream for event processing simultaneously.

4. Final Answer:

   Use a KTable to keep latest prices and a KStream to process all price update events. -> Option A

5. Quick Check:

   KTable = state, KStream = all events [OK]

Hint: Use KTable for state, KStream for event analytics [OK]

Common Mistakes:

• Using only KStream loses latest state
• Using only KTable misses event details
• Swapping roles of KStream and KTable