Stream topology in Kafka - Step-by-Step Execution

Process Flow - Stream topology

Define Source Topic

↓

Create Stream from Source

↓

Apply Transformations

↓

Send to Sink Topic

↓

Stream Processing Continues

A stream topology defines how data flows from source topics through processing steps to sink topics in Kafka Streams.

Execution Sample

Kafka

StreamsBuilder builder = new StreamsBuilder();
KStream<String, String> source = builder.stream("source-topic");
KStream<String, String> transformed = source.mapValues(value -> value.toUpperCase());
transformed.to("sink-topic");

This code creates a stream from 'source-topic', converts each message value to uppercase, and writes results to 'sink-topic'.

Process Table

Step	Action	Input Message	Transformation	Output Message	Topic
1	Read message	hello	none	hello	source-topic
2	Apply mapValues	hello	toUpperCase	HELLO	in-memory stream
3	Write message	HELLO	none	HELLO	sink-topic
4	Read message	world	none	world	source-topic
5	Apply mapValues	world	toUpperCase	WORLD	in-memory stream
6	Write message	WORLD	none	WORLD	sink-topic
7	No more messages	-	-	-	-

💡 No more messages in source-topic, stream processing ends.

Status Tracker

Variable	Start	After 1	After 2	Final
source message	-	hello	world	-
transformed message	-	HELLO	WORLD	-

Key Moments - 2 Insights

Why does the transformed message appear after the source message in the execution table?

Because the stream reads the source message first (rows 1 and 4), then applies the transformation (rows 2 and 5), and finally writes the transformed message to the sink topic (rows 3 and 6). This order shows the flow of data through the topology.

What happens if there are no messages in the source topic?

The execution stops at step 7 as shown in the exit note, because the stream topology processes messages only when they arrive from the source topic.

Visual Quiz - 3 Questions

Look at the execution table, what is the output message at step 3?

A hello

B HELLO

C world

D WORLD

💡 Hint

Check the 'Output Message' column at step 3 in the execution_table.

At which step does the transformation to uppercase happen for the message 'world'?

A Step 6

B Step 4

C Step 5

D Step 7

💡 Hint

Look for the 'Transformation' column where 'toUpperCase' is applied for 'world' in the execution_table.

If the source topic had no messages, what would the execution table show?

A Only step 7 indicating no messages

B Steps 1 to 6 with empty messages

C Only steps 1 and 2

D All steps repeated infinitely

💡 Hint

Refer to the exit_note and step 7 in the execution_table.

Concept Snapshot

Stream topology in Kafka Streams:
- Define source topic stream
- Apply transformations (map, filter, etc.)
- Write results to sink topic
- Data flows step-by-step through topology
- Processing happens as messages arrive

Full Transcript

This visual execution shows how a Kafka Streams topology processes messages. First, messages are read from the source topic. Then, each message is transformed, here by converting the value to uppercase. Finally, the transformed message is written to the sink topic. The execution table traces each step for two example messages, 'hello' and 'world'. Variables track the message values before and after transformation. Key moments clarify why the order of steps matters and what happens if no messages arrive. The quiz tests understanding of the flow and output at each step.

Practice

1. What does a Kafka Streams topology represent?

easy

A. The configuration settings for Kafka brokers

B. The flow and processing steps of data in a Kafka Streams application

C. The list of Kafka topics available in the cluster

D. The schema of messages in a Kafka topic

Solution

1. Step 1: Understand the role of topology in Kafka Streams

   A topology defines how data flows and is processed in a Kafka Streams app.

2. Step 2: Differentiate topology from other Kafka concepts

   It is not about broker configs, topic lists, or message schemas, but about processing steps.

3. Final Answer:

   The flow and processing steps of data in a Kafka Streams application -> Option B

4. Quick Check:

   Topology = Data flow and processing [OK]

Hint: Topology means data flow and processing steps [OK]

Common Mistakes:

• Confusing topology with Kafka broker settings
• Thinking topology lists topics
• Assuming topology defines message schema

2. Which code snippet correctly builds a Kafka Streams topology?

easy

A. StreamsBuilder builder = new StreamsBuilder(); builder.run();

B. StreamsBuilder builder = new StreamsBuilder(); builder.start();

C. StreamsBuilder builder = new StreamsBuilder(); builder.create();

D. StreamsBuilder builder = new StreamsBuilder(); builder.build();

Solution

1. Step 1: Recall the correct method to build topology

   The method to finalize and get the topology is build().

2. Step 2: Check each option's method

   Only StreamsBuilder builder = new StreamsBuilder(); builder.build(); uses build(), others use invalid methods.

3. Final Answer:

   StreamsBuilder builder = new StreamsBuilder(); builder.build(); -> Option D

4. Quick Check:

   Use build() to create topology [OK]

Hint: Use build() method on StreamsBuilder to create topology [OK]

Common Mistakes:

• Using start(), create(), or run() instead of build()
• Confusing topology building with starting the stream

3. Given the code snippet:

StreamsBuilder builder = new StreamsBuilder();
builder.stream("input-topic").mapValues(value -> value.toUpperCase()).to("output-topic");
Topology topology = builder.build();
System.out.println(topology.describe());

What will the printed topology description include?

medium

A. A sink node for 'input-topic' and a source node for 'output-topic'

B. Only a source node for 'input-topic' without any processing

C. A source node for 'input-topic', a processor node for mapValues, and a sink node for 'output-topic'

D. An error because mapValues is not allowed in topology

Solution

1. Step 1: Analyze the stream processing steps

   The code reads from 'input-topic', applies mapValues to uppercase, then writes to 'output-topic'.

2. Step 2: Understand topology description output

   The description shows source nodes (input), processor nodes (mapValues), and sink nodes (output).

3. Final Answer:

   A source node for 'input-topic', a processor node for mapValues, and a sink node for 'output-topic' -> Option C

4. Quick Check:

   Topology describe shows source, processor, sink [OK]

Hint: Topology describe shows source, processor, and sink nodes [OK]

Common Mistakes:

• Thinking mapValues is not part of topology
• Confusing source and sink nodes
• Expecting no processor nodes in description

4. Identify the error in this Kafka Streams topology code:

StreamsBuilder builder = new StreamsBuilder();
builder.stream("input-topic").filter(value -> value.length() > 5);
Topology topology = builder.build();
System.out.println(topology.describe());

medium

A. The filtered stream is not sent to any sink topic, so no output node is created

B. The filter method is invalid and causes a syntax error

C. StreamsBuilder cannot build topology without a source topic

D. Calling build() before defining streams causes runtime error

Solution

1. Step 1: Check the stream processing chain

   The code filters the stream but does not write the result to any output topic.

2. Step 2: Understand topology implications

   Without a sink node, the topology has no output, which is usually an error or incomplete.

3. Final Answer:

   The filtered stream is not sent to any sink topic, so no output node is created -> Option A

4. Quick Check:

   Missing sink node causes incomplete topology [OK]

Hint: Always send processed stream to sink topic [OK]

Common Mistakes:

• Assuming filter causes syntax error
• Thinking build() requires no streams
• Ignoring missing sink topic after processing

5. You want to create a Kafka Streams topology that reads from two topics, merges their streams, filters messages containing 'error', and writes to an output topic. Which code snippet correctly builds this topology?

hard

A. StreamsBuilder builder = new StreamsBuilder();
KStream stream1 = builder.stream("topic1");
KStream stream2 = builder.stream("topic2");
KStream merged = stream1.merge(stream2);
merged.filter((k, v) -> v.contains("error")).to("output-topic");
Topology topology = builder.build();

B. StreamsBuilder builder = new StreamsBuilder();
KStream stream = builder.stream("topic1", "topic2");
stream.filter(v -> v.contains("error")).to("output-topic");
Topology topology = builder.build();

C. StreamsBuilder builder = new StreamsBuilder();
KStream stream1 = builder.stream("topic1");
KStream stream2 = builder.stream("topic2");
KStream merged = stream1.join(stream2, (v1, v2) -> v1 + v2);
merged.filter((k, v) -> v.contains("error")).to("output-topic");
Topology topology = builder.build();

D. StreamsBuilder builder = new StreamsBuilder();
builder.stream("topic1").filter(v -> v.contains("error")).to("output-topic");
builder.stream("topic2").filter(v -> v.contains("error")).to("output-topic");
Topology topology = builder.build();

Solution

1. Step 1: Understand the requirement

   Read from two topics, merge streams, filter messages with 'error', write to output.

2. Step 2: Evaluate each option

   StreamsBuilder builder = new StreamsBuilder(); KStream stream1 = builder.stream("topic1"); KStream stream2 = builder.stream("topic2"); KStream merged = stream1.merge(stream2); merged.filter((k, v) -> v.contains("error")).to("output-topic"); Topology topology = builder.build(); merges streams correctly, filters, then writes once. StreamsBuilder builder = new StreamsBuilder(); KStream stream = builder.stream("topic1", "topic2"); stream.filter(v -> v.contains("error")).to("output-topic"); Topology topology = builder.build(); incorrectly tries to stream two topics in one call. StreamsBuilder builder = new StreamsBuilder(); KStream stream1 = builder.stream("topic1"); KStream stream2 = builder.stream("topic2"); KStream merged = stream1.join(stream2, (v1, v2) -> v1 + v2); merged.filter((k, v) -> v.contains("error")).to("output-topic"); Topology topology = builder.build(); joins streams (not merge). StreamsBuilder builder = new StreamsBuilder(); builder.stream("topic1").filter(v -> v.contains("error")).to("output-topic"); builder.stream("topic2").filter(v -> v.contains("error")).to("output-topic"); Topology topology = builder.build(); filters separately and writes twice, not merging.

3. Final Answer:

   Option A correctly merges, filters, and writes to output topic -> Option A

4. Quick Check:

   Merge streams, filter, then write [OK]

Hint: Merge streams before filtering and writing output [OK]

Common Mistakes:

• Using join instead of merge for combining streams
• Trying to stream multiple topics in one call incorrectly
• Writing filtered streams separately without merging