Stream topology in Kafka - Deep Dive

Overview - Stream topology

What is it?

Stream topology is the layout or design of how data flows and is processed in a Kafka Streams application. It shows the sequence of operations like filtering, mapping, joining, and aggregating data streams. Each step in the topology represents a processing node that transforms or routes data. This design helps organize complex data processing tasks into clear, manageable parts.

Why it matters

Without stream topology, managing and understanding how data moves and changes in real-time would be chaotic. It solves the problem of organizing continuous data processing so developers can build reliable, scalable, and maintainable streaming applications. Without it, debugging, scaling, or evolving streaming logic would be very difficult, leading to errors and slow systems.

Where it fits

Before learning stream topology, you should understand Kafka basics like topics, producers, and consumers. After mastering topology, you can explore advanced Kafka Streams features like state stores, windowing, and fault tolerance. It fits in the journey between basic Kafka usage and building complex real-time data pipelines.

Mental Model

Core Idea

Stream topology is the blueprint that maps how data flows through processing steps in a Kafka Streams application.

Think of it like...

Imagine a factory assembly line where raw materials enter, pass through different machines that shape, inspect, and combine parts, and finally produce a finished product. Each machine is like a processing step in the stream topology, and the conveyor belts represent the data flow between them.

┌───────────────┐     ┌───────────────┐     ┌───────────────┐
│ Source Topic  │ --> │ Processor 1   │ --> │ Processor 2   │ --> ...
└───────────────┘     └───────────────┘     └───────────────┘
                             │                     │
                             ▼                     ▼
                      ┌───────────────┐     ┌───────────────┐
                      │ State Store   │     │ Sink Topic    │
                      └───────────────┘     └───────────────┘

Build-Up - 7 Steps

1

Foundation

Understanding Kafka Streams Basics

Concept: Introduce Kafka Streams as a library to process data streams in real-time.

Kafka Streams lets you read data from Kafka topics, process it with operations like filtering or mapping, and write results back to topics. It runs inside your application, making it easy to build streaming apps without separate clusters.

Result

You can create simple stream processing apps that consume and produce Kafka data.

Knowing Kafka Streams basics is essential because stream topology builds on how streams are created and processed.

2

Foundation

What is a Stream Topology?

Concept: Define stream topology as the structure of processing steps in a Kafka Streams app.

A topology is a graph of nodes where each node is a processor that transforms or routes data. It starts from source topics, passes through processors, and ends at sink topics or state stores.

Result

You understand that stream topology is the map of your data processing flow.

Recognizing topology as a graph helps visualize and organize complex stream processing.

3

Intermediate

Building a Simple Topology Programmatically

🤔Before reading on: do you think a topology is defined by configuration files or by code? Commit to your answer.

Concept: Learn how to create a topology using Kafka Streams API in code.

You use the StreamsBuilder class to define sources, processors, and sinks. For example, builder.stream("input-topic") creates a source node. Then you chain operations like filter() or map() to add processors. Finally, you write results to output topics.

Result

You can write code that builds a stream topology representing your data flow.

Understanding that topology is built in code clarifies how flexible and dynamic stream processing can be.

4

Intermediate

Topology Components: Sources, Processors, Sinks

🤔Before reading on: do you think processors can exist without sources or sinks? Commit to your answer.

Concept: Explore the three main node types in a topology and their roles.

Source nodes read data from Kafka topics. Processor nodes transform or analyze data. Sink nodes write processed data back to Kafka topics. State stores can be attached to processors to keep data between events.

Result

You can identify and explain each component's role in a topology.

Knowing these components helps you design and debug stream applications effectively.

5

Intermediate

Visualizing and Inspecting Topologies

🤔Before reading on: do you think Kafka Streams provides tools to see your topology? Commit to your answer.

Concept: Learn how to view and understand the topology your code creates.

Kafka Streams offers the toString() method on Topology objects to print the topology graph. This helps verify the processing steps and connections. Visualizing the topology aids in debugging and optimization.

Result

You can print and interpret the topology structure from your application.

Seeing the topology output prevents errors and clarifies complex stream flows.

6

Advanced

State Stores and Their Role in Topology

🤔Before reading on: do you think state stores are optional or mandatory in all topologies? Commit to your answer.

Concept: Understand how state stores add memory to stream processing nodes.

State stores keep data locally for processors to use across multiple events, enabling operations like joins and aggregations. They are part of the topology and must be defined and connected properly.

Result

You can design topologies that maintain state for complex processing.

Knowing how state stores integrate into topology is key for building powerful, fault-tolerant streaming apps.

7

Expert

Topology Optimization and Internal Mechanics

🤔Before reading on: do you think Kafka Streams rearranges your topology internally for performance? Commit to your answer.

Concept: Explore how Kafka Streams optimizes and manages topology execution under the hood.

Kafka Streams analyzes the topology graph to optimize task assignment, parallelism, and state management. It splits the topology into tasks that run on different threads or machines. Understanding this helps tune performance and troubleshoot issues.

Result

You grasp how your topology translates into running processes and how to optimize it.

Knowing internal optimization reveals why some topology designs perform better and how to scale streaming apps.

Under the Hood

Kafka Streams represents the stream topology as a directed acyclic graph where nodes are processors and edges are data flows. At runtime, this graph is divided into tasks assigned to threads. Each task processes data from partitions, manages local state stores, and commits results back to Kafka. The library handles fault tolerance by replaying data from Kafka and restoring state stores.

Why designed this way?

This design allows Kafka Streams to be scalable, fault-tolerant, and easy to use. Using a graph model matches the natural flow of data processing. Dividing work into tasks enables parallelism and efficient resource use. Alternatives like centralized processing or manual partition management were more complex and less resilient.

┌───────────────┐       ┌───────────────┐       ┌───────────────┐
│ Source Node   │──────▶│ Processor Node│──────▶│ Sink Node     │
└───────────────┘       └───────────────┘       └───────────────┘
        │                      │                       │
        ▼                      ▼                       ▼
  ┌───────────────┐      ┌───────────────┐       ┌───────────────┐
  │ Kafka Topic   │      │ State Store   │       │ Kafka Topic   │
  └───────────────┘      └───────────────┘       └───────────────┘

Myth Busters - 4 Common Misconceptions

Quick: Is a stream topology a static configuration file or dynamic code structure? Commit to your answer.

Common Belief: Stream topology is a fixed configuration file you write once and never change.

Reality: Stream topology is defined dynamically in code using Kafka Streams API, allowing flexible and programmable data flows.

Why it matters: Believing topology is static limits understanding of how to build adaptable streaming apps and hinders debugging.

Quick: Do you think processors can run independently without source nodes? Commit to your answer.

Common Belief: Processors can exist and run without any source nodes feeding data.

Reality: Processors depend on source nodes to receive data; without sources, processors have no input to work on.

Why it matters: Misunderstanding this leads to designing incomplete topologies that never process data.

Quick: Do you think state stores are always required in Kafka Streams topologies? Commit to your answer.

Common Belief: Every Kafka Streams topology must include state stores to function.

Reality: State stores are optional and only needed for stateful operations like joins or aggregations; stateless processing does not require them.

Why it matters: Assuming state stores are mandatory can complicate simple stream designs and waste resources.

Quick: Do you think Kafka Streams executes your topology exactly as you wrote it without changes? Commit to your answer.

Common Belief: Kafka Streams runs the topology exactly as defined in code without any internal changes.

Reality: Kafka Streams optimizes and rearranges the topology internally for performance and fault tolerance.

Why it matters: Ignoring internal optimizations can cause confusion when debugging or tuning performance.

Expert Zone

1

Topology nodes can be reused internally by Kafka Streams to optimize resource use, which can affect how metrics and logs appear.

2

The order of processor nodes matters because Kafka Streams processes data in the defined sequence, impacting latency and correctness.

3

State stores are backed by changelog topics in Kafka, enabling fault tolerance by replaying state changes after failures.

When NOT to use

Stream topology is not suitable for batch processing or very low-latency microsecond-level processing. For batch jobs, use tools like Apache Spark. For ultra-low latency, consider specialized stream processors or in-memory databases.

Production Patterns

In production, topologies are designed modularly with reusable sub-topologies for common tasks. They use state stores for joins and aggregations, and leverage Kafka Streams' fault tolerance by carefully managing changelog topics and commit intervals.

Connections

Dataflow Programming

Stream topology is a specific example of dataflow programming where computation is modeled as a graph of operations.

Understanding stream topology deepens comprehension of dataflow concepts used in many real-time and parallel processing systems.

Factory Assembly Lines

Both organize sequential steps to transform inputs into outputs efficiently.

Recognizing this connection helps appreciate how stream topology structures complex data transformations clearly and reliably.

Neural Networks

Like stream topologies, neural networks are graphs of nodes processing data in layers.

Seeing this similarity reveals how graph-based processing models are powerful across domains from data engineering to AI.

Common Pitfalls

#1 Defining topology without connecting source nodes.

Wrong approach:

StreamsBuilder builder = new StreamsBuilder();
KStream stream = builder.stream(); // missing topic name
stream.filter(...).to("output-topic");

Correct approach:

StreamsBuilder builder = new StreamsBuilder();
KStream stream = builder.stream("input-topic");
stream.filter(...).to("output-topic");

Root cause:

Not specifying source topics means no data enters the topology, so processing never happens.

#2 Assuming state stores are automatically created for all operations.

Wrong approach:

builder.stream("input").map(...).to("output"); // expecting state store for map

Correct approach:

builder.stream("input").groupByKey().aggregate(...); // explicitly creates state store

Root cause:

State stores only exist for stateful operations; stateless ones like map do not create them.

#3 Ignoring topology visualization and debugging tools.

Wrong approach:

Not printing or inspecting topology structure, leading to confusion about data flow.

Correct approach:

System.out.println(builder.build().describe()); // prints topology graph

Root cause:

Skipping visualization makes it hard to verify and understand complex topologies.

Key Takeaways

Stream topology is the map of how data flows and is processed in Kafka Streams applications.

It is defined dynamically in code using Kafka Streams API, not static configuration files.

Topology consists of source nodes, processor nodes, sink nodes, and optionally state stores for stateful processing.

Kafka Streams internally optimizes and manages topology execution for scalability and fault tolerance.

Understanding topology helps build, debug, and optimize real-time streaming data pipelines effectively.

Practice

1. What does a Kafka Streams topology represent?

easy

A. The configuration settings for Kafka brokers

B. The flow and processing steps of data in a Kafka Streams application

C. The list of Kafka topics available in the cluster

D. The schema of messages in a Kafka topic

Solution

1. Step 1: Understand the role of topology in Kafka Streams

   A topology defines how data flows and is processed in a Kafka Streams app.

2. Step 2: Differentiate topology from other Kafka concepts

   It is not about broker configs, topic lists, or message schemas, but about processing steps.

3. Final Answer:

   The flow and processing steps of data in a Kafka Streams application -> Option B

4. Quick Check:

   Topology = Data flow and processing [OK]

Hint: Topology means data flow and processing steps [OK]

Common Mistakes:

• Confusing topology with Kafka broker settings
• Thinking topology lists topics
• Assuming topology defines message schema

2. Which code snippet correctly builds a Kafka Streams topology?

easy

A. StreamsBuilder builder = new StreamsBuilder(); builder.run();

B. StreamsBuilder builder = new StreamsBuilder(); builder.start();

C. StreamsBuilder builder = new StreamsBuilder(); builder.create();

D. StreamsBuilder builder = new StreamsBuilder(); builder.build();

Solution

1. Step 1: Recall the correct method to build topology

   The method to finalize and get the topology is build().

2. Step 2: Check each option's method

   Only StreamsBuilder builder = new StreamsBuilder(); builder.build(); uses build(), others use invalid methods.

3. Final Answer:

   StreamsBuilder builder = new StreamsBuilder(); builder.build(); -> Option D

4. Quick Check:

   Use build() to create topology [OK]

Hint: Use build() method on StreamsBuilder to create topology [OK]

Common Mistakes:

• Using start(), create(), or run() instead of build()
• Confusing topology building with starting the stream

3. Given the code snippet:

StreamsBuilder builder = new StreamsBuilder();
builder.stream("input-topic").mapValues(value -> value.toUpperCase()).to("output-topic");
Topology topology = builder.build();
System.out.println(topology.describe());

What will the printed topology description include?

medium

A. A sink node for 'input-topic' and a source node for 'output-topic'

B. Only a source node for 'input-topic' without any processing

C. A source node for 'input-topic', a processor node for mapValues, and a sink node for 'output-topic'

D. An error because mapValues is not allowed in topology

Solution

1. Step 1: Analyze the stream processing steps

   The code reads from 'input-topic', applies mapValues to uppercase, then writes to 'output-topic'.

2. Step 2: Understand topology description output

   The description shows source nodes (input), processor nodes (mapValues), and sink nodes (output).

3. Final Answer:

   A source node for 'input-topic', a processor node for mapValues, and a sink node for 'output-topic' -> Option C

4. Quick Check:

   Topology describe shows source, processor, sink [OK]

Hint: Topology describe shows source, processor, and sink nodes [OK]

Common Mistakes:

• Thinking mapValues is not part of topology
• Confusing source and sink nodes
• Expecting no processor nodes in description

4. Identify the error in this Kafka Streams topology code:

StreamsBuilder builder = new StreamsBuilder();
builder.stream("input-topic").filter(value -> value.length() > 5);
Topology topology = builder.build();
System.out.println(topology.describe());

medium

A. The filtered stream is not sent to any sink topic, so no output node is created

B. The filter method is invalid and causes a syntax error

C. StreamsBuilder cannot build topology without a source topic

D. Calling build() before defining streams causes runtime error

Solution

1. Step 1: Check the stream processing chain

   The code filters the stream but does not write the result to any output topic.

2. Step 2: Understand topology implications

   Without a sink node, the topology has no output, which is usually an error or incomplete.

3. Final Answer:

   The filtered stream is not sent to any sink topic, so no output node is created -> Option A

4. Quick Check:

   Missing sink node causes incomplete topology [OK]

Hint: Always send processed stream to sink topic [OK]

Common Mistakes:

• Assuming filter causes syntax error
• Thinking build() requires no streams
• Ignoring missing sink topic after processing

5. You want to create a Kafka Streams topology that reads from two topics, merges their streams, filters messages containing 'error', and writes to an output topic. Which code snippet correctly builds this topology?

hard

A. StreamsBuilder builder = new StreamsBuilder();
KStream stream1 = builder.stream("topic1");
KStream stream2 = builder.stream("topic2");
KStream merged = stream1.merge(stream2);
merged.filter((k, v) -> v.contains("error")).to("output-topic");
Topology topology = builder.build();

B. StreamsBuilder builder = new StreamsBuilder();
KStream stream = builder.stream("topic1", "topic2");
stream.filter(v -> v.contains("error")).to("output-topic");
Topology topology = builder.build();

C. StreamsBuilder builder = new StreamsBuilder();
KStream stream1 = builder.stream("topic1");
KStream stream2 = builder.stream("topic2");
KStream merged = stream1.join(stream2, (v1, v2) -> v1 + v2);
merged.filter((k, v) -> v.contains("error")).to("output-topic");
Topology topology = builder.build();

D. StreamsBuilder builder = new StreamsBuilder();
builder.stream("topic1").filter(v -> v.contains("error")).to("output-topic");
builder.stream("topic2").filter(v -> v.contains("error")).to("output-topic");
Topology topology = builder.build();

Solution

1. Step 1: Understand the requirement

   Read from two topics, merge streams, filter messages with 'error', write to output.

2. Step 2: Evaluate each option

   StreamsBuilder builder = new StreamsBuilder(); KStream stream1 = builder.stream("topic1"); KStream stream2 = builder.stream("topic2"); KStream merged = stream1.merge(stream2); merged.filter((k, v) -> v.contains("error")).to("output-topic"); Topology topology = builder.build(); merges streams correctly, filters, then writes once. StreamsBuilder builder = new StreamsBuilder(); KStream stream = builder.stream("topic1", "topic2"); stream.filter(v -> v.contains("error")).to("output-topic"); Topology topology = builder.build(); incorrectly tries to stream two topics in one call. StreamsBuilder builder = new StreamsBuilder(); KStream stream1 = builder.stream("topic1"); KStream stream2 = builder.stream("topic2"); KStream merged = stream1.join(stream2, (v1, v2) -> v1 + v2); merged.filter((k, v) -> v.contains("error")).to("output-topic"); Topology topology = builder.build(); joins streams (not merge). StreamsBuilder builder = new StreamsBuilder(); builder.stream("topic1").filter(v -> v.contains("error")).to("output-topic"); builder.stream("topic2").filter(v -> v.contains("error")).to("output-topic"); Topology topology = builder.build(); filters separately and writes twice, not merging.

3. Final Answer:

   Option A correctly merges, filters, and writes to output topic -> Option A

4. Quick Check:

   Merge streams, filter, then write [OK]

Hint: Merge streams before filtering and writing output [OK]

Common Mistakes:

• Using join instead of merge for combining streams
• Trying to stream multiple topics in one call incorrectly
• Writing filtered streams separately without merging