Stream topology in Kafka - Mini Project: Build & Apply

Stream Topology with Kafka Streams

📖 Scenario: You are working on a simple Kafka Streams application that processes user activity data. The data comes as key-value pairs where the key is a user ID and the value is the activity type. You want to build a stream topology that reads from an input topic, processes the data, and writes to an output topic.

🎯 Goal: Build a Kafka Streams topology that reads from the user-activity-input topic, processes the stream by filtering only login activities, and writes the filtered stream to the user-login-output topic.

📋 What You'll Learn

Create a Kafka Streams builder

Define a stream from the user-activity-input topic

Filter the stream to keep only records where the value is login

Write the filtered stream to the user-login-output topic

💡 Why This Matters

🌍 Real World

Kafka Streams is widely used for real-time data processing in applications like monitoring user activity, fraud detection, and event-driven microservices.

💼 Career

Understanding stream topology is essential for roles involving real-time data engineering, backend development, and building scalable event-driven systems.

1

Create the Kafka Streams builder and input stream

Create a StreamsBuilder object called builder. Then create a stream called userActivityStream from the topic "user-activity-input" using builder.stream("user-activity-input").

Kafka

import org.apache.kafka.streams.StreamsBuilder;
import org.apache.kafka.streams.kstream.KStream;

// Your code here

Need a hint?

Use StreamsBuilder to create the builder. Then use builder.stream("user-activity-input") to get the stream.

2

Add a filter to keep only login activities

Create a new stream called loginStream by filtering userActivityStream to keep only records where the value equals "login". Use filter((key, value) -> value.equals("login")).

Kafka

import org.apache.kafka.streams.StreamsBuilder;
import org.apache.kafka.streams.kstream.KStream;

StreamsBuilder builder = new StreamsBuilder();
KStream userActivityStream = builder.stream("user-activity-input");

// Your code here

Need a hint?

Use the filter method on userActivityStream with a lambda that checks if value.equals("login").

3

Write the filtered stream to the output topic

Write the loginStream to the topic "user-login-output" using loginStream.to("user-login-output").

Kafka

import org.apache.kafka.streams.StreamsBuilder;
import org.apache.kafka.streams.kstream.KStream;

StreamsBuilder builder = new StreamsBuilder();
KStream userActivityStream = builder.stream("user-activity-input");
KStream loginStream = userActivityStream.filter((key, value) -> value.equals("login"));

// Your code here

Need a hint?

Use the to method on loginStream to send data to the "user-login-output" topic.

4

Print the topology description

Create a Topology object by calling builder.build(). Then print the topology description using System.out.println(topology.describe()).

Kafka

import org.apache.kafka.streams.StreamsBuilder;
import org.apache.kafka.streams.kstream.KStream;
import org.apache.kafka.streams.Topology;

StreamsBuilder builder = new StreamsBuilder();
KStream userActivityStream = builder.stream("user-activity-input");
KStream loginStream = userActivityStream.filter((key, value) -> value.equals("login"));
loginStream.to("user-login-output");

// Your code here

Need a hint?

Call builder.build() to get the topology, then print its description with System.out.println(topology.describe()).