Testing stream topologies in Kafka - Mini Project: Build & Apply

Testing stream topologies

📖 Scenario: You are working with Kafka Streams to process data in real-time. You want to test your stream processing logic before deploying it to production.

🎯 Goal: Build a simple Kafka Streams topology and write a test to verify its processing logic.

📋 What You'll Learn

Create a Kafka Streams topology that reads from an input topic and writes to an output topic

Add a configuration variable for the application ID

Implement the core processing logic using a simple mapValues transformation

Write a test that verifies the output of the topology given some input data

💡 Why This Matters

🌍 Real World

Kafka Streams is widely used for real-time data processing in industries like finance, retail, and IoT to transform and analyze streaming data.

💼 Career

Understanding how to build and test Kafka Streams topologies is essential for roles in data engineering, backend development, and real-time analytics.

1

Create the Kafka Streams topology

Create a StreamsBuilder instance called builder and define a stream from the input topic input-topic. Then create a KStream called inputStream from builder.stream("input-topic").

Kafka

# Create a StreamsBuilder and define inputStream from 'input-topic'
# Your code here

Need a hint?

Use StreamsBuilder builder = new StreamsBuilder(); and then KStream<String, String> inputStream = builder.stream("input-topic");

2

Add configuration for application ID

Create a Properties object called props and set the property StreamsConfig.APPLICATION_ID_CONFIG to test-app.

Kafka

import java.util.Properties;

import org.apache.kafka.streams.StreamsBuilder;
import org.apache.kafka.streams.kstream.KStream;

StreamsBuilder builder = new StreamsBuilder();
KStream inputStream = builder.stream("input-topic");

# Create Properties object props and set APPLICATION_ID_CONFIG to 'test-app'
# Your code here

Need a hint?

Use Properties props = new Properties(); and then props.put(StreamsConfig.APPLICATION_ID_CONFIG, "test-app");

3

Implement the core processing logic

Use mapValues on inputStream to convert all values to uppercase and assign the result to a KStream<String, String> called upperStream. Then send upperStream to the output topic output-topic using to("output-topic").

Kafka

import java.util.Properties;
import org.apache.kafka.streams.StreamsConfig;
import org.apache.kafka.streams.StreamsBuilder;
import org.apache.kafka.streams.kstream.KStream;

StreamsBuilder builder = new StreamsBuilder();
KStream inputStream = builder.stream("input-topic");

Properties props = new Properties();
props.put(StreamsConfig.APPLICATION_ID_CONFIG, "test-app");

# Use mapValues on inputStream to convert values to uppercase and assign to upperStream
# Send upperStream to 'output-topic'
# Your code here

Need a hint?

Use inputStream.mapValues(value -> value.toUpperCase()) and then upperStream.to("output-topic")

4

Print the test output

Print the string "Topology test passed" to confirm the test ran successfully.

Kafka

import java.util.Properties;
import org.apache.kafka.streams.StreamsConfig;
import org.apache.kafka.streams.StreamsBuilder;
import org.apache.kafka.streams.kstream.KStream;

StreamsBuilder builder = new StreamsBuilder();
KStream inputStream = builder.stream("input-topic");

Properties props = new Properties();
props.put(StreamsConfig.APPLICATION_ID_CONFIG, "test-app");

KStream upperStream = inputStream.mapValues(value -> value.toUpperCase());
upperStream.to("output-topic");

# Print "Topology test passed" to confirm success
# Your code here

Need a hint?

Use System.out.println("Topology test passed"); to print the confirmation message.