0
0
Kafkadevops~10 mins

Testing stream topologies in Kafka - Commands & Configuration

Choose your learning style9 modes available
Introduction
When building applications that process data streams, you need to test how your stream processing logic works before running it live. Testing stream topologies helps you check if your data flows and transformations behave correctly.
When you want to verify that your Kafka Streams application processes input data correctly before deployment
When you need to simulate input and output topics to check your stream transformations
When you want to catch bugs early by running automated tests on your stream processing logic
When you want to test complex data flows involving joins, filters, and aggregations in your stream
When you want to ensure your stream topology handles edge cases and error scenarios properly
Commands
This command runs the unit tests in your Kafka Streams project, including tests for your stream topology using the TopologyTestDriver.
Terminal
mvn clean test
Expected OutputExpected
[INFO] --- maven-surefire-plugin:3.0.0-M5:test (default-test) @ kafka-streams-app --- [INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.345 s - in com.example.StreamsTopologyTest [INFO] [INFO] Results: [INFO] [INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0 [INFO] [INFO] BUILD SUCCESS
This command runs your Kafka Streams application in production mode after you have tested the topology.
Terminal
java -jar kafka-streams-app.jar
Expected OutputExpected
INFO KafkaStreams: Started Kafka Streams application INFO KafkaStreams: Stream thread started INFO KafkaStreams: Processing records from input topic
Key Concept

If you remember nothing else from this pattern, remember: use the TopologyTestDriver to simulate input and output topics and verify your stream processing logic without needing a live Kafka cluster.

Common Mistakes
Trying to test Kafka Streams logic by running the full application without mocks
This requires a live Kafka cluster and makes tests slow and flaky.
Use the TopologyTestDriver to run fast, isolated unit tests of your stream topology.
Not closing the TopologyTestDriver after tests
Leads to resource leaks and can cause tests to interfere with each other.
Always call close() on the TopologyTestDriver in a finally block or test teardown.
Not providing correct serializers and deserializers in tests
Causes serialization errors and test failures that do not reflect real logic issues.
Use the same Serdes as your application when creating the test driver and input/output records.
Summary
Use the TopologyTestDriver to simulate Kafka topics and test your stream processing logic.
Run 'mvn clean test' to execute your stream topology unit tests quickly without a live Kafka cluster.
After testing, run your Kafka Streams application normally with 'java -jar' to process real data.