Exactly-once stream processing in Kafka - Mini Project: Build & Apply

Exactly-once stream processing with Kafka

📖 Scenario: You are working with a Kafka stream that processes orders. You want to ensure that each order is processed exactly once to avoid duplicates or missing data.

🎯 Goal: Build a simple Kafka Streams application that reads from an input topic, processes the data with exactly-once semantics enabled, and writes to an output topic.

📋 What You'll Learn

Create a Kafka Streams configuration with exactly-once processing enabled

Define input and output topics named orders-input and orders-output

Implement a simple stream processing topology that reads from orders-input and writes to orders-output

Print the final topology description to verify the setup

💡 Why This Matters

🌍 Real World

Exactly-once processing is critical in financial transactions, order processing, and inventory management to avoid duplicates or data loss.

💼 Career

Many companies require engineers to build reliable stream processing pipelines with Kafka that guarantee data consistency and fault tolerance.

1

Create Kafka Streams configuration

Create a Properties object called props and set the following properties exactly:
APPLICATION_ID_CONFIG to "exactly-once-app",
BOOTSTRAP_SERVERS_CONFIG to "localhost:9092",
and PROCESSING_GUARANTEE_CONFIG to StreamsConfig.EXACTLY_ONCE_V2.

Kafka

# Your code here

Need a hint?

Use props.put to set each configuration property exactly as specified.

2

Define input and output topics

Create two String variables named inputTopic and outputTopic and assign them the exact values "orders-input" and "orders-output" respectively.

Kafka

Properties props = new Properties();
props.put(StreamsConfig.APPLICATION_ID_CONFIG, "exactly-once-app");
props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
props.put(StreamsConfig.PROCESSING_GUARANTEE_CONFIG, StreamsConfig.EXACTLY_ONCE_V2);
# Your code here

Need a hint?

Declare two string variables with the exact topic names.

3

Build the Kafka Streams topology

Create a StreamsBuilder object called builder. Use it to create a KStream<String, String> named stream that reads from inputTopic. Then write this stream to outputTopic using the to() method.

Kafka

Properties props = new Properties();
props.put(StreamsConfig.APPLICATION_ID_CONFIG, "exactly-once-app");
props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
props.put(StreamsConfig.PROCESSING_GUARANTEE_CONFIG, StreamsConfig.EXACTLY_ONCE_V2);

String inputTopic = "orders-input";
String outputTopic = "orders-output";

# Your code here

Need a hint?

Use builder.stream() to read and stream.to() to write.

4

Print the topology description

Create a KafkaStreams object called streams using builder.build() and props. Then print the topology description by calling streams.toString() inside a System.out.println().

Kafka

Properties props = new Properties();
props.put(StreamsConfig.APPLICATION_ID_CONFIG, "exactly-once-app");
props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
props.put(StreamsConfig.PROCESSING_GUARANTEE_CONFIG, StreamsConfig.EXACTLY_ONCE_V2);

String inputTopic = "orders-input";
String outputTopic = "orders-output";

StreamsBuilder builder = new StreamsBuilder();
KStream stream = builder.stream(inputTopic);
stream.to(outputTopic);

# Your code here

Need a hint?

Create the KafkaStreams object and print its topology description.