Windowed operations in Kafka - Commands & Configuration

Introduction

Windowed operations let you group streaming data into time chunks to analyze events that happen close together. This helps find patterns or counts over short periods instead of the whole stream.

When you want to count how many times a user clicks a button every minute.

When you need to detect spikes in sensor data over 5-minute intervals.

When you want to aggregate sales totals by hour from a continuous stream.

When you want to join two streams but only consider events that happened within the same time window.

When you want to calculate moving averages or trends over recent data chunks.

Config File - windowed_streams.properties

windowed_streams.properties

bootstrap.servers=localhost:9092
application.id=windowed-operations-app
processing.guarantee=exactly_once
cache.max.bytes.buffering=10485760
commit.interval.ms=1000

This configuration file sets up the Kafka Streams application to connect to the Kafka server at localhost:9092. The application.id uniquely identifies the stream processing app. processing.guarantee=exactly_once ensures data is processed without duplication. cache.max.bytes.buffering controls memory buffering for performance. commit.interval.ms sets how often processed results are saved.

Commands

Create a Kafka topic named 'clicks' with 3 partitions to hold streaming click events.

Terminal

kafka-topics --create --topic clicks --bootstrap-server localhost:9092 --partitions 3 --replication-factor 1

Expected Output

Created topic clicks.

→

--partitions - Number of partitions for parallelism

→

--replication-factor - Number of copies for fault tolerance

Start a producer to send click events to the 'clicks' topic. You can type messages here to simulate clicks.

Terminal

kafka-console-producer --topic clicks --bootstrap-server localhost:9092

Expected Output

No output (command runs silently)

→

--topic - Topic to send messages to

→

--bootstrap-server - Kafka server address

Run the Kafka Streams app that reads from 'clicks', counts events in 1-minute windows, and writes results to 'clicks-counts'.

Terminal

kafka-streams-application --config windowed_streams.properties --input-topic clicks --output-topic clicks-counts --window-size 60000

Expected Output

Starting Kafka Streams application with window size 60000 ms

Processing events...

→

--config - Configuration file for the streams app

→

--input-topic - Input topic to read events from

→

--output-topic - Output topic to write windowed counts

→

--window-size - Size of the time window in milliseconds

Consume and display the windowed counts from the 'clicks-counts' topic to verify the windowed operation results.

Terminal

kafka-console-consumer --topic clicks-counts --bootstrap-server localhost:9092 --from-beginning

Expected Output

{"window_start":1680000000000,"count":5}

{"window_start":1680000060000,"count":3}

→

--from-beginning - Read all messages from the start

Key Concept

If you remember nothing else from this pattern, remember: windowed operations group streaming data into fixed time chunks to analyze recent events together.

Common Mistakes

Not setting the window size correctly or forgetting to specify it.

Without a window size, Kafka Streams cannot group events by time, so windowed operations won't work.

Always specify the window size in milliseconds when defining windowed operations.

Using a topic with too few partitions for the expected load.

Too few partitions limit parallel processing and can cause bottlenecks.

Create topics with enough partitions to handle your data volume and processing needs.

Not consuming from the output topic to verify results.

Without checking output, you won't know if windowed operations are working as expected.

Use a console consumer to read from the output topic and confirm windowed counts.

Summary

Create a Kafka topic to hold streaming events.

Run a Kafka Streams app configured to group events into time windows.

Produce events to the input topic and consume windowed results from the output topic.