Punctuators for time-based triggers in Kafka - Commands & Configuration

Introduction

Sometimes you want your Kafka Streams application to do something regularly, like every few seconds or minutes. Punctuators help by triggering actions based on time, so you can process or check data at fixed intervals.

When you want to update a dashboard every minute with the latest data from Kafka.

When you need to clean up old state data every hour in your stream processing app.

When you want to emit summary statistics every 10 seconds from a stream.

When you want to trigger alerts if no new data arrives within a certain time.

When you want to batch process events every fixed time period instead of immediately.

Commands

This command starts a Kafka producer to send messages to the topic 'example-topic'. We use this to generate data that the stream app will process.

Terminal

kafka-console-producer --broker-list localhost:9092 --topic example-topic

Expected Output

No output (command runs silently)

→

--broker-list - Specifies the Kafka broker address to connect to.

→

--topic - Specifies the topic to send messages to.

Runs a Kafka Streams example app that uses a punctuator to trigger an action every 5 seconds. This shows how time-based triggers work in practice.

Terminal

java -cp kafka-streams-examples.jar org.apache.kafka.streams.examples.punctuator.PunctuatorExample

Expected Output

Starting PunctuatorExample

Punctuator triggered at 2024-06-01T12:00:05

Punctuator triggered at 2024-06-01T12:00:10

Punctuator triggered at 2024-06-01T12:00:15

Starts a Kafka consumer to read messages from 'output-topic' where the punctuator app sends its output. This verifies the punctuator triggered actions.

Terminal

kafka-console-consumer --bootstrap-server localhost:9092 --topic output-topic --from-beginning

Expected Output

Punctuator output at 2024-06-01T12:00:05

Punctuator output at 2024-06-01T12:00:10

Punctuator output at 2024-06-01T12:00:15

→

--bootstrap-server - Specifies the Kafka broker address.

→

--topic - Specifies the topic to consume messages from.

→

--from-beginning - Reads all messages from the start of the topic.

Key Concept

If you remember nothing else from this pattern, remember: punctuators let your Kafka Streams app run code regularly based on time, not just when new data arrives.

Common Mistakes

Not scheduling the punctuator with the correct interval or forgetting to call schedule()

The punctuator will never run if it is not scheduled properly, so no time-based actions happen.

Always call context.schedule() with the desired interval inside your processor to activate the punctuator.

Using wall-clock time instead of stream time for scheduling

This can cause unexpected behavior if event time and processing time differ, leading to missed or early triggers.

Choose the correct time type (STREAM_TIME or WALL_CLOCK_TIME) based on your use case when scheduling.

Doing heavy processing inside the punctuator callback

It can block the stream thread and slow down processing of other records.

Keep punctuator code lightweight or offload heavy work asynchronously.

Summary

Use kafka-console-producer to send test messages to a topic your stream app reads.

Run a Kafka Streams app that schedules a punctuator to trigger actions every fixed time interval.

Use kafka-console-consumer to verify the punctuator output messages appear as expected.