How to Use Spring-Kafka: Simple Guide with Examples

To use spring-kafka, add the dependency to your Spring Boot project, configure Kafka properties, and create @KafkaListener methods to consume messages and KafkaTemplate to send messages. Spring-Kafka simplifies integrating Kafka by handling serialization, deserialization, and listener container management.

Syntax

Spring-Kafka uses annotations and configuration to connect your Spring Boot app with Kafka. Key parts include:

• @KafkaListener: Marks a method to receive messages from a Kafka topic.
• KafkaTemplate: Sends messages to Kafka topics.
• Configuration properties: Set Kafka server address, serializers, and group IDs.

properties/java

spring.kafka.bootstrap-servers=localhost:9092
spring.kafka.consumer.group-id=my-group
spring.kafka.consumer.key-deserializer=org.apache.kafka.common.serialization.StringDeserializer
spring.kafka.consumer.value-deserializer=org.apache.kafka.common.serialization.StringDeserializer
spring.kafka.producer.key-serializer=org.apache.kafka.common.serialization.StringSerializer
spring.kafka.producer.value-serializer=org.apache.kafka.common.serialization.StringSerializer

// Example listener method
@KafkaListener(topics = "my-topic", groupId = "my-group")
public void listen(String message) {
    System.out.println("Received: " + message);
}

// Example sending message
@Autowired
private KafkaTemplate<String, String> kafkaTemplate;

public void sendMessage(String msg) {
    kafkaTemplate.send("my-topic", msg);
}

Example

This example shows a simple Spring Boot application that sends and receives messages using spring-kafka.

java

package com.example.kafka;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.kafka.annotation.KafkaListener;
import org.springframework.kafka.core.KafkaTemplate;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.RestController;

@SpringBootApplication
public class KafkaDemoApplication {
    public static void main(String[] args) {
        SpringApplication.run(KafkaDemoApplication.class, args);
    }
}

@RestController
class KafkaController {

    @Autowired
    private KafkaTemplate<String, String> kafkaTemplate;

    @GetMapping("/send")
    public String sendMessage(@RequestParam String msg) {
        kafkaTemplate.send("test-topic", msg);
        return "Message sent: " + msg;
    }

    @KafkaListener(topics = "test-topic", groupId = "group_id")
    public void listen(String message) {
        System.out.println("Received message: " + message);
    }
}

Output

Received message: HelloKafka

Common Pitfalls

Common mistakes when using spring-kafka include:

• Not setting the correct groupId for consumers, causing messages to be missed or duplicated.
• Forgetting to configure serializers and deserializers, leading to serialization errors.
• Using @KafkaListener without enabling @EnableKafka annotation in configuration.
• Not handling exceptions inside listener methods, which can stop message consumption.

java

/* Wrong: Missing @EnableKafka annotation
@SpringBootApplication
public class App {}

@KafkaListener(topics = "topic")
public void listen(String msg) {}

// This will not work because listener container is not enabled.
*/

/* Right: Add @EnableKafka to enable listener support
import org.springframework.kafka.annotation.EnableKafka;

@EnableKafka
@SpringBootApplication
public class App {}

@KafkaListener(topics = "topic")
public void listen(String msg) {}
*/

Quick Reference

Here is a quick summary of key spring-kafka components and properties:

Component/Property	Purpose
@KafkaListener	Marks method to consume Kafka messages
KafkaTemplate	Sends messages to Kafka topics
spring.kafka.bootstrap-servers	Kafka server address
spring.kafka.consumer.group-id	Consumer group identifier
spring.kafka.consumer.key-deserializer	Class to deserialize message keys
spring.kafka.consumer.value-deserializer	Class to deserialize message values
spring.kafka.producer.key-serializer	Class to serialize message keys
spring.kafka.producer.value-serializer	Class to serialize message values
@EnableKafka	Enables Kafka listener annotation processing

Key Takeaways

Add spring-kafka dependency and configure Kafka properties in application.yml or application.properties.

Use @KafkaListener to receive messages and KafkaTemplate to send messages.

Always enable @EnableKafka annotation to activate listener support.

Set correct serializers and deserializers to avoid message format errors.

Handle exceptions inside listeners to keep consuming messages reliably.