How to Use Kafka with Spring Boot: Simple Guide

To use Kafka with Spring Boot, add the spring-kafka dependency, configure Kafka properties in application.yml, and create producer and consumer beans. Spring Boot simplifies Kafka integration by managing connection and message serialization automatically.

Syntax

Using Kafka with Spring Boot involves three main parts:

• Dependency: Add spring-kafka to your project.
• Configuration: Set Kafka server details and topic in application.yml or application.properties.
• Producer and Consumer: Create beans or services to send and receive messages.

yaml

dependencies {
    implementation 'org.springframework.kafka:spring-kafka'
}

spring:
  kafka:
    bootstrap-servers: localhost:9092
    consumer:
      group-id: my-group
      auto-offset-reset: earliest
      key-deserializer: org.apache.kafka.common.serialization.StringDeserializer
      value-deserializer: org.apache.kafka.common.serialization.StringDeserializer
    producer:
      key-serializer: org.apache.kafka.common.serialization.StringSerializer
      value-serializer: org.apache.kafka.common.serialization.StringSerializer

Example

This example shows a Spring Boot application that sends a message to Kafka and listens for messages on a topic.

java

package com.example.kafkademo;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.kafka.annotation.KafkaListener;
import org.springframework.kafka.core.KafkaTemplate;
import org.springframework.stereotype.Service;

@SpringBootApplication
public class KafkaDemoApplication {

    public static void main(String[] args) {
        SpringApplication.run(KafkaDemoApplication.class, args);
    }
}

@Service
class KafkaProducer {
    private final KafkaTemplate<String, String> kafkaTemplate;

    public KafkaProducer(KafkaTemplate<String, String> kafkaTemplate) {
        this.kafkaTemplate = kafkaTemplate;
    }

    public void sendMessage(String topic, String message) {
        kafkaTemplate.send(topic, message);
    }
}

@Service
class KafkaConsumer {

    @KafkaListener(topics = "test-topic", groupId = "my-group")
    public void listen(String message) {
        System.out.println("Received message: " + message);
    }
}

Output

Received message: Hello Kafka

Common Pitfalls

Common mistakes when using Kafka with Spring Boot include:

• Not setting the correct bootstrap-servers address, causing connection failures.
• Missing or incorrect serializer/deserializer classes, leading to message format errors.
• Forgetting to add @KafkaListener annotation on consumer methods, so messages are not received.
• Not configuring consumer group IDs, which can cause unexpected message consumption behavior.

properties

/* Wrong: Missing serializer config */
spring.kafka.producer.key-serializer= 
spring.kafka.producer.value-serializer= 

/* Right: Proper serializer config */
spring.kafka.producer.key-serializer=org.apache.kafka.common.serialization.StringSerializer
spring.kafka.producer.value-serializer=org.apache.kafka.common.serialization.StringSerializer

Quick Reference

Remember these key points when using Kafka with Spring Boot:

• Add spring-kafka dependency.
• Configure bootstrap-servers and serializers in application.yml.
• Use KafkaTemplate to send messages.
• Use @KafkaListener to receive messages.
• Set consumer group ID to manage message consumption.

Key Takeaways

Add spring-kafka dependency and configure Kafka properties in your Spring Boot app.

Use KafkaTemplate to produce messages and @KafkaListener to consume them.

Always set correct serializers and deserializers to avoid message errors.

Configure consumer group IDs to control message consumption behavior.

Test connection to Kafka broker by verifying bootstrap-servers setting.