How to Use Kafka with Spring Boot: Simple Guide and Example

To use Kafka with Spring Boot, add the spring-kafka dependency, configure Kafka properties in application.yml, and create producer and consumer beans using @KafkaListener and KafkaTemplate. This setup enables sending and receiving messages asynchronously within your Spring Boot application.

Syntax

Using Kafka with Spring Boot involves these main parts:

• Dependency: Add spring-kafka to your project.
• Configuration: Set Kafka server details in application.yml or application.properties.
• Producer: Use KafkaTemplate to send messages.
• Consumer: Use @KafkaListener to receive messages.

yaml

dependencies {
  implementation 'org.springframework.kafka:spring-kafka'
}

spring:
  kafka:
    bootstrap-servers: localhost:9092
    consumer:
      group-id: my-group
      auto-offset-reset: earliest
      key-deserializer: org.apache.kafka.common.serialization.StringDeserializer
      value-deserializer: org.apache.kafka.common.serialization.StringDeserializer
    producer:
      key-serializer: org.apache.kafka.common.serialization.StringSerializer
      value-serializer: org.apache.kafka.common.serialization.StringSerializer

Example

This example shows a Spring Boot app sending a message to Kafka and receiving it with a listener.

java

package com.example.kafkademo;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.kafka.annotation.KafkaListener;
import org.springframework.kafka.core.KafkaTemplate;
import org.springframework.stereotype.Component;
import org.springframework.beans.factory.annotation.Autowired;

@SpringBootApplication
public class KafkaDemoApplication {

    public static void main(String[] args) {
        SpringApplication.run(KafkaDemoApplication.class, args);
    }
}

@Component
class KafkaProducer {

    @Autowired
    private KafkaTemplate<String, String> kafkaTemplate;

    public void sendMessage(String topic, String message) {
        kafkaTemplate.send(topic, message);
    }
}

@Component
class KafkaConsumer {

    @KafkaListener(topics = "test-topic", groupId = "my-group")
    public void listen(String message) {
        System.out.println("Received message: " + message);
    }
}

// Usage example (e.g., in a CommandLineRunner or REST controller):
// kafkaProducer.sendMessage("test-topic", "Hello Kafka with Spring Boot!");

Output

Received message: Hello Kafka with Spring Boot!

Common Pitfalls

Common mistakes when using Kafka with Spring Boot include:

• Not setting the correct bootstrap-servers address, causing connection failures.
• Missing or incorrect serializers/deserializers, leading to message format errors.
• Forgetting to set a group-id for consumers, which is required for consumer group management.
• Not enabling @EnableKafka annotation in older Spring Boot versions (before 2.3).

Always verify Kafka server is running and reachable before testing your app.

properties

/* Wrong: Missing group-id in consumer config */
spring.kafka.consumer.auto-offset-reset=earliest

/* Right: Add group-id */
spring.kafka.consumer.group-id=my-group

Quick Reference

Concept	Description
spring-kafka dependency	Provides Kafka integration for Spring Boot
bootstrap-servers	Kafka broker address, e.g., localhost:9092
KafkaTemplate	Used to send messages to Kafka topics
@KafkaListener	Marks a method to consume messages from a topic
group-id	Consumer group identifier for message load balancing
Serializer/Deserializer	Convert messages to/from bytes for Kafka

Key Takeaways

Add spring-kafka dependency and configure bootstrap servers to connect Spring Boot with Kafka.

Use KafkaTemplate to send messages and @KafkaListener to receive them asynchronously.

Always set consumer group-id and proper serializers/deserializers to avoid common errors.

Verify Kafka broker is running and reachable before running your Spring Boot app.

Keep configuration in application.yml or properties for easy management and changes.