Kafka integration basics in Spring Boot

Introduction

Kafka integration lets your Spring Boot app send and receive messages easily. It helps different parts of your system talk to each other without waiting.

You want to send data from one app to another without delay.

You need to process messages in order and handle many at once.

You want to build a system that keeps working even if parts fail.

You want to collect logs or events from many sources in one place.

Syntax

Spring Boot

spring.kafka.bootstrap-servers=localhost:9092

@Bean
public NewTopic topic() {
    return TopicBuilder.name("myTopic").build();
}

@KafkaListener(topics = "myTopic")
public void listen(String message) {
    System.out.println("Received: " + message);
}

@Autowired
private KafkaTemplate<String, String> kafkaTemplate;

public void sendMessage(String msg) {
    kafkaTemplate.send("myTopic", msg);
}

Use spring.kafka.bootstrap-servers to set Kafka server address.

@KafkaListener marks a method to receive messages from a topic.

Examples

This listens to the 'orders' topic and prints each order message.

Spring Boot

@KafkaListener(topics = "orders")
public void receiveOrder(String order) {
    System.out.println("Order received: " + order);
}

This sends a simple text message to the 'notifications' topic.

Spring Boot

kafkaTemplate.send("notifications", "Hello World!");

This creates a topic named 'events' with 3 partitions and 1 replica.

Spring Boot

@Bean
public NewTopic createTopic() {
    return TopicBuilder.name("events").partitions(3).replicas(1).build();
}

Sample Program

This Spring Boot app creates a Kafka topic named 'demoTopic'. It sends messages to this topic and listens for messages from it. When a message arrives, it prints it to the console.

Spring Boot

package com.example.kafkademo;

import org.apache.kafka.clients.admin.NewTopic;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.context.annotation.Bean;
import org.springframework.kafka.annotation.KafkaListener;
import org.springframework.kafka.annotation.EnableKafka;
import org.springframework.kafka.core.KafkaTemplate;
import org.springframework.beans.factory.annotation.Autowired;

@EnableKafka
@SpringBootApplication
public class KafkaDemoApplication {

    public static void main(String[] args) {
        SpringApplication.run(KafkaDemoApplication.class, args);
    }

    @Bean
    public NewTopic demoTopic() {
        return org.springframework.kafka.config.TopicBuilder.name("demoTopic").build();
    }

    @Autowired
    private KafkaTemplate<String, String> kafkaTemplate;

    public void sendMessage(String message) {
        kafkaTemplate.send("demoTopic", message);
    }

    @KafkaListener(topics = "demoTopic")
    public void listen(String message) {
        System.out.println("Received message: " + message);
    }
}

Important Notes

Make sure Kafka server is running on the address you set in bootstrap-servers.

Use @KafkaListener only on methods inside Spring-managed beans.

Sending and receiving messages happen asynchronously, so messages may arrive shortly after sending.

Summary

Kafka integration in Spring Boot helps apps send and receive messages easily.

Use @KafkaListener to listen to topics and KafkaTemplate to send messages.

Define topics with NewTopic beans to manage Kafka topics from your app.

Practice

1. What is the primary purpose of @KafkaListener in a Spring Boot application?

easy

A. To listen and consume messages from a Kafka topic

B. To send messages to a Kafka topic

C. To configure Kafka broker settings

D. To create new Kafka topics automatically

Solution

1. Step 1: Understand the role of @KafkaListener

   This annotation marks a method to receive messages from a Kafka topic automatically.

2. Step 2: Differentiate sending and listening

   Sending messages is done with KafkaTemplate, not @KafkaListener.

3. Final Answer:

   To listen and consume messages from a Kafka topic -> Option A

4. Quick Check:

   @KafkaListener listens to topics [OK]

Hint: Remember: Listener receives, Template sends [OK]

Common Mistakes:

• Confusing @KafkaListener with KafkaTemplate
• Thinking @KafkaListener creates topics
• Assuming it configures Kafka brokers

2. Which of the following is the correct way to send a message to a Kafka topic using Spring Boot?

easy

A. new KafkaProducer().send("topicName", "message")

B. @KafkaListener(topic = "topicName")

C. KafkaTemplate.listen("topicName", "message")

D. kafkaTemplate.send("topicName", "message")

Solution

1. Step 1: Identify the sending method in Spring Boot

   Spring Boot uses KafkaTemplate with the send() method to send messages.

2. Step 2: Check syntax correctness

   kafkaTemplate.send("topicName", "message") is the correct syntax to send a message.

3. Final Answer:

   kafkaTemplate.send("topicName", "message") -> Option D

4. Quick Check:

   Sending uses KafkaTemplate.send() [OK]

Hint: Send messages with kafkaTemplate.send() [OK]

Common Mistakes:

• Using @KafkaListener to send messages
• Calling listen() instead of send()
• Instantiating KafkaProducer manually in Spring Boot

3. Given this Spring Boot Kafka listener method:

@KafkaListener(topics = "myTopic")
public void listen(String message) {
    System.out.println("Received: " + message);
}

What will be printed if a message "Hello Kafka" is sent to myTopic?

medium

A. Received: Hello Kafka

B. Hello Kafka

C. Received message: Hello Kafka

D. Error: No message received

Solution

1. Step 1: Analyze the listener method output

   The method prints "Received: " concatenated with the message string.

2. Step 2: Substitute the message value

   Message is "Hello Kafka", so output is "Received: Hello Kafka".

3. Final Answer:

   Received: Hello Kafka -> Option A

4. Quick Check:

   Prints "Received: " + message [OK]

Hint: Check print statement format carefully [OK]

Common Mistakes:

• Ignoring the prefix in the print statement
• Assuming only the message prints
• Confusing topic name with output

4. Consider this Spring Boot Kafka configuration snippet:

@Bean
public NewTopic topic() {
    return new NewTopic("testTopic", 1, (short) 1);
}

What is the likely error if the application fails to create the topic?

medium

A. Incorrect topic name format

B. Missing Kafka broker connection settings

C. NewTopic requires 3 parameters but only 2 provided

D. NewTopic bean must be annotated with @KafkaListener

Solution

1. Step 1: Understand NewTopic bean usage

   NewTopic defines topic name, partitions, and replication factor correctly here.

2. Step 2: Identify common cause for topic creation failure

   Failure usually happens if Kafka broker connection settings are missing or incorrect.

3. Final Answer:

   Missing Kafka broker connection settings -> Option B

4. Quick Check:

   Broker settings needed for topic creation [OK]

Hint: Check Kafka broker config if topic creation fails [OK]

Common Mistakes:

• Thinking @KafkaListener is needed on NewTopic bean
• Assuming topic name format is invalid
• Miscounting NewTopic constructor parameters

5. You want to create a Spring Boot Kafka consumer that listens to two topics: topicA and topicB. Which of the following @KafkaListener annotations correctly achieves this?

hard

A. @KafkaListener(topics = "topicA, topicB")

B. @KafkaListener(topic = "topicA,topicB")

C. @KafkaListener(topics = {"topicA", "topicB"})

D. @KafkaListener(topics = ["topicA", "topicB"])

Solution

1. Step 1: Recall correct syntax for multiple topics

   Spring Boot expects an array of strings for multiple topics: topics = {"topicA", "topicB"}.

2. Step 2: Evaluate other options

   @KafkaListener(topic = "topicA,topicB") uses wrong attribute name and comma string, C uses string with comma but not array, D uses invalid syntax with square brackets.

3. Final Answer:

   @KafkaListener(topics = {"topicA", "topicB"}) -> Option C

4. Quick Check:

   Multiple topics require array syntax [OK]

Hint: Use curly braces for multiple topics in @KafkaListener [OK]

Common Mistakes:

• Using comma-separated string instead of array
• Wrong attribute name 'topic' instead of 'topics'
• Using square brackets instead of curly braces