How to Configure Kafka Producer: Simple Setup Guide

To configure a KafkaProducer, set essential properties like bootstrap.servers for Kafka brokers, key.serializer, and value.serializer to convert data into bytes. Additional settings like acks and retries control message delivery guarantees and error handling.

Syntax

The Kafka producer configuration uses a Properties object in Java to set key-value pairs for connection and message settings.

Key properties include:

• bootstrap.servers: Kafka broker addresses
• key.serializer: Class to serialize message keys
• value.serializer: Class to serialize message values
• acks: Message acknowledgment level
• retries: Number of retry attempts on failure

java

Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("acks", "all");
props.put("retries", "3");
KafkaProducer<String, String> producer = new KafkaProducer<>(props);

Example

This example shows a simple Kafka producer sending a message to a topic named test-topic. It demonstrates setting up the producer, sending a message, and closing the producer.

java

import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.ProducerRecord;
import org.apache.kafka.clients.producer.RecordMetadata;
import java.util.Properties;
import java.util.concurrent.Future;

public class SimpleProducer {
    public static void main(String[] args) throws Exception {
        Properties props = new Properties();
        props.put("bootstrap.servers", "localhost:9092");
        props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
        props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
        props.put("acks", "all");
        props.put("retries", "3");

        KafkaProducer<String, String> producer = new KafkaProducer<>(props);
        ProducerRecord<String, String> record = new ProducerRecord<>("test-topic", "key1", "Hello Kafka");

        Future<RecordMetadata> future = producer.send(record);
        RecordMetadata metadata = future.get();

        System.out.println("Message sent to topic " + metadata.topic() + " partition " + metadata.partition() + " offset " + metadata.offset());

        producer.close();
    }
}

Output

Message sent to topic test-topic partition 0 offset 15

Common Pitfalls

Common mistakes when configuring Kafka producers include:

• Not setting bootstrap.servers correctly, causing connection failures.
• Using wrong serializers that do not match the message data type.
• Ignoring acks setting, which can lead to lost messages if set to 0.
• Not handling retries or exceptions, causing message loss on transient errors.

Always verify broker addresses, use matching serializers, and configure acks and retries for reliability.

java

/* Wrong way: Missing serializers and wrong bootstrap server */
Properties wrongProps = new Properties();
wrongProps.put("bootstrap.servers", "wronghost:9092");
// Missing serializers

/* Right way: Correct bootstrap and serializers */
Properties rightProps = new Properties();
rightProps.put("bootstrap.servers", "localhost:9092");
rightProps.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
rightProps.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");

Quick Reference

Property	Description	Example Value
bootstrap.servers	Kafka broker addresses	"localhost:9092"
key.serializer	Serializer class for message keys	"org.apache.kafka.common.serialization.StringSerializer"
value.serializer	Serializer class for message values	"org.apache.kafka.common.serialization.StringSerializer"
acks	Message acknowledgment level (0, 1, all)	"all"
retries	Number of retry attempts on failure	"3"
batch.size	Batch size in bytes	16384
linger.ms	Delay to wait before sending batch	1

Key Takeaways

Always set bootstrap.servers to point to your Kafka brokers.

Use matching serializers for your message key and value types.

Configure acks to control message delivery guarantees; 'all' is safest.

Set retries to handle transient send failures automatically.

Test your producer configuration with a simple message send before production use.