Java producer client in Kafka - Commands & Configuration

Introduction

Sending messages to Kafka topics is essential for building real-time data pipelines. The Java producer client lets you write programs that send data to Kafka servers easily and reliably.

When you want to send user activity logs from a web app to Kafka for processing.

When you need to stream sensor data from IoT devices into Kafka topics.

When your backend service must publish events to Kafka for other services to consume.

When you want to batch and send messages efficiently to Kafka with retries.

When you want to control message keys and partitions for ordering guarantees.

Config File - producer.properties

producer.properties

bootstrap.servers=localhost:9092
key.serializer=org.apache.kafka.common.serialization.StringSerializer
value.serializer=org.apache.kafka.common.serialization.StringSerializer
acks=all
retries=3
linger.ms=5

This file configures the Kafka producer client:

• bootstrap.servers: Kafka server address to connect.
• key.serializer and value.serializer: Classes to convert keys and values to bytes.
• acks: Wait for all replicas to confirm message write for reliability.
• retries: Number of retry attempts on failure.
• linger.ms: Time to wait before sending batch to improve throughput.

Commands

Compile the Java producer client code with Kafka client library in the classpath.

Terminal

javac -cp kafka-clients-3.5.1.jar:. KafkaProducerExample.java

Expected Output

No output (command runs silently)

Run the compiled Java producer client to send messages to Kafka.

Terminal

java -cp kafka-clients-3.5.1.jar:. KafkaProducerExample

Expected Output

Sent message with key=key1 and value=Hello Kafka

Sent message with key=key2 and value=Welcome to Kafka

Key Concept

If you remember nothing else from this pattern, remember: configure serializers and bootstrap servers correctly to send messages to Kafka.

Code Example

Kafka

import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.ProducerRecord;
import org.apache.kafka.clients.producer.RecordMetadata;
import java.util.Properties;

public class KafkaProducerExample {
    public static void main(String[] args) {
        Properties props = new Properties();
        props.put("bootstrap.servers", "localhost:9092");
        props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
        props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
        props.put("acks", "all");
        props.put("retries", 3);
        props.put("linger.ms", 5);

        try (KafkaProducer<String, String> producer = new KafkaProducer<>(props)) {
            ProducerRecord<String, String> record1 = new ProducerRecord<>("my-topic", "key1", "Hello Kafka");
            RecordMetadata metadata1 = producer.send(record1).get();
            System.out.println("Sent message with key=key1 and value=Hello Kafka");

            ProducerRecord<String, String> record2 = new ProducerRecord<>("my-topic", "key2", "Welcome to Kafka");
            RecordMetadata metadata2 = producer.send(record2).get();
            System.out.println("Sent message with key=key2 and value=Welcome to Kafka");
        } catch (Exception e) {
            e.printStackTrace();
        }
    }
}

Common Mistakes

Not setting the key.serializer and value.serializer properties.

Kafka producer cannot convert messages to bytes and will throw serialization errors.

Always specify the correct serializer classes for your message key and value types.

Using incorrect bootstrap server address or port.

Producer cannot connect to Kafka cluster and messages fail to send.

Verify Kafka server address and port are correct and reachable.

Not calling producer.close() after sending messages.

Messages may remain in buffer and not be sent to Kafka.

Always close the producer to flush and release resources.

Summary

Create a properties file to configure Kafka producer settings like server address and serializers.

Compile and run the Java producer client code to send messages to a Kafka topic.

Always close the producer to ensure all messages are sent and resources freed.