Java producer client in Kafka - Commands & Configuration

Introduction

Sending messages to Kafka topics is essential for building real-time data pipelines. The Java producer client lets you write programs that send data to Kafka servers easily and reliably.

When you want to send user activity logs from a web app to Kafka for processing.

When you need to stream sensor data from IoT devices into Kafka topics.

When your backend service must publish events to Kafka for other services to consume.

When you want to batch and send messages efficiently to Kafka with retries.

When you want to control message keys and partitions for ordering guarantees.

Config File - producer.properties

producer.properties

bootstrap.servers=localhost:9092
key.serializer=org.apache.kafka.common.serialization.StringSerializer
value.serializer=org.apache.kafka.common.serialization.StringSerializer
acks=all
retries=3
linger.ms=5

This file configures the Kafka producer client:

• bootstrap.servers: Kafka server address to connect.
• key.serializer and value.serializer: Classes to convert keys and values to bytes.
• acks: Wait for all replicas to confirm message write for reliability.
• retries: Number of retry attempts on failure.
• linger.ms: Time to wait before sending batch to improve throughput.

Commands

Compile the Java producer client code with Kafka client library in the classpath.

Terminal

javac -cp kafka-clients-3.5.1.jar:. KafkaProducerExample.java

Expected Output

No output (command runs silently)

Run the compiled Java producer client to send messages to Kafka.

Terminal

java -cp kafka-clients-3.5.1.jar:. KafkaProducerExample

Expected Output

Sent message with key=key1 and value=Hello Kafka

Sent message with key=key2 and value=Welcome to Kafka

Key Concept

If you remember nothing else from this pattern, remember: configure serializers and bootstrap servers correctly to send messages to Kafka.

Code Example

Kafka

import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.ProducerRecord;
import org.apache.kafka.clients.producer.RecordMetadata;
import java.util.Properties;

public class KafkaProducerExample {
    public static void main(String[] args) {
        Properties props = new Properties();
        props.put("bootstrap.servers", "localhost:9092");
        props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
        props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
        props.put("acks", "all");
        props.put("retries", 3);
        props.put("linger.ms", 5);

        try (KafkaProducer<String, String> producer = new KafkaProducer<>(props)) {
            ProducerRecord<String, String> record1 = new ProducerRecord<>("my-topic", "key1", "Hello Kafka");
            RecordMetadata metadata1 = producer.send(record1).get();
            System.out.println("Sent message with key=key1 and value=Hello Kafka");

            ProducerRecord<String, String> record2 = new ProducerRecord<>("my-topic", "key2", "Welcome to Kafka");
            RecordMetadata metadata2 = producer.send(record2).get();
            System.out.println("Sent message with key=key2 and value=Welcome to Kafka");
        } catch (Exception e) {
            e.printStackTrace();
        }
    }
}

Common Mistakes

Not setting the key.serializer and value.serializer properties.

Kafka producer cannot convert messages to bytes and will throw serialization errors.

Always specify the correct serializer classes for your message key and value types.

Using incorrect bootstrap server address or port.

Producer cannot connect to Kafka cluster and messages fail to send.

Verify Kafka server address and port are correct and reachable.

Not calling producer.close() after sending messages.

Messages may remain in buffer and not be sent to Kafka.

Always close the producer to flush and release resources.

Summary

Create a properties file to configure Kafka producer settings like server address and serializers.

Compile and run the Java producer client code to send messages to a Kafka topic.

Always close the producer to ensure all messages are sent and resources freed.

Practice

1. What is the primary role of a Java producer client in Kafka?

easy

A. To send messages to Kafka topics

B. To consume messages from Kafka topics

C. To manage Kafka brokers

D. To monitor Kafka cluster health

Solution

1. Step 1: Understand Kafka client roles

   Kafka has producers that send messages and consumers that receive messages.

2. Step 2: Identify the producer's function

   A Java producer client specifically sends messages to Kafka topics.

3. Final Answer:

   To send messages to Kafka topics -> Option A

4. Quick Check:

   Producer = send messages [OK]

Hint: Producer sends, consumer receives messages [OK]

Common Mistakes:

• Confusing producer with consumer
• Thinking producer manages brokers
• Assuming producer monitors cluster

2. Which of the following is the correct way to set the serializer for the key in a Java Kafka producer?

easy

A. props.add("key.serializer", "StringSerializer");

B. props.set("key.serializer", "StringSerializer");

C. props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");

D. props.put("key.serializer", String.class);

Solution

1. Step 1: Recall how to set properties in Kafka producer

   Kafka producer uses Properties object with put(key, value) method.

2. Step 2: Identify correct serializer setting syntax

   The key.serializer must be set to the full class name as a string.

3. Final Answer:

   props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer"); -> Option C

4. Quick Check:

   Use props.put with full serializer class name [OK]

Hint: Use props.put with full serializer class name string [OK]

Common Mistakes:

• Using props.set or props.add instead of put
• Not using full class name string
• Passing class type instead of string

3. What will be the output of the following Java Kafka producer code snippet?

import java.util.Properties;
import org.apache.kafka.clients.producer.*;
Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");

Producer<String, String> producer = new KafkaProducer<>(props);
ProducerRecord<String, String> record = new ProducerRecord<>("my-topic", "key1", "value1");
producer.send(record);
producer.close();
System.out.println("Message sent");

medium

A. Message sent

B. Compilation error due to missing imports

C. Runtime error due to missing Kafka server

D. No output

Solution

1. Step 1: Analyze the code flow

   The code creates a producer, sends a message, closes the producer, then prints "Message sent".

2. Step 2: Consider output behavior

   Assuming Kafka server is running and imports are correct, the print statement executes after send and close.

3. Final Answer:

   Message sent -> Option A

4. Quick Check:

   Print after send and close = "Message sent" [OK]

Hint: Print statement runs after send, shows "Message sent" [OK]

Common Mistakes:

• Assuming send() blocks until message delivered
• Ignoring that print runs regardless of Kafka server
• Confusing compile errors with runtime issues

4. Identify the error in this Java Kafka producer code snippet:

import java.util.Properties;
import org.apache.kafka.clients.producer.*;
Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");

KafkaProducer<String, String> producer = new KafkaProducer<>(props);
ProducerRecord<String, String> record = new ProducerRecord<>("my-topic");
producer.send(record);
producer.close();

medium

A. Producer not closed properly

B. Incorrect serializer class name

C. Missing bootstrap.servers property

D. ProducerRecord missing key argument

Solution

1. Step 1: Check ProducerRecord constructor usage

   The constructor used has only topic, but the generic type expects key and value.

2. Step 2: Identify mismatch in arguments

   ProducerRecord<String, String> expects key and value, but no key/value is provided, causing a compile error.

3. Final Answer:

   ProducerRecord missing key argument -> Option D

4. Quick Check:

   ProducerRecord requires key and value [OK]

Hint: ProducerRecord needs key and value if generics specify both [OK]

Common Mistakes:

• Using wrong ProducerRecord constructor
• Ignoring generic type mismatch
• Assuming value-only constructor works with key type

5. You want to send multiple messages to different Kafka topics using a single Java producer instance efficiently. Which approach is best?

hard

A. Create a new KafkaProducer for each message

B. Reuse one KafkaProducer and send ProducerRecords with different topics

C. Use multiple threads each with its own KafkaProducer

D. Send all messages to one topic and then move them to others

Solution

1. Step 1: Understand KafkaProducer resource usage

   Creating a producer is expensive; reusing one instance is efficient.

2. Step 2: Sending to multiple topics

   ProducerRecords can specify different topics; one producer can send to many topics.

3. Step 3: Evaluate other options

   Creating many producers wastes resources; sending all to one topic is incorrect.

4. Final Answer:

   Reuse one KafkaProducer and send ProducerRecords with different topics -> Option B

5. Quick Check:

   One producer, multiple topics via records [OK]

Hint: Reuse one producer; specify topic per ProducerRecord [OK]

Common Mistakes:

• Creating many producers wastes resources
• Sending all messages to one topic incorrectly
• Assuming each thread needs its own producer