Producer API basics in Kafka - Commands & Configuration

Introduction

Sending messages to a Kafka topic is done using the Producer API. It helps you put data into Kafka so other applications can read it later.

When you want to send user activity logs from a web app to Kafka for processing.

When you need to push sensor data from IoT devices into Kafka for real-time analysis.

When your application generates events that other services should consume asynchronously.

When you want to batch and send messages efficiently to Kafka topics.

When you need to ensure messages are sent reliably with acknowledgments.

Config File - producer.properties

producer.properties

bootstrap.servers=localhost:9092
key.serializer=org.apache.kafka.common.serialization.StringSerializer
value.serializer=org.apache.kafka.common.serialization.StringSerializer
acks=all
retries=3
batch.size=16384
linger.ms=1
buffer.memory=33554432

bootstrap.servers: Kafka server addresses to connect to.

key.serializer and value.serializer: Convert keys and values to bytes.

acks: Wait for all replicas to confirm message receipt for reliability.

retries: Number of times to retry sending on failure.

batch.size and linger.ms: Control batching for efficiency.

buffer.memory: Memory size for buffering messages before sending.

Commands

Starts a simple command-line Kafka producer to send messages to 'example-topic'. You type messages and press Enter to send each one.

Terminal

kafka-console-producer --broker-list localhost:9092 --topic example-topic

Expected Output

No output (command runs silently)

→

--broker-list - Specifies Kafka server addresses to connect to.

→

--topic - Specifies the Kafka topic to send messages to.

Sends a single message 'Hello Kafka' to 'example-topic' using the console producer.

Terminal

echo "Hello Kafka" | kafka-console-producer --broker-list localhost:9092 --topic example-topic

Expected Output

No output (command runs silently)

→

--broker-list - Specifies Kafka server addresses to connect to.

→

--topic - Specifies the Kafka topic to send messages to.

Reads one message from 'example-topic' starting from the beginning to verify the message was sent.

Terminal

kafka-console-consumer --bootstrap-server localhost:9092 --topic example-topic --from-beginning --max-messages 1

Expected Output

Hello Kafka

→

--bootstrap-server - Specifies Kafka server addresses to connect to.

→

--topic - Specifies the Kafka topic to read messages from.

→

--from-beginning - Reads messages from the start of the topic.

→

--max-messages - Limits the number of messages to read.

Key Concept

If you remember nothing else from this pattern, remember: the Producer API sends messages to Kafka topics so other apps can consume them later.

Common Mistakes

Not setting the correct serializers for key and value.

Kafka needs to convert data to bytes; wrong serializers cause errors or unreadable messages.

Always set key.serializer and value.serializer to match your data types, like StringSerializer for text.

Using the wrong broker address or port.

The producer cannot connect to Kafka, so messages fail to send.

Verify Kafka is running and use the correct host and port in bootstrap.servers or --broker-list.

Not verifying messages by consuming them after sending.

You might think messages were sent but they were not, causing silent failures.

Use kafka-console-consumer to read messages and confirm delivery.

Summary

Use kafka-console-producer to send messages to a Kafka topic interactively or via echo.

Configure producer properties like serializers and broker addresses in a properties file.

Verify messages are sent by consuming them with kafka-console-consumer.

Practice

1. What is the main purpose of the Kafka Producer API?

easy

A. To send messages to Kafka topics

B. To read messages from Kafka topics

C. To manage Kafka cluster configurations

D. To monitor Kafka broker health

Solution

1. Step 1: Understand Kafka Producer role

   The Producer API is designed to send data to Kafka topics, acting as the message sender.

2. Step 2: Differentiate from other Kafka APIs

   Reading messages is done by the Consumer API, not the Producer API.

3. Final Answer:

   To send messages to Kafka topics -> Option A

4. Quick Check:

   Producer API = send messages [OK]

Hint: Producer API sends data; Consumer API reads data [OK]

Common Mistakes:

• Confusing Producer with Consumer API
• Thinking Producer manages cluster
• Assuming Producer monitors brokers

2. Which of the following is the correct way to create a Kafka Producer in Java?

easy

A. Producer<String> producer = new Producer(props);

B. KafkaProducer producer = new KafkaProducer<>(props);

C. KafkaProducer producer = new KafkaProducer();

D. KafkaProducer producer = KafkaProducer.create(props);

Solution

1. Step 1: Recall KafkaProducer constructor usage

   The KafkaProducer class requires properties and type parameters for key and value types.

2. Step 2: Check each option's syntax

   KafkaProducer producer = new KafkaProducer<>(props); correctly uses generics and constructor with properties. Others miss generics, constructor, or use invalid methods.

3. Final Answer:

   KafkaProducer producer = new KafkaProducer<>(props); -> Option B

4. Quick Check:

   Correct constructor with generics = KafkaProducer producer = new KafkaProducer<>(props); [OK]

Hint: Use new KafkaProducer<>(props) with key and value types [OK]

Common Mistakes:

• Omitting generics for key and value
• Using wrong constructor or method
• Not passing properties object

3. What will be the output of the following Java code snippet?

Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
KafkaProducer producer = new KafkaProducer<>(props);
ProducerRecord record = new ProducerRecord<>("my-topic", "key1", "value1");
producer.send(record);
System.out.println("Message sent");
producer.close();

medium

A. No output

B. Compilation error

C. Message sent

D. Runtime exception due to missing topic

Solution

1. Step 1: Analyze producer setup and send call

   The code sets up properties correctly, creates a producer, and sends a message to "my-topic" with key and value.

2. Step 2: Check output statement and close

   After sending, it prints "Message sent" and closes the producer. No errors expected if Kafka broker is running.

3. Final Answer:

   Message sent -> Option C

4. Quick Check:

   Successful send prints message [OK]

Hint: If send() succeeds, print statement runs [OK]

Common Mistakes:

• Assuming send() is synchronous and blocks
• Thinking missing topic causes error here
• Ignoring need for running Kafka broker

4. Identify the error in this Kafka Producer code snippet:

Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("key.serializer", "StringSerializer");
props.put("value.serializer", "StringSerializer");
KafkaProducer producer = new KafkaProducer<>(props);
ProducerRecord record = new ProducerRecord<>("topic", "key", "value");
producer.send(record);
producer.close();

medium

A. Missing topic name in ProducerRecord

B. Properties object not initialized

C. Producer not closed properly

D. Incorrect serializer class names

Solution

1. Step 1: Check serializer class names in properties

   The serializer values must be full class names like "org.apache.kafka.common.serialization.StringSerializer".

2. Step 2: Verify other code parts

   Topic name is present, producer is closed properly, and properties object is initialized correctly.

3. Final Answer:

   Incorrect serializer class names -> Option D

4. Quick Check:

   Serializer class names must be full package names [OK]

Hint: Use full serializer class names with package [OK]

Common Mistakes:

• Using short class names without package
• Forgetting to close producer
• Leaving topic name empty

5. You want to send messages with keys to ensure ordering per key in Kafka. Which ProducerRecord constructor should you use?

hard

A. ProducerRecord(String topic, K key, V value)

B. ProducerRecord(String topic, V value)

C. ProducerRecord(K key, V value)

D. ProducerRecord(String topic)

Solution

1. Step 1: Understand ordering by key in Kafka

   Kafka guarantees ordering within a partition, and keys determine partition assignment.

2. Step 2: Identify correct ProducerRecord constructor

   To send with a key, use the constructor that includes topic, key, and value.

3. Final Answer:

   ProducerRecord(String topic, K key, V value) -> Option A

4. Quick Check:

   Use constructor with topic, key, and value for ordering [OK]

Hint: Include key in ProducerRecord to maintain order [OK]

Common Mistakes:

• Using constructor without key loses ordering
• Omitting topic causes errors
• Confusing key and value parameters