How to Use Kafka Producer in Node.js: Simple Guide

To use a Kafka producer in Node.js, install the kafkajs library, create a Kafka client, then create and connect a producer to send messages to a Kafka topic. Use producer.send() to publish messages asynchronously.

Syntax

The main parts to use a Kafka producer in Node.js are:

• Import KafkaJS: Load the KafkaJS library.
• Create Kafka client: Define the Kafka broker address.
• Create producer: Initialize the producer instance.
• Connect producer: Establish connection to Kafka.
• Send messages: Use producer.send() with topic and messages.
• Disconnect: Close the producer connection when done.

javascript

const { Kafka } = require('kafkajs');

async function run() {
  const kafka = new Kafka({
    clientId: 'my-app',
    brokers: ['localhost:9092']
  });

  const producer = kafka.producer();

  await producer.connect();

  await producer.send({
    topic: 'topic-name',
    messages: [
      { value: 'message content' }
    ]
  });

  await producer.disconnect();
}

run().catch(console.error);

Example

This example shows how to send a simple message to a Kafka topic named test-topic. It connects the producer, sends one message, and then disconnects.

javascript

const { Kafka } = require('kafkajs');

async function runProducer() {
  const kafka = new Kafka({
    clientId: 'example-producer',
    brokers: ['localhost:9092']
  });

  const producer = kafka.producer();

  await producer.connect();

  await producer.send({
    topic: 'test-topic',
    messages: [
      { value: 'Hello Kafka from Node.js!' }
    ]
  });

  console.log('Message sent successfully');

  await producer.disconnect();
}

runProducer().catch(console.error);

Output

Message sent successfully

Common Pitfalls

Common mistakes when using Kafka producer in Node.js include:

• Not awaiting producer.connect() before sending messages causes errors.
• Forgetting to disconnect() can leave connections open.
• Sending messages to a non-existent topic without creating it first.
• Not handling errors with try-catch or .catch() on async calls.

Always ensure the Kafka broker is running and reachable at the specified address.

javascript

/* Wrong: Not awaiting connect */
const producer = kafka.producer();
producer.connect(); // Missing await
producer.send({ topic: 'test', messages: [{ value: 'msg' }] });

/* Right: Await connect */
await producer.connect();
await producer.send({ topic: 'test', messages: [{ value: 'msg' }] });

Quick Reference

Remember these key points when using Kafka producer in Node.js:

• Use kafkajs library for easy Kafka integration.
• Always await connect() before sending messages.
• Send messages as an array of objects with a value field.
• Disconnect the producer after use to free resources.
• Handle errors to avoid crashes.

Key Takeaways

Install and use the kafkajs library to create a Kafka producer in Node.js.

Always await producer.connect() before sending messages to avoid errors.

Send messages as an array with objects containing a value property.

Disconnect the producer after sending messages to close connections.

Handle errors properly to keep your application stable.