How to Use Kafka for Microservices: Simple Guide

Use Kafka as a message broker to enable asynchronous communication between microservices by producing and consuming events. Each microservice publishes events to Kafka topics and listens to relevant topics to react to changes, ensuring loose coupling and scalability.

Syntax

Kafka communication involves two main parts: Producer and Consumer. A Producer sends messages (events) to a topic. A Consumer subscribes to one or more topics to receive messages.

Key parts explained:

• Producer.send(topic, message): Sends a message to a Kafka topic.
• Consumer.subscribe(topics): Subscribes to topics to receive messages.
• Consumer.poll(): Retrieves messages from subscribed topics.

javascript

producer.send('order-events', { orderId: '123', status: 'created' });
consumer.subscribe(['order-events']);
const messages = consumer.poll();

Example

This example shows a simple Node.js microservice producing an event when an order is created, and another microservice consuming that event to process it.

javascript

import { Kafka } from 'kafkajs';

// Producer microservice
async function produceOrderCreated() {
  const kafka = new Kafka({ clientId: 'order-service', brokers: ['localhost:9092'] });
  const producer = kafka.producer();
  await producer.connect();
  await producer.send({
    topic: 'order-events',
    messages: [{ key: 'order1', value: JSON.stringify({ orderId: 'order1', status: 'created' }) }],
  });
  await producer.disconnect();
  console.log('Order created event sent');
}

// Consumer microservice
async function consumeOrderEvents() {
  const kafka = new Kafka({ clientId: 'payment-service', brokers: ['localhost:9092'] });
  const consumer = kafka.consumer({ groupId: 'payment-group' });
  await consumer.connect();
  await consumer.subscribe({ topic: 'order-events', fromBeginning: true });
  await consumer.run({
    eachMessage: async ({ message }) => {
      const event = JSON.parse(message.value.toString());
      console.log('Received order event:', event);
      // Process event (e.g., start payment)
    },
  });
}

produceOrderCreated();
consumeOrderEvents();

Output

Order created event sent
Received order event: { orderId: 'order1', status: 'created' }

Common Pitfalls

Common mistakes when using Kafka with microservices include:

• Not handling message duplication: Kafka may deliver messages more than once, so consumers must be idempotent.
• Ignoring consumer groups: Without proper group IDs, multiple consumers may process the same message unexpectedly.
• Not managing offsets: Failing to commit offsets can cause message loss or reprocessing.
• Overloading topics: Putting unrelated events in one topic reduces clarity and scalability.

javascript

/* Wrong: No consumer group, no offset commit */
consumer.subscribe(['order-events']);
consumer.run({ eachMessage: async ({ message }) => {
  console.log(message.value.toString());
} });

/* Right: Use consumer group and commit offsets automatically */
const consumer = kafka.consumer({ groupId: 'order-group' });
await consumer.subscribe({ topic: 'order-events' });
await consumer.run({ eachMessage: async ({ message }) => {
  console.log(message.value.toString());
} });

Quick Reference

Concept	Description
Producer	Sends messages (events) to Kafka topics.
Consumer	Reads messages from Kafka topics.
Topic	Named channel where messages are stored.
Partition	Sub-division of a topic for parallelism.
Consumer Group	Set of consumers sharing message load.
Offset	Position of a message in a partition.

Key Takeaways

Use Kafka producers and consumers to enable asynchronous event-driven communication between microservices.

Assign consumer groups to balance message processing and avoid duplicates.

Design microservices to handle possible message duplication and out-of-order delivery.

Separate events into different topics based on business domains for clarity and scalability.

Manage offsets properly to ensure reliable message processing and avoid data loss.