Message broker architecture in Kafka - Commands & Configuration

Introduction

A message broker helps different parts of a system talk to each other by sending messages. It makes sure messages get delivered safely and in order, even if parts of the system are busy or offline.

When you want to connect different apps or services that run separately and need to share data.

When you want to handle lots of messages quickly without losing any.

When you want to keep messages safe even if one part of your system crashes.

When you want to process messages in order or group them by topic.

When you want to scale your system easily by adding more message handlers.

Config File - server.properties

server.properties

broker.id=1
listeners=PLAINTEXT://:9092
log.dirs=/tmp/kafka-logs
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connect=localhost:2181
zookeeper.connection.timeout.ms=6000

broker.id: Unique ID for this Kafka broker.

listeners: Network address where Kafka listens for connections.

log.dirs: Folder where Kafka stores message logs.

num.network.threads and num.io.threads: Threads for handling network and disk I/O.

log.retention.hours: How long Kafka keeps messages before deleting.

zookeeper.connect: Address of ZooKeeper which manages Kafka cluster metadata.

Commands

Start ZooKeeper, which Kafka uses to keep track of brokers and topics.

Terminal

bin/zookeeper-server-start.sh config/zookeeper.properties

Expected Output

[2024-06-01 12:00:00,000] INFO Starting zookeeper server ...

[2024-06-01 12:00:01,000] INFO Binding to port 2181

[2024-06-01 12:00:02,000] INFO Started serving clients

Start the Kafka broker using the configuration file to accept and store messages.

Terminal

bin/kafka-server-start.sh config/server.properties

Expected Output

[2024-06-01 12:00:05,000] INFO Kafka server started (kafka.server.KafkaServer)

[2024-06-01 12:00:06,000] INFO Awaiting client connections on port 9092

Create a topic named 'example-topic' with 3 partitions to organize messages and allow parallel processing.

Terminal

bin/kafka-topics.sh --create --topic example-topic --bootstrap-server localhost:9092 --partitions 3 --replication-factor 1

Expected Output

Created topic example-topic.

→

--partitions - Number of partitions to split the topic into for parallelism

→

--replication-factor - Number of copies of data for fault tolerance

List all topics available on the Kafka broker to verify the topic creation.

Terminal

bin/kafka-topics.sh --list --bootstrap-server localhost:9092

Expected Output

example-topic

Key Concept

If you remember nothing else from this pattern, remember: a message broker safely stores and forwards messages between parts of a system, organizing them by topics and partitions for reliability and speed.

Common Mistakes

Starting Kafka broker before starting ZooKeeper

Kafka depends on ZooKeeper to manage cluster metadata, so it won't start properly without ZooKeeper running.

Always start ZooKeeper first, then start the Kafka broker.

Creating a topic without specifying partitions

Without partitions, the topic cannot handle parallel processing or scale well.

Specify the number of partitions when creating a topic to allow better performance.

Using replication-factor higher than the number of brokers

Kafka cannot create more copies than brokers available, causing topic creation to fail.

Set replication-factor equal or less than the number of running brokers.

Summary

Start ZooKeeper first to manage Kafka cluster metadata.

Start Kafka broker with a proper configuration file to accept messages.

Create topics with partitions and replication for organizing and protecting messages.

List topics to verify they exist and are ready for use.