Broker configuration basics in Kafka - Commands & Configuration

Introduction

Kafka brokers manage the storage and transmission of messages in a Kafka cluster. Configuring brokers correctly ensures reliable message delivery and efficient resource use.

When setting up a new Kafka broker to join a cluster

When adjusting broker settings to improve performance or reliability

When configuring log retention policies to control disk usage

When setting network ports and listeners for broker communication

When tuning broker parameters for specific workload requirements

Config File - server.properties

server.properties

broker.id=1
listeners=PLAINTEXT://:9092
log.dirs=/tmp/kafka-logs
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connect=localhost:2181

broker.id: Unique ID for this broker in the cluster.

listeners: Network address and port where the broker listens for client connections.

log.dirs: Directory where Kafka stores message logs.

num.network.threads and num.io.threads: Threads for handling network and disk I/O.

socket.*: Network buffer sizes for efficient data transfer.

log.retention.hours: How long to keep logs before deletion.

log.segment.bytes: Size of log segments before rolling over.

log.retention.check.interval.ms: How often Kafka checks for expired logs.

zookeeper.connect: Address of ZooKeeper managing the cluster metadata.

Commands

Starts the Kafka broker using the configuration in server.properties to join the cluster and begin handling messages.

Terminal

kafka-server-start.sh server.properties

Expected Output

[2024-06-01 12:00:00,000] INFO Kafka version : 3.5.0 (org.apache.kafka.common.utils.AppInfoParser)

[2024-06-01 12:00:00,001] INFO Kafka commitId : abcdef1234567890 (org.apache.kafka.common.utils.AppInfoParser)

[2024-06-01 12:00:00,500] INFO [KafkaServer id=1] started (kafka.server.KafkaServer)

Verifies that the Kafka broker is listening on port 9092 as configured.

Terminal

netstat -tuln | grep 9092

Expected Output

tcp        0      0 0.0.0.0:9092            0.0.0.0:*               LISTEN

Checks the log directory to confirm Kafka is storing message logs as per configuration.

Terminal

ls /tmp/kafka-logs

Expected Output

00000000000000000000.log

00000000000000001000.log

Key Concept

If you remember nothing else from this pattern, remember: the broker configuration file controls how the Kafka broker connects, stores data, and communicates, so setting it correctly is essential for a healthy Kafka cluster.

Common Mistakes

Using the same broker.id for multiple brokers

Kafka requires unique broker IDs; duplicates cause conflicts and cluster instability.

Assign a unique broker.id value to each broker in the cluster.

Not setting the listeners property or using a wrong port

The broker won't accept client connections if listeners are misconfigured, causing connection failures.

Set listeners to a valid network address and port that clients can reach, like PLAINTEXT://:9092.

Not configuring log.dirs or pointing it to a non-existent directory

Kafka cannot store messages without a valid log directory, leading to startup errors or data loss.

Set log.dirs to an existing directory with sufficient disk space.

Summary

Create a server.properties file to configure broker ID, network listeners, log storage, and other settings.

Start the Kafka broker using kafka-server-start.sh with the configuration file.

Verify the broker listens on the configured port and stores logs in the specified directory.