Configuration best practices in Kafka - Commands & Configuration

Introduction

Kafka needs careful setup to run smoothly and handle data well. Good configuration helps avoid crashes, data loss, and slow performance.

When setting up a new Kafka broker to handle message streaming reliably.

When tuning Kafka to handle more users or higher message volumes without delays.

When securing Kafka so only allowed users can send or read messages.

When you want to keep Kafka logs and data safe even if a server fails.

When you need to balance performance and resource use on your Kafka servers.

Config File - server.properties

server.properties

broker.id=1
listeners=PLAINTEXT://:9092
log.dirs=/var/lib/kafka-logs
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connect=localhost:2181
num.partitions=3
auto.create.topics.enable=false
log.cleaner.enable=true
log.cleaner.threads=2
log.cleaner.dedupe.buffer.size=134217728
log.cleaner.io.max.bytes.per.second=1048576
log.cleaner.backoff.ms=15000

broker.id: Unique ID for each Kafka broker.

listeners: Network address Kafka listens on.

log.dirs: Where Kafka stores data logs.

num.network.threads and num.io.threads: Threads for handling network and disk I/O.

socket.*: Buffer sizes for network communication.

log.retention.hours: How long logs are kept before deletion.

log.segment.bytes: Size of each log file segment.

zookeeper.connect: Address of ZooKeeper managing Kafka cluster.

num.partitions: Default number of partitions per topic.

auto.create.topics.enable: Prevents automatic topic creation to avoid mistakes.

log.cleaner.*: Settings for log compaction to save space and keep latest data.

Commands

Starts the Kafka broker using the configuration file to apply all settings.

Terminal

kafka-server-start.sh /opt/kafka/config/server.properties

Expected Output

[2024-06-01 12:00:00,000] INFO Kafka version : 3.5.0 (org.apache.kafka.common.utils.AppInfoParser)

[2024-06-01 12:00:00,001] INFO Kafka startTimeMs : 1685611200000 (org.apache.kafka.common.utils.AppInfoParser)

[2024-06-01 12:00:01,500] INFO [KafkaServer id=1] started (kafka.server.KafkaServer)

Lists all topics on the Kafka broker to verify it is running and topics are managed correctly.

Terminal

kafka-topics.sh --bootstrap-server localhost:9092 --list

Expected Output

example-topic

user-events

system-logs

→

--bootstrap-server - Specifies the Kafka server address to connect to.

Shows the current configuration of the Kafka broker with ID 1 to check applied settings.

Terminal

kafka-configs.sh --bootstrap-server localhost:9092 --entity-type brokers --entity-name 1 --describe

Expected Output

Configs for broker 1 are:

  log.retention.hours=168

  num.partitions=3

  auto.create.topics.enable=false

  log.cleaner.enable=true

→

--entity-type - Defines the type of entity to describe, here a broker.

→

--entity-name - Specifies the broker ID to show configuration for.

Key Concept

If you remember nothing else from this pattern, remember: carefully setting Kafka's configuration file ensures reliable, secure, and efficient message streaming.

Common Mistakes

Leaving auto.create.topics.enable set to true in production.

It can create unwanted topics automatically, causing confusion and resource waste.

Set auto.create.topics.enable=false to control topic creation explicitly.

Not setting unique broker.id for each Kafka broker.

Kafka brokers must have unique IDs or they will conflict and fail to start properly.

Assign a unique broker.id number to each broker in the cluster.

Using default log retention without adjusting for data needs.

Logs may fill disk space too quickly or be deleted too soon, risking data loss or downtime.

Adjust log.retention.hours and log.segment.bytes based on your data volume and storage capacity.

Summary

Create and edit the server.properties file to set Kafka broker settings.

Start the Kafka broker with kafka-server-start.sh using the config file.

Verify topics and broker configuration with kafka-topics.sh and kafka-configs.sh commands.