Broker configuration basics in Kafka - Commands & Configuration

Introduction

Kafka brokers manage the storage and transmission of messages in a Kafka cluster. Configuring brokers correctly ensures reliable message delivery and efficient resource use.

When setting up a new Kafka broker to join a cluster

When adjusting broker settings to improve performance or reliability

When configuring log retention policies to control disk usage

When setting network ports and listeners for broker communication

When tuning broker parameters for specific workload requirements

Config File - server.properties

server.properties

broker.id=1
listeners=PLAINTEXT://:9092
log.dirs=/tmp/kafka-logs
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connect=localhost:2181

broker.id: Unique ID for this broker in the cluster.

listeners: Network address and port where the broker listens for client connections.

log.dirs: Directory where Kafka stores message logs.

num.network.threads and num.io.threads: Threads for handling network and disk I/O.

socket.*: Network buffer sizes for efficient data transfer.

log.retention.hours: How long to keep logs before deletion.

log.segment.bytes: Size of log segments before rolling over.

log.retention.check.interval.ms: How often Kafka checks for expired logs.

zookeeper.connect: Address of ZooKeeper managing the cluster metadata.

Commands

Starts the Kafka broker using the configuration in server.properties to join the cluster and begin handling messages.

Terminal

kafka-server-start.sh server.properties

Expected Output

[2024-06-01 12:00:00,000] INFO Kafka version : 3.5.0 (org.apache.kafka.common.utils.AppInfoParser)

[2024-06-01 12:00:00,001] INFO Kafka commitId : abcdef1234567890 (org.apache.kafka.common.utils.AppInfoParser)

[2024-06-01 12:00:00,500] INFO [KafkaServer id=1] started (kafka.server.KafkaServer)

Verifies that the Kafka broker is listening on port 9092 as configured.

Terminal

netstat -tuln | grep 9092

Expected Output

tcp        0      0 0.0.0.0:9092            0.0.0.0:*               LISTEN

Checks the log directory to confirm Kafka is storing message logs as per configuration.

Terminal

ls /tmp/kafka-logs

Expected Output

00000000000000000000.log

00000000000000001000.log

Key Concept

If you remember nothing else from this pattern, remember: the broker configuration file controls how the Kafka broker connects, stores data, and communicates, so setting it correctly is essential for a healthy Kafka cluster.

Common Mistakes

Using the same broker.id for multiple brokers

Kafka requires unique broker IDs; duplicates cause conflicts and cluster instability.

Assign a unique broker.id value to each broker in the cluster.

Not setting the listeners property or using a wrong port

The broker won't accept client connections if listeners are misconfigured, causing connection failures.

Set listeners to a valid network address and port that clients can reach, like PLAINTEXT://:9092.

Not configuring log.dirs or pointing it to a non-existent directory

Kafka cannot store messages without a valid log directory, leading to startup errors or data loss.

Set log.dirs to an existing directory with sufficient disk space.

Summary

Create a server.properties file to configure broker ID, network listeners, log storage, and other settings.

Start the Kafka broker using kafka-server-start.sh with the configuration file.

Verify the broker listens on the configured port and stores logs in the specified directory.

Practice

1. What is the main purpose of Kafka broker configuration files?

easy

A. To set key=value pairs that control broker behavior

B. To write Kafka client application code

C. To store user data permanently

D. To configure the operating system settings

Solution

1. Step 1: Understand the role of broker configuration

   Broker configuration files contain settings that control how Kafka brokers behave, such as storage and network handling.

2. Step 2: Identify the correct description

   Only To set key=value pairs that control broker behavior correctly describes the purpose as setting key=value pairs for broker behavior.

3. Final Answer:

   To set key=value pairs that control broker behavior -> Option A

4. Quick Check:

   Broker config = key=value pairs [OK]

Hint: Broker config files set key=value pairs for broker control [OK]

Common Mistakes:

• Confusing broker config with client code
• Thinking config stores user data
• Mixing OS settings with Kafka broker config

2. Which of the following is the correct syntax to set the broker ID in Kafka's configuration file?

easy

A. broker_id=1

B. broker-id:1

C. broker.id=1

D. brokerId=1

Solution

1. Step 1: Recall Kafka config syntax

   Kafka uses key=value pairs with dots to separate words, like broker.id=1.

2. Step 2: Compare options

   Only broker.id=1 uses the correct dot notation and equal sign syntax.

3. Final Answer:

   broker.id=1 -> Option C

4. Quick Check:

   Correct syntax uses dot and equal sign [OK]

Hint: Kafka configs use dot notation and equal sign [OK]

Common Mistakes:

• Using underscores instead of dots
• Using colon instead of equal sign
• CamelCase keys instead of lowercase with dots

3. Given this Kafka broker configuration snippet:

log.dirs=/tmp/kafka-logs
num.network.threads=3
num.io.threads=8

What does num.network.threads=3 control?

medium

A. Number of threads for message processing

B. Number of threads writing logs to disk

C. Number of threads for client connections

D. Number of threads handling network requests

Solution

1. Step 1: Understand the meaning of num.network.threads

   This setting controls how many threads the broker uses to handle network requests from clients.

2. Step 2: Match options with setting

   Number of threads handling network requests correctly states it controls threads handling network requests; others describe unrelated functions.

3. Final Answer:

   Number of threads handling network requests -> Option D

4. Quick Check:

   num.network.threads = network threads count [OK]

Hint: network.threads means threads for network requests [OK]

Common Mistakes:

• Confusing network threads with IO threads
• Thinking it controls log writing threads
• Mixing client connections with message processing

4. You see this broker config line:

log.retention.hours 168

What is wrong with this line?

medium

A. Missing equal sign between key and value

B. Invalid value; hours must be a string

C. Key name is incorrect; should be log.retention.ms

D. No error; this line is correct

Solution

1. Step 1: Check syntax for key-value pairs

   Kafka config requires an equal sign (=) between key and value, not just a space.

2. Step 2: Identify the error in the line

   The line uses a space instead of '=', so it is a syntax error.

3. Final Answer:

   Missing equal sign between key and value -> Option A

4. Quick Check:

   Config lines need key=value format [OK]

Hint: Config lines must have key=value, not space [OK]

Common Mistakes:

• Assuming space works as separator
• Confusing retention hours with milliseconds
• Thinking the key name is wrong

5. You want to configure a Kafka broker to store logs in two directories: /data/kafka1 and /data/kafka2. Which is the correct way to set log.dirs in the broker config?

hard

A. log.dirs=/data/kafka1;/data/kafka2

B. log.dirs=/data/kafka1,/data/kafka2

C. log.dirs=/data/kafka1 /data/kafka2

D. log.dirs=[/data/kafka1,/data/kafka2]

Solution

1. Step 1: Understand log.dirs format

   Kafka expects multiple directories separated by commas without spaces or brackets.

2. Step 2: Evaluate each option

   log.dirs=/data/kafka1,/data/kafka2 uses commas correctly; B uses semicolon (wrong), C uses space (wrong), D uses brackets (wrong).

3. Final Answer:

   log.dirs=/data/kafka1,/data/kafka2 -> Option B

4. Quick Check:

   Multiple dirs separated by commas [OK]

Hint: Separate multiple log dirs with commas, no spaces [OK]

Common Mistakes:

• Using semicolons or spaces instead of commas
• Adding brackets around paths
• Including spaces between paths