Broker nodes in Kafka - Commands & Configuration

Introduction

Kafka broker nodes are servers that store and manage messages in Kafka. They help distribute data and handle requests from producers and consumers, making sure messages are saved and delivered reliably.

When you want to run a Kafka server that stores and manages message data.

When you need to add more capacity to handle more messages or users by adding more brokers.

When you want to ensure your messaging system keeps working even if one server fails.

When you want to balance the load of message traffic across multiple servers.

When you want to configure Kafka to listen on specific network addresses and ports.

Config File - server.properties

server.properties

broker.id=1
listeners=PLAINTEXT://0.0.0.0:9092
log.dirs=/tmp/kafka-logs
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connect=localhost:2181

broker.id: Unique ID for this broker node.

listeners: Network address and port where Kafka listens for connections.

log.dirs: Directory where Kafka stores message logs.

num.network.threads and num.io.threads: Threads for handling network and disk I/O.

socket.*: Network buffer sizes and max request size.

log.retention.*: How long and how big logs are kept.

zookeeper.connect: Address of ZooKeeper managing Kafka cluster metadata.

Commands

Starts the Kafka broker node using the configuration in server.properties. This launches the server to accept connections and manage messages.

Terminal

kafka-server-start.sh server.properties

Expected Output

[2024-06-01 12:00:00,000] INFO Kafka version : 3.5.0 (org.apache.kafka.common.utils.AppInfoParser)

[2024-06-01 12:00:00,001] INFO Kafka startTimeMs : 1685611200000 (org.apache.kafka.common.utils.AppInfoParser)

[2024-06-01 12:00:01,500] INFO [KafkaServer id=1] started (kafka.server.KafkaServer)

Lists all topics managed by the Kafka broker to verify the broker is running and accessible.

Terminal

kafka-topics.sh --bootstrap-server localhost:9092 --list

Expected Output

example-topic

my-topic

→

--bootstrap-server - Specifies the Kafka broker address to connect to.

Shows the API versions supported by the broker, confirming the broker is active and communicating.

Terminal

kafka-broker-api-versions.sh --bootstrap-server localhost:9092

Expected Output

Broker: 1 (id: 1 rack: null)

  ApiVersions:

    Produce (0-9)

    Fetch (0-11)

    ListOffsets (0-4)

    Metadata (0-8)

    ...

→

--bootstrap-server - Specifies the Kafka broker address to connect to.

Key Concept

If you remember nothing else from this pattern, remember: each Kafka broker node stores and manages messages and must have a unique broker.id and proper network settings to work correctly.

Common Mistakes

Using the same broker.id on multiple Kafka broker nodes.

Kafka requires unique broker IDs to identify each node; duplicates cause conflicts and cluster errors.

Assign a unique broker.id number to each Kafka broker node in its server.properties file.

Not setting the listeners property correctly, causing the broker to be unreachable.

If listeners is not set or set incorrectly, clients cannot connect to the broker.

Set listeners to the correct IP address and port, e.g., PLAINTEXT://0.0.0.0:9092 to listen on all interfaces.

Summary

Create a server.properties file to configure each Kafka broker node with unique ID and network settings.

Start the Kafka broker using kafka-server-start.sh with the configuration file.

Verify the broker is running by listing topics and checking API versions.

Practice

1.

What is the primary role of a broker node in Kafka?

easy

A. Store and manage Kafka messages

B. Produce messages to topics

C. Consume messages from topics

D. Monitor network traffic

Solution

1. Step 1: Understand broker node function

   A broker node is a Kafka server that stores and manages messages.

2. Step 2: Differentiate roles

   Producers send messages, consumers read them, but brokers store/manage them.

3. Final Answer:

   Store and manage Kafka messages -> Option A

4. Quick Check:

   Broker = Store/manage messages [OK]

Hint: Broker nodes store messages; producers and consumers do not [OK]

Common Mistakes:

• Confusing brokers with producers or consumers
• Thinking brokers monitor network traffic
• Assuming brokers only route messages

2.

Which of the following is the correct way to specify a broker's unique ID in the Kafka server.properties file?

easy

A. broker.id=1

B. brokerID=1

C. broker-id=1

D. brokerId=1

Solution

1. Step 1: Recall Kafka broker ID syntax

   The correct property name is broker.id with a dot separator.

2. Step 2: Check options for exact match

   Only broker.id=1 matches Kafka's expected syntax.

3. Final Answer:

   broker.id=1 -> Option A

4. Quick Check:

   Broker ID property uses dot notation [OK]

Hint: Broker ID uses dot notation: broker.id [OK]

Common Mistakes:

• Using camelCase or hyphens instead of dot
• Misspelling the property name
• Confusing broker.id with other properties

3.

Given the following snippet from a Kafka server.properties file:

broker.id=2
listeners=PLAINTEXT://localhost:9093
log.dirs=/tmp/kafka-logs-2

What will be the broker's ID and listening port when Kafka starts?

medium

A. Broker ID 1, listening on port 9092

B. Broker ID 2, listening on port 9093

C. Broker ID 2, listening on port 9092

D. Broker ID 1, listening on port 9093

Solution

1. Step 1: Identify broker ID from properties

   The property broker.id=2 sets the broker ID to 2.

2. Step 2: Identify listener port

   listeners=PLAINTEXT://localhost:9093 means the broker listens on port 9093.

3. Final Answer:

   Broker ID 2, listening on port 9093 -> Option B

4. Quick Check:

   broker.id=2 and port=9093 [OK]

Hint: Broker ID and port come directly from properties file [OK]

Common Mistakes:

• Assuming default port 9092 without checking listeners
• Mixing up broker IDs
• Ignoring the listeners property

4.

Consider this server.properties snippet:

broker.id=abc
listeners=PLAINTEXT://localhost:9092

What is the issue when starting Kafka with this configuration?

medium

A. broker.id can be a string, so no issue

B. listeners property is missing a port

C. Kafka will ignore broker.id and assign default

D. broker.id must be an integer, so 'abc' causes an error

Solution

1. Step 1: Check broker.id value type

   Kafka requires broker.id to be an integer number.

2. Step 2: Identify invalid value

   Setting broker.id=abc is invalid and causes startup error.

3. Final Answer:

   broker.id must be an integer, so 'abc' causes an error -> Option D

4. Quick Check:

   broker.id integer required [OK]

Hint: broker.id must be a number, not letters [OK]

Common Mistakes:

• Assuming broker.id can be string
• Ignoring error messages on startup
• Thinking Kafka assigns default broker.id automatically

5.

You want to run three Kafka broker nodes on the same machine for testing. Which configuration change is essential for each broker to avoid conflicts?

hard

A. Use the same broker.id but different log.dirs

B. Keep listeners port same but change broker.id

C. Assign unique broker.id and different listeners ports

D. Only change log.dirs for each broker

Solution

1. Step 1: Understand broker uniqueness requirements

   Each broker must have a unique broker.id to identify it in the cluster.

2. Step 2: Avoid port conflicts

   Each broker must listen on a different port to avoid network conflicts.

3. Final Answer:

   Assign unique broker.id and different listeners ports -> Option C

4. Quick Check:

   Unique IDs and ports prevent conflicts [OK]

Hint: Unique broker.id and ports prevent conflicts [OK]

Common Mistakes:

• Using same broker.id for multiple brokers
• Using same port for multiple brokers
• Only changing log directories without ports or IDs