Kafka CLI tools overview - Commands & Configuration

Introduction

Kafka CLI tools help you manage and interact with Kafka servers and topics using simple commands. They solve the problem of controlling Kafka without needing a complex interface.

When you want to create or delete Kafka topics quickly from the command line.

When you need to send messages to a Kafka topic for testing or data input.

When you want to read messages from a Kafka topic to check data flow.

When you need to check the status or configuration of Kafka topics and brokers.

When you want to monitor Kafka consumer groups and their offsets.

Commands

This command creates a new Kafka topic named 'example-topic' with 3 partitions and a replication factor of 1 on the Kafka server running at localhost:9092.

Terminal

kafka-topics.sh --bootstrap-server localhost:9092 --create --topic example-topic --partitions 3 --replication-factor 1

Expected Output

Created topic example-topic.

→

--bootstrap-server - Specifies the Kafka server address to connect to

→

--create - Indicates that a new topic should be created

→

--topic - Names the topic to create

This command lists all the topics currently available on the Kafka server at localhost:9092.

Terminal

kafka-topics.sh --bootstrap-server localhost:9092 --list

Expected Output

example-topic

__consumer_offsets

→

--bootstrap-server - Specifies the Kafka server address to connect to

→

--list - Lists all topics

This command starts a producer that sends messages to the 'example-topic'. You can type messages and press Enter to send them.

Terminal

kafka-console-producer.sh --broker-list localhost:9092 --topic example-topic

Expected Output

No output (command runs silently)

→

--broker-list - Specifies the Kafka broker address to connect to

→

--topic - Specifies the topic to send messages to

This command reads messages from the 'example-topic' starting from the beginning and stops after reading 3 messages.

Terminal

kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic example-topic --from-beginning --max-messages 3

Expected Output

First message

Second message

Third message

→

--bootstrap-server - Specifies the Kafka server address to connect to

→

--topic - Specifies the topic to read messages from

→

--from-beginning - Reads messages from the start of the topic

→

--max-messages - Limits the number of messages to read

This command lists all consumer groups currently registered on the Kafka server.

Terminal

kafka-consumer-groups.sh --bootstrap-server localhost:9092 --list

Expected Output

example-group

console-consumer-12345

→

--bootstrap-server - Specifies the Kafka server address to connect to

→

--list - Lists all consumer groups

Key Concept

If you remember nothing else from Kafka CLI tools, remember: they let you create, send, read, and monitor Kafka topics and groups directly from the command line.

Common Mistakes

Using --zookeeper flag instead of --bootstrap-server with newer Kafka versions

The --zookeeper flag is deprecated and may not work with recent Kafka versions.

Always use --bootstrap-server to specify the Kafka server address.

Not specifying --from-beginning when consuming messages

Without --from-beginning, the consumer reads only new messages, missing existing ones.

Add --from-beginning to read all messages from the start.

Not creating a topic before producing messages

Producing to a non-existent topic will fail or create a topic with default settings, which may not be desired.

Create the topic explicitly with desired partitions and replication before producing.

Summary

Use kafka-topics.sh to create and list Kafka topics.

Use kafka-console-producer.sh to send messages to a topic.

Use kafka-console-consumer.sh to read messages from a topic.

Use kafka-consumer-groups.sh to list and monitor consumer groups.