Kafka CLI tools overview - Mini Project: Build & Apply

Kafka CLI tools overview

📖 Scenario: You are working with Apache Kafka, a system that helps send messages between different parts of a program. To manage Kafka, you use command-line tools that let you create topics, send messages, and check the system's status.

🎯 Goal: Learn how to use basic Kafka command-line tools to create a topic, list topics, send messages, and read messages from a topic.

📋 What You'll Learn

Use the kafka-topics.sh tool to create and list topics

Use the kafka-console-producer.sh tool to send messages

Use the kafka-console-consumer.sh tool to read messages

Use exact commands and options as specified

💡 Why This Matters

🌍 Real World

Kafka CLI tools are used by developers and system administrators to manage Kafka clusters, topics, and messages directly from the command line.

💼 Career

Knowing Kafka CLI commands is essential for roles involving data streaming, real-time analytics, and distributed system management.

1

Create a Kafka topic

Use the kafka-topics.sh command with --create to create a topic named test-topic with 1 partition and replication factor 1. Include --bootstrap-server localhost:9092 in the command.

Kafka

# Type the command to create the topic test-topic
# Your code here

Hint

Remember to specify the topic name exactly as test-topic and use --bootstrap-server localhost:9092 to connect.

2

List Kafka topics

Use the kafka-topics.sh command with --list and --bootstrap-server localhost:9092 to list all topics.

Kafka

kafka-topics.sh --create --topic test-topic --partitions 1 --replication-factor 1 --bootstrap-server localhost:9092
# Type the command to list all topics
# Your code here

Hint

Use --list to see all topics available on the server.

3

Send messages to the topic

Use the kafka-console-producer.sh command with --topic test-topic and --bootstrap-server localhost:9092 to send messages. Type the message Hello Kafka and then stop the producer.

Kafka

kafka-topics.sh --create --topic test-topic --partitions 1 --replication-factor 1 --bootstrap-server localhost:9092
kafka-topics.sh --list --bootstrap-server localhost:9092
# Type the command to start the producer for test-topic
# Then type 'Hello Kafka' as the message
# Your code here

Hint

Run the producer command, then type the message Hello Kafka and press Enter.

4

Read messages from the topic

Use the kafka-console-consumer.sh command with --topic test-topic, --from-beginning, and --bootstrap-server localhost:9092 to read all messages from the start. The output should show Hello Kafka.

Kafka

kafka-topics.sh --create --topic test-topic --partitions 1 --replication-factor 1 --bootstrap-server localhost:9092
kafka-topics.sh --list --bootstrap-server localhost:9092
kafka-console-producer.sh --topic test-topic --bootstrap-server localhost:9092
Hello Kafka
# Type the command to consume messages from test-topic from the beginning
# Your code here

Hint

Use --from-beginning to read all messages from the start of the topic.

Practice

1. Which Kafka CLI tool is used to create a new topic?

easy

A. kafka-topics.sh --create

B. kafka-console-producer.sh

C. kafka-console-consumer.sh

D. kafka-configs.sh

Solution

1. Step 1: Identify the tool for topic management

   The kafka-topics.sh tool is used to manage topics including creation, deletion, and listing.

2. Step 2: Confirm the create option usage

   Using --create with kafka-topics.sh creates a new topic.

3. Final Answer:

   kafka-topics.sh --create -> Option A

4. Quick Check:

   Topic creation = kafka-topics.sh --create [OK]

Hint: Topic creation always uses kafka-topics.sh with --create [OK]

Common Mistakes:

• Confusing producer or consumer tools for topic creation
• Using kafka-configs.sh which manages configs, not topics
• Omitting --create option

2. Which of the following is the correct syntax to start a Kafka console producer connecting to a server at localhost:9092?

easy

A. kafka-console-producer.sh --connect localhost:9092 --topic mytopic

B. kafka-console-producer.sh --broker-list localhost:9092 --topic mytopic

C. kafka-console-producer.sh --server localhost:9092 --topic mytopic

D. kafka-console-producer.sh --bootstrap-server localhost:9092 --topic mytopic

Solution

1. Step 1: Identify the correct option for specifying Kafka server

   Modern Kafka CLI tools use --bootstrap-server to specify the Kafka server address.

2. Step 2: Check the full command syntax

   The correct syntax includes kafka-console-producer.sh --bootstrap-server localhost:9092 --topic mytopic.

3. Final Answer:

   kafka-console-producer.sh --bootstrap-server localhost:9092 --topic mytopic -> Option D

4. Quick Check:

   Use --bootstrap-server for server address [OK]

Hint: Always use --bootstrap-server to specify Kafka server [OK]

Common Mistakes:

• Using deprecated --broker-list instead of --bootstrap-server
• Using incorrect flags like --server or --connect
• Omitting the --topic option

3. What will be the output of this command if the topic test-topic has 3 messages: kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test-topic --from-beginning --max-messages 2?

medium

A. It will print all 3 messages from test-topic and then exit.

B. It will print the first 2 messages from test-topic and then exit.

C. It will print no messages because --max-messages is ignored.

D. It will print only the last message from test-topic.

Solution

1. Step 1: Understand the --max-messages option

   The --max-messages 2 option limits the consumer to read only 2 messages before exiting.

2. Step 2: Analyze the command behavior

   The command reads from the beginning and stops after printing 2 messages, even though the topic has 3 messages.

3. Final Answer:

   It will print the first 2 messages from test-topic and then exit. -> Option B

4. Quick Check:

   --max-messages limits output count [OK]

Hint: --max-messages limits how many messages are consumed [OK]

Common Mistakes:

• Assuming all messages will be printed ignoring --max-messages
• Thinking --max-messages prints last messages only
• Confusing --from-beginning with consuming only latest messages

4. You run this command but get an error: kafka-console-producer.sh --topic mytopic. What is missing?

medium

A. The --bootstrap-server option specifying the Kafka server address.

B. The --create option to create the topic.

C. The --from-beginning option to read messages.

D. The --max-messages option to limit output.

Solution

1. Step 1: Identify required options for kafka-console-producer.sh

   The producer needs to know which Kafka server to connect to, specified by --bootstrap-server.

2. Step 2: Check the given command

   The command lacks --bootstrap-server, so it cannot connect and throws an error.

3. Final Answer:

   The --bootstrap-server option specifying the Kafka server address. -> Option A

4. Quick Check:

   Producer needs --bootstrap-server to connect [OK]

Hint: Always include --bootstrap-server to avoid connection errors [OK]

Common Mistakes:

• Assuming topic creation is automatic without --create
• Confusing producer options with consumer options
• Adding irrelevant options like --from-beginning for producer

5. You want to list all topics on a Kafka server at localhost:9092. Which command correctly lists the topics?

hard

A. kafka-console-producer.sh --list --bootstrap-server localhost:9092

B. kafka-console-consumer.sh --list --bootstrap-server localhost:9092

C. kafka-topics.sh --list --bootstrap-server localhost:9092

D. kafka-configs.sh --list --bootstrap-server localhost:9092

Solution

1. Step 1: Identify the tool for listing topics

   The kafka-topics.sh tool manages topics and supports the --list option to show all topics.

2. Step 2: Confirm the correct command syntax

   Using --bootstrap-server localhost:9092 specifies the Kafka server to connect to.

3. Final Answer:

   kafka-topics.sh --list --bootstrap-server localhost:9092 -> Option C

4. Quick Check:

   List topics with kafka-topics.sh --list [OK]

Hint: Use kafka-topics.sh with --list to see all topics [OK]

Common Mistakes:

• Using consumer or producer tools to list topics
• Omitting --bootstrap-server causing connection failure
• Using kafka-configs.sh which lists configs, not topics