Topic creation in Kafka - Commands & Configuration

Introduction

Kafka topics are like folders where messages are stored. Creating a topic lets you organize and separate different streams of data so your apps can read and write messages clearly.

When you want to separate logs from different services to keep data organized.

When you need to create a new channel for a new feature in your app to send messages.

When you want to control how many copies of data are kept for safety.

When you want to set how many parts a topic has to allow multiple consumers to read in parallel.

When you want to test sending and receiving messages on a fresh topic.

Commands

This command creates a new Kafka topic named 'example-topic' with 3 partitions and a replication factor of 1, connecting to the Kafka server at localhost on port 9092.

Terminal

kafka-topics.sh --create --topic example-topic --bootstrap-server localhost:9092 --partitions 3 --replication-factor 1

Expected Output

Created topic example-topic.

→

--topic - Name of the topic to create

→

--partitions - Number of partitions for parallelism

→

--replication-factor - Number of copies of data for fault tolerance

This command lists all topics currently available on the Kafka server to verify that 'example-topic' was created.

Terminal

kafka-topics.sh --list --bootstrap-server localhost:9092

Expected Output

example-topic

__consumer_offsets

→

--list - List all topics

→

--bootstrap-server - Kafka server address

This command shows detailed information about the 'example-topic', including partition count and replication details.

Terminal

kafka-topics.sh --describe --topic example-topic --bootstrap-server localhost:9092

Expected Output

Topic: example-topic	PartitionCount: 3	ReplicationFactor: 1	Configs: 

	Topic: example-topic	Partition: 0	Leader: 0	Replicas: 0	Isr: 0

	Topic: example-topic	Partition: 1	Leader: 0	Replicas: 0	Isr: 0

	Topic: example-topic	Partition: 2	Leader: 0	Replicas: 0	Isr: 0

→

--describe - Show topic details

→

--topic - Specify the topic to describe

Key Concept

If you remember nothing else from this pattern, remember: creating a Kafka topic sets up a named place to store and organize messages with partitions and replication for performance and safety.

Common Mistakes

Not specifying the replication factor when creating a topic

Kafka requires a replication factor to know how many copies of data to keep; missing it causes an error or defaults that may not suit your needs.

Always include --replication-factor with a valid number when creating a topic.

Using the wrong Kafka server address in --bootstrap-server

If the address is wrong, the command cannot connect to Kafka and fails to create or list topics.

Use the correct hostname and port where your Kafka broker is running, like localhost:9092.

Trying to create a topic that already exists without checking

Kafka will return an error saying the topic exists, causing confusion or script failure.

List existing topics first with --list to avoid duplicates before creating a new topic.

Summary

Use kafka-topics.sh with --create to make a new topic with partitions and replication.

Verify topic creation by listing all topics with --list.

Check topic details like partitions and replicas with --describe.

Practice

1. What is the main purpose of a topic in Kafka?

easy

A. To compile Java programs

B. To store user login credentials

C. To manage database connections

D. To organize messages like folders organize files

Solution

1. Step 1: Understand Kafka topic role

   A Kafka topic is a category or feed name to which messages are published.

2. Step 2: Compare options with topic function

   Only To organize messages like folders organize files correctly describes topics as organizing messages like folders organize files.

3. Final Answer:

   To organize messages like folders organize files -> Option D

4. Quick Check:

   Topic = Organize messages [OK]

Hint: Topics group messages like folders group files [OK]

Common Mistakes:

• Confusing topics with databases
• Thinking topics store user data
• Mixing topics with program compilation

2. Which command correctly creates a Kafka topic named orders with 3 partitions and replication factor 2?

easy

A. kafka-topics.sh --create --topic orders --partitions 3 --replication-factor 2 --bootstrap-server localhost:9092

B. kafka-create-topic orders 3 2

C. create-topic --name orders --partitions 3 --replicas 2

D. kafka-topics.sh --new --topic orders --partitions=3 --replicas=2

Solution

1. Step 1: Identify correct Kafka topic creation syntax

   The official command uses kafka-topics.sh with --create, --topic, --partitions, --replication-factor, and --bootstrap-server options.

2. Step 2: Match options with correct syntax

   kafka-topics.sh --create --topic orders --partitions 3 --replication-factor 2 --bootstrap-server localhost:9092 matches the correct syntax exactly; others use wrong commands or options.

3. Final Answer:

   kafka-topics.sh --create --topic orders --partitions 3 --replication-factor 2 --bootstrap-server localhost:9092 -> Option A

4. Quick Check:

   Correct syntax = kafka-topics.sh --create --topic orders --partitions 3 --replication-factor 2 --bootstrap-server localhost:9092 [OK]

Hint: Use kafka-topics.sh with --create and proper flags [OK]

Common Mistakes:

• Using wrong command names
• Missing --bootstrap-server option
• Incorrect option names like --replicas

3. What will happen if you run this command?

kafka-topics.sh --create --topic sales --partitions 5 --replication-factor 1 --bootstrap-server localhost:9092

medium

A. Creates a topic named 'sales' with 1 partition and replication factor 5

B. Fails because replication factor cannot be 1

C. Creates a topic named 'sales' with 5 partitions and replication factor 1

D. Fails due to missing zookeeper connection

Solution

1. Step 1: Analyze command parameters

   The command specifies topic 'sales', 5 partitions, replication factor 1, and bootstrap server.

2. Step 2: Understand Kafka topic creation rules

   Replication factor 1 is valid (means no replicas), and partitions set to 5 means 5 partitions created.

3. Final Answer:

   Creates a topic named 'sales' with 5 partitions and replication factor 1 -> Option C

4. Quick Check:

   Partitions=5, Replication=1 [OK]

Hint: Replication factor 1 means no replicas, partitions as given [OK]

Common Mistakes:

• Confusing partitions with replication factor
• Assuming replication factor 1 is invalid
• Expecting zookeeper connection with bootstrap-server

4. You tried to create a topic with this command:

kafka-topics.sh --create --topic logs --partitions two --replication-factor 3 --bootstrap-server localhost:9092

What is the error?

medium

A. Missing --create flag

B. Invalid number for partitions: 'two' should be a number

C. Replication factor cannot be 3

D. Bootstrap server address is incorrect

Solution

1. Step 1: Check partitions parameter

   The partitions option expects a number, but 'two' is a word, causing a syntax error.

2. Step 2: Verify other parameters

   Replication factor 3 and bootstrap server are valid; --create flag is present.

3. Final Answer:

   Invalid number for partitions: 'two' should be a number -> Option B

4. Quick Check:

   Partitions must be numeric [OK]

Hint: Partitions must be numeric, not words [OK]

Common Mistakes:

• Using words instead of numbers for partitions
• Assuming replication factor 3 is invalid
• Forgetting --create flag

5. You want to create a Kafka topic for a critical system that must not lose messages and should handle high load. Which topic creation settings are best?

hard

A. Partitions: 10, Replication factor: 3

B. Partitions: 1, Replication factor: 1

C. Partitions: 5, Replication factor: 1

D. Partitions: 3, Replication factor: 2

Solution

1. Step 1: Understand requirements for critical system

   High load needs many partitions for parallelism; no message loss needs high replication factor.

2. Step 2: Evaluate options

   Partitions: 10, Replication factor: 3 has 10 partitions (high load) and replication factor 3 (high safety). Others have fewer partitions or lower replication.

3. Final Answer:

   Partitions: 10, Replication factor: 3 -> Option A

4. Quick Check:

   High load + safety = many partitions + high replication [OK]

Hint: More partitions + higher replication = speed + safety [OK]

Common Mistakes:

• Choosing low replication for safety
• Choosing few partitions for high load
• Ignoring replication factor importance