Topic creation in Kafka - Step-by-Step Execution

Process Flow - Topic creation

Start

↓

Define Topic Name

↓

Set Configurations

↓

Send Create Topic Request

↓

Kafka Server Validates

↓

If Valid

→Topic Created

↓

If Invalid

→Error Returned

↓

End

This flow shows how a topic is created by defining its name and settings, sending a request to Kafka, which then validates and either creates the topic or returns an error.

Execution Sample

Kafka

from kafka.admin import KafkaAdminClient, NewTopic

admin_client = KafkaAdminClient(bootstrap_servers='localhost:9092')
new_topic = NewTopic(name='my_topic', num_partitions=3, replication_factor=1)
admin_client.create_topics(new_topics=[new_topic])

This code creates a Kafka topic named 'my_topic' with 3 partitions and a replication factor of 1.

Process Table

Step	Action	Input/Condition	Result
1	Initialize KafkaAdminClient	bootstrap_servers='localhost:9092'	Client ready to send requests
2	Define NewTopic	name='my_topic', num_partitions=3, replication_factor=1	Topic object created
3	Call create_topics	new_topics=[new_topic]	Request sent to Kafka server
4	Kafka server validates	Topic name valid, partitions and replication valid	Topic created successfully
5	Return from create_topics	No errors	Topic creation confirmed

💡 Topic created successfully after validation by Kafka server

Status Tracker

Variable	Start	After Step 2	After Step 3	Final
admin_client	None	KafkaAdminClient instance	KafkaAdminClient instance	KafkaAdminClient instance
new_topic	None	NewTopic(name='my_topic', num_partitions=3, replication_factor=1)	NewTopic object passed to create_topics	NewTopic object created on server

Key Moments - 2 Insights

Why do we need to specify the number of partitions and replication factor when creating a topic?

Partitions determine how data is split and parallelized; replication factor controls fault tolerance. This is shown in execution_table step 2 where these values are set before creation.

What happens if the topic name is invalid or already exists?

Kafka server validates the request (step 4). If invalid or duplicate, an error is returned instead of creating the topic.

Visual Quiz - 3 Questions

Look at the execution_table, what is the result after step 3?

A Request sent to Kafka server

B Topic created successfully

C Error returned

D Client initialization

💡 Hint

Check the 'Result' column for step 3 in execution_table

At which step does Kafka validate the topic creation request?

A Step 2

B Step 3

C Step 4

D Step 5

💡 Hint

Look at the 'Action' column in execution_table for validation

If you change the replication factor to 2, which variable in variable_tracker changes?

A admin_client

B new_topic

C bootstrap_servers

D None

💡 Hint

Replication factor is part of the NewTopic object tracked in variable_tracker

Concept Snapshot

Kafka Topic Creation:
- Use KafkaAdminClient to connect
- Define NewTopic with name, partitions, replication
- Call create_topics with the NewTopic list
- Kafka validates and creates topic
- Errors if invalid or duplicate

Full Transcript

This visual trace shows how to create a Kafka topic programmatically. First, a KafkaAdminClient is initialized to connect to the Kafka server. Then, a NewTopic object is defined with the topic name, number of partitions, and replication factor. The create_topics method is called with this topic list, sending a request to the Kafka server. The server validates the topic name and settings. If valid, the topic is created successfully; otherwise, an error is returned. Variables like admin_client and new_topic track the client and topic state through the process.

Practice

1. What is the main purpose of a topic in Kafka?

easy

A. To compile Java programs

B. To store user login credentials

C. To manage database connections

D. To organize messages like folders organize files

Solution

1. Step 1: Understand Kafka topic role

   A Kafka topic is a category or feed name to which messages are published.

2. Step 2: Compare options with topic function

   Only To organize messages like folders organize files correctly describes topics as organizing messages like folders organize files.

3. Final Answer:

   To organize messages like folders organize files -> Option D

4. Quick Check:

   Topic = Organize messages [OK]

Hint: Topics group messages like folders group files [OK]

Common Mistakes:

• Confusing topics with databases
• Thinking topics store user data
• Mixing topics with program compilation

2. Which command correctly creates a Kafka topic named orders with 3 partitions and replication factor 2?

easy

A. kafka-topics.sh --create --topic orders --partitions 3 --replication-factor 2 --bootstrap-server localhost:9092

B. kafka-create-topic orders 3 2

C. create-topic --name orders --partitions 3 --replicas 2

D. kafka-topics.sh --new --topic orders --partitions=3 --replicas=2

Solution

1. Step 1: Identify correct Kafka topic creation syntax

   The official command uses kafka-topics.sh with --create, --topic, --partitions, --replication-factor, and --bootstrap-server options.

2. Step 2: Match options with correct syntax

   kafka-topics.sh --create --topic orders --partitions 3 --replication-factor 2 --bootstrap-server localhost:9092 matches the correct syntax exactly; others use wrong commands or options.

3. Final Answer:

   kafka-topics.sh --create --topic orders --partitions 3 --replication-factor 2 --bootstrap-server localhost:9092 -> Option A

4. Quick Check:

   Correct syntax = kafka-topics.sh --create --topic orders --partitions 3 --replication-factor 2 --bootstrap-server localhost:9092 [OK]

Hint: Use kafka-topics.sh with --create and proper flags [OK]

Common Mistakes:

• Using wrong command names
• Missing --bootstrap-server option
• Incorrect option names like --replicas

3. What will happen if you run this command?

kafka-topics.sh --create --topic sales --partitions 5 --replication-factor 1 --bootstrap-server localhost:9092

medium

A. Creates a topic named 'sales' with 1 partition and replication factor 5

B. Fails because replication factor cannot be 1

C. Creates a topic named 'sales' with 5 partitions and replication factor 1

D. Fails due to missing zookeeper connection

Solution

1. Step 1: Analyze command parameters

   The command specifies topic 'sales', 5 partitions, replication factor 1, and bootstrap server.

2. Step 2: Understand Kafka topic creation rules

   Replication factor 1 is valid (means no replicas), and partitions set to 5 means 5 partitions created.

3. Final Answer:

   Creates a topic named 'sales' with 5 partitions and replication factor 1 -> Option C

4. Quick Check:

   Partitions=5, Replication=1 [OK]

Hint: Replication factor 1 means no replicas, partitions as given [OK]

Common Mistakes:

• Confusing partitions with replication factor
• Assuming replication factor 1 is invalid
• Expecting zookeeper connection with bootstrap-server

4. You tried to create a topic with this command:

kafka-topics.sh --create --topic logs --partitions two --replication-factor 3 --bootstrap-server localhost:9092

What is the error?

medium

A. Missing --create flag

B. Invalid number for partitions: 'two' should be a number

C. Replication factor cannot be 3

D. Bootstrap server address is incorrect

Solution

1. Step 1: Check partitions parameter

   The partitions option expects a number, but 'two' is a word, causing a syntax error.

2. Step 2: Verify other parameters

   Replication factor 3 and bootstrap server are valid; --create flag is present.

3. Final Answer:

   Invalid number for partitions: 'two' should be a number -> Option B

4. Quick Check:

   Partitions must be numeric [OK]

Hint: Partitions must be numeric, not words [OK]

Common Mistakes:

• Using words instead of numbers for partitions
• Assuming replication factor 3 is invalid
• Forgetting --create flag

5. You want to create a Kafka topic for a critical system that must not lose messages and should handle high load. Which topic creation settings are best?

hard

A. Partitions: 10, Replication factor: 3

B. Partitions: 1, Replication factor: 1

C. Partitions: 5, Replication factor: 1

D. Partitions: 3, Replication factor: 2

Solution

1. Step 1: Understand requirements for critical system

   High load needs many partitions for parallelism; no message loss needs high replication factor.

2. Step 2: Evaluate options

   Partitions: 10, Replication factor: 3 has 10 partitions (high load) and replication factor 3 (high safety). Others have fewer partitions or lower replication.

3. Final Answer:

   Partitions: 10, Replication factor: 3 -> Option A

4. Quick Check:

   High load + safety = many partitions + high replication [OK]

Hint: More partitions + higher replication = speed + safety [OK]

Common Mistakes:

• Choosing low replication for safety
• Choosing few partitions for high load
• Ignoring replication factor importance