Topic configuration in Kafka - Step-by-Step Execution

Process Flow - Topic configuration

Create Topic Request

↓

Broker Receives Request

↓

Validate Configurations

↓

Apply Configurations

↓

Topic Created with Settings

↓

Topic Ready for Use

This flow shows how a Kafka topic is created with specific configurations, validated by the broker, and then made ready for use.

Execution Sample

Kafka

bin/kafka-topics.sh --create --topic my-topic --partitions 3 --replication-factor 2 --config cleanup.policy=compact

This command creates a Kafka topic named 'my-topic' with 3 partitions, replication factor 2, and a cleanup policy set to compact.

Process Table

Step	Action	Configuration Parameter	Value	Result
1	Receive create topic request	topic	my-topic	Request accepted
2	Validate partitions	partitions	3	Valid number of partitions
3	Validate replication factor	replication-factor	2	Valid replication factor
4	Apply topic config	cleanup.policy	compact	Config applied
5	Create topic metadata	N/A	N/A	Topic metadata created
6	Topic creation complete	N/A	N/A	Topic 'my-topic' ready for use

💡 Topic creation ends after all validations and configurations are successfully applied.

Status Tracker

Variable	Start	After Step 2	After Step 3	After Step 4	Final
topic	N/A	my-topic	my-topic	my-topic	my-topic
partitions	N/A	3	3	3	3
replication-factor	N/A	N/A	2	2	2
cleanup.policy	default (delete)	default (delete)	default (delete)	compact	compact
topic status	not created	valid partitions	valid replication	config applied	ready

Key Moments - 3 Insights

Why must the replication factor be less than or equal to the number of brokers?

Because each replica must be on a different broker, so replication factor cannot exceed available brokers. See execution_table step 3 where replication factor is validated.

What happens if an invalid configuration is provided?

The broker rejects the topic creation request during validation steps, stopping the process before topic creation. This is implied in execution_table steps 2 and 3 validations.

Why is the cleanup.policy configuration important?

It controls how Kafka manages old data in the topic. Setting it to 'compact' changes retention behavior, as shown in step 4 where the config is applied.

Visual Quiz - 3 Questions

Look at the execution_table, what is the value of 'partitions' after step 2?

A default

B 2

C 3

D N/A

💡 Hint

Check the 'partitions' column in the execution_table row for step 2.

At which step is the cleanup.policy configuration applied?

A Step 2

B Step 4

C Step 3

D Step 5

💡 Hint

Look for the row mentioning 'Apply topic config' in the execution_table.

If the replication factor was set to 5 but only 3 brokers exist, what would happen during execution?

A Validation would fail at step 3

B Topic would be created with replication factor 5

C Cleanup policy would be ignored

D Partitions number would be adjusted automatically

💡 Hint

Refer to key_moments about replication factor validation and execution_table step 3.

Concept Snapshot

Kafka Topic Configuration:
- Create topic with partitions and replication factor
- Validate configs on broker
- Apply configs like cleanup.policy
- Topic ready after successful creation
- Invalid configs cause creation failure

Full Transcript

This visual execution shows how Kafka creates a topic with specific configurations. First, the create topic request is received with parameters like topic name, partitions, replication factor, and configs. The broker validates the number of partitions and replication factor to ensure they are valid. Then, it applies configurations such as cleanup.policy. After all validations and configurations are applied, the topic metadata is created and the topic becomes ready for use. If any validation fails, the topic creation stops. Variables like partitions and replication factor are tracked through each step to show their values. Key moments clarify why replication factor must be less than or equal to the number of brokers, what happens with invalid configs, and the importance of cleanup.policy. The quiz questions help reinforce understanding of the steps and values during topic creation.