How to Configure Topic Settings in Kafka: Step-by-Step Guide

You configure Kafka topic settings using the kafka-topics.sh command with options like --partitions, --replication-factor, and --config to set properties such as retention time. These settings can be applied when creating a topic or altered later with the --alter flag.

Syntax

The main command to configure Kafka topic settings is kafka-topics.sh. You use it with these key options:

• --create: to create a new topic
• --alter: to change settings of an existing topic
• --topic: specify the topic name
• --partitions: number of partitions
• --replication-factor: number of replicas
• --config: set specific configuration properties like retention
• --bootstrap-server: specify the Kafka server address

Example syntax to create a topic:

kafka-topics.sh --create --topic <topic-name> --partitions <num> --replication-factor <num> --config <key=value> --bootstrap-server <server>

bash

kafka-topics.sh --create --topic <topic-name> --partitions <num> --replication-factor <num> --config <key=value> --bootstrap-server <server>

Example

This example creates a topic named my-topic with 3 partitions, a replication factor of 2, and sets the retention time to 1 day (86400000 ms).

bash

kafka-topics.sh --create --topic my-topic --partitions 3 --replication-factor 2 --config retention.ms=86400000 --bootstrap-server localhost:9092

Output

Created topic my-topic.

Common Pitfalls

Common mistakes when configuring Kafka topics include:

• Setting a replication factor higher than the number of brokers, which causes errors.
• Forgetting to specify --bootstrap-server, so the command cannot connect to Kafka.
• Trying to change immutable settings like partitions without using the correct command or understanding limitations.
• Misconfiguring retention settings with wrong units or keys.

Example of wrong and right usage:

bash

## Wrong: replication factor too high
kafka-topics.sh --create --topic test --partitions 1 --replication-factor 5 --bootstrap-server localhost:9092

## Right: replication factor matches broker count
kafka-topics.sh --create --topic test --partitions 1 --replication-factor 3 --bootstrap-server localhost:9092

Output

Error: Replication factor: 5 larger than available brokers: 3
Created topic test.

Quick Reference

Option	Description	Example
--create	Create a new topic	kafka-topics.sh --create --topic my-topic --bootstrap-server localhost:9092
--alter	Change settings of existing topic	kafka-topics.sh --alter --topic my-topic --bootstrap-server localhost:9092
--partitions	Number of partitions	--partitions 3
--replication-factor	Number of replicas	--replication-factor 2
--config	Set topic config key=value	--config retention.ms=86400000
--bootstrap-server	Kafka server address	--bootstrap-server localhost:9092

Key Takeaways

Use kafka-topics.sh with --create or --alter to configure topic settings.

Set partitions and replication factor carefully to match your cluster size.

Use --config to customize topic properties like retention time.

Always specify --bootstrap-server to connect to your Kafka cluster.

Check for errors when replication factor exceeds broker count.