Topic configuration in Kafka - Commands & Configuration

Introduction

Kafka topics are where messages are stored and organized. Configuring topics lets you control how data is kept, how long it stays, and how it is shared among users.

When you want to set how many copies of your data Kafka keeps for safety.

When you need to decide how long messages stay before they are deleted.

When you want to control how many parts a topic has to allow multiple users to read and write at the same time.

When you want to change the way Kafka cleans up old messages to save space.

When you want to adjust the speed and size of data Kafka handles for better performance.

Commands

This command creates a new Kafka topic named 'example-topic' with 3 partitions and 2 copies of data for safety. It also sets messages to be kept for 7 days (604800000 milliseconds).

Terminal

kafka-topics.sh --create --topic example-topic --bootstrap-server localhost:9092 --partitions 3 --replication-factor 2 --config retention.ms=604800000

Expected Output

Created topic example-topic.

→

--partitions - Sets how many parts the topic has for parallel processing

→

--replication-factor - Sets how many copies of data Kafka keeps

→

--config retention.ms - Sets how long messages stay before deletion

This command shows the details of the 'example-topic' to check its configuration and status.

Terminal

kafka-topics.sh --describe --topic example-topic --bootstrap-server localhost:9092

Expected Output

Topic: example-topic	PartitionCount: 3	ReplicationFactor: 2	Configs: retention.ms=604800000

	Topic: example-topic	Partition: 0	Leader: 1	Replicas: 1,2	Isr: 1,2

	Topic: example-topic	Partition: 1	Leader: 2	Replicas: 2,1	Isr: 2,1

	Topic: example-topic	Partition: 2	Leader: 1	Replicas: 1,2	Isr: 1,2

This command changes the retention time of 'example-topic' to 3 days (259200000 milliseconds) after creation.

Terminal

kafka-configs.sh --alter --entity-type topics --entity-name example-topic --add-config retention.ms=259200000 --bootstrap-server localhost:9092

Expected Output

No output (command runs silently)

→

--add-config - Adds or changes a configuration for the topic

This command checks the updated configuration of 'example-topic' to confirm the retention time change.

Terminal

kafka-topics.sh --describe --topic example-topic --bootstrap-server localhost:9092

Expected Output

Topic: example-topic	PartitionCount: 3	ReplicationFactor: 2	Configs: retention.ms=259200000

	Topic: example-topic	Partition: 0	Leader: 1	Replicas: 1,2	Isr: 1,2

	Topic: example-topic	Partition: 1	Leader: 2	Replicas: 2,1	Isr: 2,1

	Topic: example-topic	Partition: 2	Leader: 1	Replicas: 1,2	Isr: 1,2

Key Concept

If you remember nothing else from this pattern, remember: Kafka topic configurations control how data is stored, copied, and cleaned up to fit your needs.

Common Mistakes

Creating a topic without specifying replication-factor or partitions.

Kafka uses defaults that may not fit your needs, risking data loss or poor performance.

Always specify replication-factor and partitions explicitly when creating topics.

Changing topic configurations without using the correct command or flags.

Changes won't apply and the topic will keep old settings.

Use kafka-configs.sh with --alter and --add-config flags to update topic settings.

Setting retention.ms to a very low value without understanding impact.

Messages may be deleted too soon, causing data loss.

Set retention.ms carefully based on how long you need to keep messages.

Summary

Use kafka-topics.sh with --create to make a topic with partitions, replication, and configs.

Use kafka-topics.sh with --describe to check topic details and confirm settings.

Use kafka-configs.sh with --alter and --add-config to change topic configurations after creation.

Practice

1. What does the retention.ms configuration control in a Kafka topic?

easy

A. How long messages are kept before deletion

B. The number of partitions in the topic

C. The replication factor of the topic

D. The maximum message size allowed

Solution

1. Step 1: Understand retention.ms meaning

   The retention.ms setting defines the time in milliseconds that Kafka retains messages before deleting them.

2. Step 2: Compare with other options

   Other options like partitions, replication factor, and message size are controlled by different configurations.

3. Final Answer:

   How long messages are kept before deletion -> Option A

4. Quick Check:

   retention.ms = message retention time [OK]

Hint: Retention means how long messages stay before removal [OK]

Common Mistakes:

• Confusing retention.ms with replication factor
• Thinking retention.ms controls partition count
• Assuming retention.ms limits message size

2. Which of the following is the correct syntax to create a Kafka topic with a replication factor of 3 using the command line?

easy

A. kafka-topics.sh --create --topic myTopic --replication-factor 3 --partitions 1 --bootstrap-server localhost:9092

B. kafka-topics.sh --create --topic myTopic --replication 3 --partitions 1 --bootstrap-server localhost:9092

C. kafka-topics.sh --create --topic myTopic --replica-factor 3 --partitions 1 --bootstrap-server localhost:9092

D. kafka-topics.sh --create --topic myTopic --replication-factor=3 --partitions 1 --bootstrap-server localhost:9092

Solution

1. Step 1: Identify correct option name

   The correct option for replication factor is --replication-factor without equals sign.

2. Step 2: Check syntax correctness

   kafka-topics.sh --create --topic myTopic --replication-factor 3 --partitions 1 --bootstrap-server localhost:9092 uses the correct syntax with spaces and proper option names; others have wrong option names or equals sign which is invalid here.

3. Final Answer:

   kafka-topics.sh --create --topic myTopic --replication-factor 3 --partitions 1 --bootstrap-server localhost:9092 -> Option A

4. Quick Check:

   Use --replication-factor with space, no equals [OK]

Hint: Use --replication-factor with space, no equals sign [OK]

Common Mistakes:

• Using --replication instead of --replication-factor
• Adding equals sign (=) after option name
• Misspelling replication-factor option

3. Given the command:
kafka-topics.sh --alter --topic logs --config retention.ms=60000 --bootstrap-server localhost:9092
What will be the effect on the topic logs?

medium

A. Messages will be retained indefinitely

B. The topic's replication factor will be set to 60000

C. The topic will be deleted after 60000 messages

D. Messages will be retained for 1 minute before deletion

Solution

1. Step 1: Understand the command's purpose

   The command alters the topic logs by setting retention.ms to 60000 milliseconds (1 minute).

2. Step 2: Interpret retention.ms effect

   This means messages older than 1 minute will be deleted automatically.

3. Final Answer:

   Messages will be retained for 1 minute before deletion -> Option D

4. Quick Check:

   retention.ms=60000 means 1 minute retention [OK]

Hint: retention.ms value is in milliseconds for message lifetime [OK]

Common Mistakes:

• Thinking retention.ms controls message count
• Confusing retention.ms with replication factor
• Assuming retention.ms disables deletion

4. You tried to set the cleanup policy of a Kafka topic with:
kafka-topics.sh --alter --topic data --config cleanup.policy=delete,compact --bootstrap-server localhost:9092
But it failed. What is the likely cause?

medium

A. The --alter command cannot change cleanup.policy

B. The cleanup.policy value is invalid; it must be either 'delete' or 'compact', not both

C. The topic name 'data' is reserved and cannot be altered

D. The bootstrap server address is incorrect

Solution

1. Step 1: Check cleanup.policy valid values

   Kafka allows cleanup.policy to be either 'delete' or 'compact' or both separated by comma, but the syntax must be exact and supported by the Kafka version.

2. Step 2: Identify common error

   Some Kafka versions or setups do not support combined policies or require no spaces; if the command failed, likely the value was invalid or syntax incorrect.

3. Final Answer:

   The cleanup.policy value is invalid; it must be either 'delete' or 'compact', not both -> Option B

4. Quick Check:

   cleanup.policy must be valid single or combined value [OK]

Hint: Check cleanup.policy values carefully; some setups disallow combined [OK]

Common Mistakes:

• Using unsupported combined cleanup policies
• Trying to alter topic with wrong bootstrap server
• Assuming topic names cannot be altered

5. You want to create a Kafka topic named events with 4 partitions, replication factor 2, and a retention time of 2 hours. Which command correctly sets all these configurations?

hard

A. kafka-topics.sh --create --topic events --partitions 4 --replication-factor 2 --config retention.ms=120 --bootstrap-server localhost:9092

B. kafka-topics.sh --create --topic events --partitions 2 --replication-factor 4 --config retention.ms=7200000 --bootstrap-server localhost:9092

C. kafka-topics.sh --create --topic events --partitions 4 --replication-factor 2 --config retention.ms=7200000 --bootstrap-server localhost:9092

D. kafka-topics.sh --create --topic events --partitions 4 --replication-factor 2 --config retention.ms=7200 --bootstrap-server localhost:9092

Solution

1. Step 1: Verify partitions and replication factor

   The topic requires 4 partitions and replication factor 2, so options with these exact values are correct.

2. Step 2: Calculate retention.ms for 2 hours

   2 hours = 2 * 60 * 60 * 1000 = 7,200,000 milliseconds. Only kafka-topics.sh --create --topic events --partitions 4 --replication-factor 2 --config retention.ms=7200000 --bootstrap-server localhost:9092 uses this correct retention.ms value.

3. Final Answer:

   kafka-topics.sh --create --topic events --partitions 4 --replication-factor 2 --config retention.ms=7200000 --bootstrap-server localhost:9092 -> Option C

4. Quick Check:

   Retention in ms for 2 hours = 7200000 [OK]

Hint: Convert hours to ms: hours * 3600000 for retention.ms [OK]

Common Mistakes:

• Mixing partitions and replication factor values
• Using retention.ms in seconds or minutes instead of ms
• Incorrect retention.ms calculation