Topic configuration in Kafka - Mini Project: Build & Apply

Kafka Topic Configuration

📖 Scenario: You are setting up a Kafka topic for a simple messaging system. You need to create the topic with specific settings to control how messages are stored and replicated.

🎯 Goal: Create a Kafka topic named user-events with a specific number of partitions and replication factor. Then configure the topic to have a retention time of 1 day (86400000 milliseconds). Finally, display the topic configuration.

📋 What You'll Learn

Create a topic named user-events with 3 partitions and a replication factor of 2

Set the topic retention time to 86400000 milliseconds (1 day)

Print the topic configuration to verify the settings

💡 Why This Matters

🌍 Real World

Kafka topics are used to organize and manage streams of messages in real-time data pipelines and event-driven applications.

💼 Career

Knowing how to configure Kafka topics is essential for roles like data engineers, backend developers, and system administrators working with distributed messaging systems.

1

Create the Kafka topic with partitions and replication factor

Write the command to create a Kafka topic named user-events with 3 partitions and a replication factor of 2 using kafka-topics.sh.

Kafka

# Your command here to create the topic

Need a hint?

Use the kafka-topics.sh script with the --create option and specify the topic name, partitions, and replication factor.

2

Configure the topic retention time

Write the command to set the retention time of the user-events topic to 86400000 milliseconds using kafka-configs.sh.

Kafka

kafka-topics.sh --create --topic user-events --partitions 3 --replication-factor 2 --bootstrap-server localhost:9092
# Your command here to set retention.ms

Need a hint?

Use kafka-configs.sh with --alter to change the topic configuration and set retention.ms.

3

Describe the topic to check configuration

Write the command to describe the user-events topic and show its configuration using kafka-topics.sh.

Kafka

kafka-topics.sh --create --topic user-events --partitions 3 --replication-factor 2 --bootstrap-server localhost:9092
kafka-configs.sh --alter --entity-type topics --entity-name user-events --add-config retention.ms=86400000 --bootstrap-server localhost:9092
# Your command here to describe the topic

Need a hint?

Use kafka-topics.sh --describe to see the topic details and configuration.

4

Output the topic configuration

Run the command to describe the user-events topic and print the output showing partitions, replication factor, and retention time.

Kafka

kafka-topics.sh --create --topic user-events --partitions 3 --replication-factor 2 --bootstrap-server localhost:9092
kafka-configs.sh --alter --entity-type topics --entity-name user-events --add-config retention.ms=86400000 --bootstrap-server localhost:9092
kafka-topics.sh --describe --topic user-events --bootstrap-server localhost:9092
# Run the above command and print the output below

Need a hint?

Check that the output shows the topic name, number of partitions, replication factor, and retention.ms configuration.