Disaster recovery planning in Kafka - Mini Project: Build & Apply

Disaster Recovery Planning with Kafka

📖 Scenario: You work for a company that uses Apache Kafka to handle real-time data streams. To keep the system safe from failures, you need to plan how to recover data if something goes wrong.This project will guide you through creating a simple Kafka topic setup with replication and configuring a consumer group to read messages safely, which are key parts of disaster recovery planning.

🎯 Goal: Build a basic Kafka setup with a topic that has replication enabled and a consumer group that reads messages reliably. This setup helps ensure data is not lost and can be recovered if a server fails.

📋 What You'll Learn

Create a Kafka topic named orders with 3 partitions and replication factor 2

Set a configuration variable consumer_group with the value order_processors

Write a Kafka consumer that subscribes to the orders topic using the consumer_group

Print the consumed messages to the console

💡 Why This Matters

🌍 Real World

Kafka is widely used in industries like finance, retail, and tech to process data streams reliably. Disaster recovery planning ensures data is safe and systems can recover quickly from failures.

💼 Career

Understanding Kafka topic replication and consumer groups is essential for roles like data engineer, site reliability engineer, and backend developer working with real-time data pipelines.

1

Create Kafka topic with replication

Create a Kafka topic named orders with 3 partitions and a replication factor of 2 using the kafka-topics.sh command.

Kafka

# Use kafka-topics.sh to create the topic
# Your code here

Need a hint?

Use the kafka-topics.sh tool with the --create option and specify the topic name, partitions, and replication factor.

2

Set consumer group variable

Create a variable called consumer_group and set it to the string order_processors.

Kafka

kafka-topics.sh --create --topic orders --partitions 3 --replication-factor 2 --bootstrap-server localhost:9092

# Create a variable called consumer_group with value 'order_processors'
# Your code here

Need a hint?

Assign the string 'order_processors' to the variable consumer_group.

3

Write Kafka consumer subscribing to topic

Write a Kafka consumer in Python that subscribes to the orders topic using the consumer_group variable. Use the KafkaConsumer class from the kafka library and set the group_id to consumer_group.

Kafka

kafka-topics.sh --create --topic orders --partitions 3 --replication-factor 2 --bootstrap-server localhost:9092

consumer_group = 'order_processors'

# Import KafkaConsumer and create a consumer subscribing to 'orders' with group_id consumer_group
# Your code here

Need a hint?

Import KafkaConsumer and create a consumer subscribing to the orders topic with the group ID set to consumer_group.

4

Print consumed messages

Write a for loop to read messages from the consumer and print each message's value decoded as UTF-8.

Kafka

kafka-topics.sh --create --topic orders --partitions 3 --replication-factor 2 --bootstrap-server localhost:9092

consumer_group = 'order_processors'

from kafka import KafkaConsumer
consumer = KafkaConsumer('orders', group_id=consumer_group, bootstrap_servers=['localhost:9092'])

# Loop over consumer and print each message value decoded as UTF-8
# Your code here

Need a hint?

Use a for loop to read messages from the consumer and print the decoded message values.