Cross-datacenter replication in Kafka - Mini Project: Build & Apply

Cross-datacenter replication with Kafka

📖 Scenario: You work for a company that has two data centers in different cities. They want to keep data synchronized between these centers using Kafka. This helps keep data safe and available even if one center has problems.

🎯 Goal: You will set up a simple Kafka topic and configure cross-datacenter replication so messages sent in one data center appear in the other.

📋 What You'll Learn

Create a Kafka topic named orders with replication factor 2

Set up a variable source_cluster with the source Kafka cluster address

Write a command to start MirrorMaker to replicate orders from source_cluster

Print the MirrorMaker command to verify the setup

💡 Why This Matters

🌍 Real World

Cross-datacenter replication keeps data safe and available by copying Kafka topics between data centers automatically.

💼 Career

Kafka administrators and DevOps engineers use these skills to ensure reliable data replication and disaster recovery.

1

Create Kafka topic orders

Create a Kafka topic called orders with replication-factor set to 2 and partitions set to 3. Use the command kafka-topics --create with these exact options.

Kafka

# Your code here

Need a hint?

Use the kafka-topics command with --create and specify --topic orders, --partitions 3, and --replication-factor 2.

2

Set source cluster address

Create a variable called source_cluster and set it to the string localhost:9092 which is the address of the source Kafka cluster.

Kafka

kafka-topics --create --topic orders --partitions 3 --replication-factor 2 --bootstrap-server localhost:9092

# Your code here

Need a hint?

Assign the string "localhost:9092" to the variable source_cluster.

3

Write MirrorMaker command for replication

Write a command string called mirror_command that uses kafka-mirror-maker to replicate the orders topic from the source_cluster. Use --consumer.config and --producer.config files named consumer.properties and producer.properties respectively. The command should include --whitelist orders to replicate only the orders topic.

Kafka

kafka-topics --create --topic orders --partitions 3 --replication-factor 2 --bootstrap-server localhost:9092

source_cluster = "localhost:9092"

# Your code here

Need a hint?

Create a string variable mirror_command with the full MirrorMaker command including --consumer.config consumer.properties, --producer.config producer.properties, and --whitelist orders.

4

Print the MirrorMaker command

Print the variable mirror_command to display the full MirrorMaker replication command.

Kafka

kafka-topics --create --topic orders --partitions 3 --replication-factor 2 --bootstrap-server localhost:9092

source_cluster = "localhost:9092"

mirror_command = f"kafka-mirror-maker --consumer.config consumer.properties --producer.config producer.properties --whitelist orders"

# Your code here

Need a hint?

Use print(mirror_command) to show the command.