Geo-replication strategies in Kafka - Mini Project: Build & Apply

Geo-replication Strategies with Kafka

📖 Scenario: You work for a company that has data centers in two cities. You want to make sure messages sent in one city are also available in the other city. This helps keep data safe and available even if one city has problems.

🎯 Goal: You will create a simple Kafka setup that shows how to replicate messages from one cluster to another using geo-replication strategies.

📋 What You'll Learn

Create a list of topics to replicate

Set a replication factor for topics

Configure a MirrorMaker 2 connector for geo-replication

Print the final MirrorMaker 2 configuration

💡 Why This Matters

🌍 Real World

Companies use geo-replication to keep data safe and available across different locations. This helps avoid data loss if one data center fails.

💼 Career

Understanding Kafka geo-replication is important for roles like DevOps engineers, data engineers, and system architects who manage distributed data systems.

1

Create a list of Kafka topics to replicate

Create a list called topics_to_replicate with these exact topic names: 'orders', 'payments', and 'shipments'.

Kafka

# Create the list of topics to replicate
# Your code here

Need a hint?

Use square brackets to create a list and put the topic names as strings inside.

2

Set the replication factor for the topics

Create a variable called replication_factor and set it to 3 to specify how many copies of each topic should exist.

Kafka

topics_to_replicate = ['orders', 'payments', 'shipments']
# Set the replication factor to 3
# Your code here

Need a hint?

Just assign the number 3 to the variable replication_factor.

3

Configure MirrorMaker 2 for geo-replication

Create a dictionary called mirrormaker_config with these exact keys and values: 'source.cluster.bootstrap.servers' set to 'source.kafka:9092', 'target.cluster.bootstrap.servers' set to 'target.kafka:9092', and 'topics' set to a comma-separated string of the topics in topics_to_replicate.

Kafka

topics_to_replicate = ['orders', 'payments', 'shipments']
replication_factor = 3
# Create MirrorMaker 2 configuration dictionary
# Your code here

Need a hint?

Use a dictionary with the keys and values exactly as shown. Use ','.join(topics_to_replicate) to make a string of topics separated by commas.

4

Print the MirrorMaker 2 configuration

Write a print statement to display the mirrormaker_config dictionary.

Kafka

topics_to_replicate = ['orders', 'payments', 'shipments']
replication_factor = 3
mirrormaker_config = {
    'source.cluster.bootstrap.servers': 'source.kafka:9092',
    'target.cluster.bootstrap.servers': 'target.kafka:9092',
    'topics': ','.join(topics_to_replicate)
}
# Print the MirrorMaker 2 configuration
# Your code here

Need a hint?

Use print(mirrormaker_config) to show the dictionary.