Geo-replication strategies in Kafka - Mini Project: Build & Apply

Geo-replication Strategies with Kafka

📖 Scenario: You work for a company that has data centers in two cities. You want to make sure messages sent in one city are also available in the other city. This helps keep data safe and available even if one city has problems.

🎯 Goal: You will create a simple Kafka setup that shows how to replicate messages from one cluster to another using geo-replication strategies.

📋 What You'll Learn

Create a list of topics to replicate

Set a replication factor for topics

Configure a MirrorMaker 2 connector for geo-replication

Print the final MirrorMaker 2 configuration

💡 Why This Matters

🌍 Real World

Companies use geo-replication to keep data safe and available across different locations. This helps avoid data loss if one data center fails.

💼 Career

Understanding Kafka geo-replication is important for roles like DevOps engineers, data engineers, and system architects who manage distributed data systems.

1

Create a list of Kafka topics to replicate

Create a list called topics_to_replicate with these exact topic names: 'orders', 'payments', and 'shipments'.

Kafka

# Create the list of topics to replicate
# Your code here

Hint

Use square brackets to create a list and put the topic names as strings inside.

2

Set the replication factor for the topics

Create a variable called replication_factor and set it to 3 to specify how many copies of each topic should exist.

Kafka

topics_to_replicate = ['orders', 'payments', 'shipments']
# Set the replication factor to 3
# Your code here

Hint

Just assign the number 3 to the variable replication_factor.

3

Configure MirrorMaker 2 for geo-replication

Create a dictionary called mirrormaker_config with these exact keys and values: 'source.cluster.bootstrap.servers' set to 'source.kafka:9092', 'target.cluster.bootstrap.servers' set to 'target.kafka:9092', and 'topics' set to a comma-separated string of the topics in topics_to_replicate.

Kafka

topics_to_replicate = ['orders', 'payments', 'shipments']
replication_factor = 3
# Create MirrorMaker 2 configuration dictionary
# Your code here

Hint

Use a dictionary with the keys and values exactly as shown. Use ','.join(topics_to_replicate) to make a string of topics separated by commas.

4

Print the MirrorMaker 2 configuration

Write a print statement to display the mirrormaker_config dictionary.

Kafka

topics_to_replicate = ['orders', 'payments', 'shipments']
replication_factor = 3
mirrormaker_config = {
    'source.cluster.bootstrap.servers': 'source.kafka:9092',
    'target.cluster.bootstrap.servers': 'target.kafka:9092',
    'topics': ','.join(topics_to_replicate)
}
# Print the MirrorMaker 2 configuration
# Your code here

Hint

Use print(mirrormaker_config) to show the dictionary.

Practice

1. What is the main purpose of geo-replication in Kafka?

easy

A. To encrypt Kafka topics for security

B. To compress Kafka messages for faster transmission

C. To copy data between Kafka clusters in different locations for safety and availability

D. To delete old Kafka data automatically

Solution

1. Step 1: Understand geo-replication concept

   Geo-replication means copying data between clusters in different places to keep data safe and accessible.

2. Step 2: Identify the main goal

   The main goal is to keep data safe and close to users worldwide by copying it across locations.

3. Final Answer:

   To copy data between Kafka clusters in different locations for safety and availability -> Option C

4. Quick Check:

   Geo-replication = Data copying across locations [OK]

Hint: Geo-replication means copying data across locations [OK]

Common Mistakes:

• Confusing geo-replication with data compression
• Thinking geo-replication encrypts data
• Assuming geo-replication deletes data

2. Which Kafka tool is commonly used for geo-replication?

easy

A. MirrorMaker 2

B. Kafka Connect Sink

C. Kafka Streams

D. Kafka Producer API

Solution

1. Step 1: Recall Kafka geo-replication tools

   MirrorMaker 2 is designed to replicate data between Kafka clusters in different locations.

2. Step 2: Differentiate from other tools

   Kafka Streams processes data streams, Kafka Connect Sink moves data to external systems, and Producer API sends data to Kafka but does not replicate clusters.

3. Final Answer:

   MirrorMaker 2 -> Option A

4. Quick Check:

   Geo-replication tool = MirrorMaker 2 [OK]

Hint: MirrorMaker 2 is for replicating Kafka clusters [OK]

Common Mistakes:

• Choosing Kafka Streams which is for processing, not replication
• Confusing Kafka Connect Sink with replication tool
• Selecting Producer API which only sends data

3. Given the following MirrorMaker 2 configuration snippet, what is the effect?

{
  "clusters": {
    "A": {"bootstrap.servers": "a.kafka:9092"},
    "B": {"bootstrap.servers": "b.kafka:9092"}
  },
  "mirrors": [{"source.cluster": "A", "target.cluster": "B", "topics": ["orders"]}]
}

medium

A. Data from cluster B is copied to cluster A for topic 'orders'

B. Data from cluster A is copied to cluster B for topic 'orders'

C. Data from both clusters A and B are merged into a new topic 'orders'

D. No data is copied because topics list is empty

Solution

1. Step 1: Analyze source and target clusters

   The config shows source.cluster as A and target.cluster as B, meaning data flows from A to B.

2. Step 2: Check topics list

   The topics list includes "orders", so only the 'orders' topic is replicated from A to B.

3. Final Answer:

   Data from cluster A is copied to cluster B for topic 'orders' -> Option B

4. Quick Check:

   Source A to target B for 'orders' topic [OK]

Hint: Source cluster data replicates to target cluster [OK]

Common Mistakes:

• Reversing source and target clusters
• Thinking topics list is empty
• Assuming data merges instead of copies

4. You configured MirrorMaker 2 but no data is replicating. Which is the most likely error in this config?

{
  "clusters": {
    "source": {"bootstrap.servers": "source.kafka:9092"},
    "target": {"bootstrap.servers": "target.kafka:9092"}
  },
  "mirrors": [{"source.cluster": "source", "target.cluster": "target", "topics": []}]
}

medium

A. Bootstrap servers are incorrectly formatted

B. MirrorMaker 2 requires only one cluster in config

C. Cluster names must be numeric, not strings

D. Empty topics list means no topics are selected for replication

Solution

1. Step 1: Check topics list in mirror config

   The topics list is empty [], so no topics are selected to replicate.

2. Step 2: Validate other config parts

   Bootstrap servers are correctly formatted, cluster names as strings are valid, and multiple clusters are allowed.

3. Final Answer:

   Empty topics list means no topics are selected for replication -> Option D

4. Quick Check:

   Empty topics list = no replication [OK]

Hint: Empty topics list disables replication [OK]

Common Mistakes:

• Assuming bootstrap.servers format is wrong
• Thinking cluster names must be numbers
• Believing MirrorMaker 2 supports only one cluster

5. You want to set up geo-replication between three Kafka clusters (US, EU, Asia) so that each cluster has all topics from the others. Which strategy should you use?

hard

A. Set up MirrorMaker 2 with a full mesh replication where each cluster replicates to the other two

B. Use a single MirrorMaker 2 instance replicating only from US to EU and Asia

C. Configure MirrorMaker 2 to replicate only the largest topic across clusters

D. Use Kafka Connect Sink to move data between clusters

Solution

1. Step 1: Understand full data availability goal

   Each cluster should have all topics from the others, meaning data must flow both ways between all clusters.

2. Step 2: Choose replication strategy

   A full mesh replication with MirrorMaker 2 means each cluster replicates to the other two, ensuring complete data presence everywhere.

3. Final Answer:

   Set up MirrorMaker 2 with a full mesh replication where each cluster replicates to the other two -> Option A

4. Quick Check:

   Full mesh replication = all clusters have all topics [OK]

Hint: Full mesh replicates all topics between all clusters [OK]

Common Mistakes:

• Using one-way replication only
• Replicating only largest topic, missing others
• Confusing Kafka Connect Sink with cluster replication