Geo-replication strategies in Kafka - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

Output of Kafka MirrorMaker 2 configuration

Given the following MirrorMaker 2 configuration snippet, what will be the output in the target cluster after replication?

Kafka

clusters = {
  "source": {"bootstrap.servers": "source.kafka:9092"},
  "target": {"bootstrap.servers": "target.kafka:9092"}
}
tasks.max = 2

replication.policy.class = "org.apache.kafka.connect.mirror.DefaultReplicationPolicy"

# Topics to replicate
replication.topics = "topicA,topicB"

# MirrorMaker 2 replicates topicA and topicB from source to target

# Assume topicA has 3 partitions with messages 1 to 3, topicB has 2 partitions with messages 10 to 11

# What messages will appear in target cluster topics after MirrorMaker 2 runs?

A topicA with partitions 0-2 containing messages 10 to 11, topicB with partitions 0-1 containing messages 1 to 3

B topicA with partitions 0-2 containing messages 1 to 3, topicB with partitions 0-1 containing messages 10 to 11

C topicA and topicB will be empty in target cluster

D Only topicA will be replicated with messages 1 to 3; topicB will not be replicated

💡 Hint

MirrorMaker 2 replicates specified topics with their partitions and messages from source to target.

MirrorMaker 2 replicates the specified topics (topicA and topicB) from the source cluster to the target cluster, preserving partitions and messages.

🧠 Conceptual

intermediate

Understanding active-active geo-replication

Which statement best describes active-active geo-replication in Kafka?

A Replication happens only during off-peak hours to reduce network load.

B Data is replicated only from one primary cluster to multiple secondary clusters without writes on secondaries.

C Multiple clusters can accept writes and replicate data to each other, enabling multi-region active writes.

D Data is replicated from secondary clusters back to the primary cluster only.

💡 Hint

Active-active means all clusters can write data and replicate.

Active-active geo-replication allows multiple Kafka clusters in different regions to accept writes and replicate data to each other, supporting multi-region active writes.

🔧 Debug

advanced

Identify the error in MirrorMaker 2 replication config

What error will occur when running MirrorMaker 2 with this configuration snippet? replication.policy.class = "org.apache.kafka.connect.mirror.NonExistentPolicy" clusters = { "source": {"bootstrap.servers": "source.kafka:9092"}, "target": {"bootstrap.servers": "target.kafka:9092"} } replication.topics = "topicX"

A ClassNotFoundException for org.apache.kafka.connect.mirror.NonExistentPolicy

B No error, MirrorMaker 2 will replicate topicX correctly

C Timeout error connecting to source.kafka:9092

D SyntaxError in configuration file

💡 Hint

Check if the replication policy class exists in Kafka MirrorMaker 2.

MirrorMaker 2 requires a valid replication policy class. Using a non-existent class causes a ClassNotFoundException at startup.

📝 Syntax

advanced

Correct syntax for MirrorMaker 2 topic whitelist

Which option correctly sets MirrorMaker 2 to replicate only topics starting with 'logs-'?

A replication.topics.regex = "logs-.*"

B replication.topics = "logs-*"

C replication.topics = logs-*

D replication.topics.regex = logs-*

💡 Hint

MirrorMaker 2 uses regex property for pattern matching topics.

To replicate topics by pattern, MirrorMaker 2 uses replication.topics.regex with a valid regex string.

🚀 Application

expert

Designing a geo-replication strategy for low latency writes

You have two Kafka clusters in different continents. You want to allow applications in both regions to write data with minimal latency and keep data synchronized. Which geo-replication strategy should you choose?

A Active-passive replication with MirrorMaker 2 replicating from primary to secondary cluster only

B No replication, rely on manual data export/import between clusters

C Single cluster with global clients connecting remotely

D Active-active replication with multi-master clusters replicating data bidirectionally

💡 Hint

Consider latency and write availability in multiple regions.

Active-active replication allows both clusters to accept writes locally, reducing latency and replicating data bidirectionally to keep clusters synchronized.

Practice

1. What is the main purpose of geo-replication in Kafka?

easy

A. To encrypt Kafka topics for security

B. To compress Kafka messages for faster transmission

C. To copy data between Kafka clusters in different locations for safety and availability

D. To delete old Kafka data automatically

Solution

1. Step 1: Understand geo-replication concept

   Geo-replication means copying data between clusters in different places to keep data safe and accessible.

2. Step 2: Identify the main goal

   The main goal is to keep data safe and close to users worldwide by copying it across locations.

3. Final Answer:

   To copy data between Kafka clusters in different locations for safety and availability -> Option C

4. Quick Check:

   Geo-replication = Data copying across locations [OK]

Hint: Geo-replication means copying data across locations [OK]

Common Mistakes:

• Confusing geo-replication with data compression
• Thinking geo-replication encrypts data
• Assuming geo-replication deletes data

2. Which Kafka tool is commonly used for geo-replication?

easy

A. MirrorMaker 2

B. Kafka Connect Sink

C. Kafka Streams

D. Kafka Producer API

Solution

1. Step 1: Recall Kafka geo-replication tools

   MirrorMaker 2 is designed to replicate data between Kafka clusters in different locations.

2. Step 2: Differentiate from other tools

   Kafka Streams processes data streams, Kafka Connect Sink moves data to external systems, and Producer API sends data to Kafka but does not replicate clusters.

3. Final Answer:

   MirrorMaker 2 -> Option A

4. Quick Check:

   Geo-replication tool = MirrorMaker 2 [OK]

Hint: MirrorMaker 2 is for replicating Kafka clusters [OK]

Common Mistakes:

• Choosing Kafka Streams which is for processing, not replication
• Confusing Kafka Connect Sink with replication tool
• Selecting Producer API which only sends data

3. Given the following MirrorMaker 2 configuration snippet, what is the effect?

{
  "clusters": {
    "A": {"bootstrap.servers": "a.kafka:9092"},
    "B": {"bootstrap.servers": "b.kafka:9092"}
  },
  "mirrors": [{"source.cluster": "A", "target.cluster": "B", "topics": ["orders"]}]
}

medium

A. Data from cluster B is copied to cluster A for topic 'orders'

B. Data from cluster A is copied to cluster B for topic 'orders'

C. Data from both clusters A and B are merged into a new topic 'orders'

D. No data is copied because topics list is empty

Solution

1. Step 1: Analyze source and target clusters

   The config shows source.cluster as A and target.cluster as B, meaning data flows from A to B.

2. Step 2: Check topics list

   The topics list includes "orders", so only the 'orders' topic is replicated from A to B.

3. Final Answer:

   Data from cluster A is copied to cluster B for topic 'orders' -> Option B

4. Quick Check:

   Source A to target B for 'orders' topic [OK]

Hint: Source cluster data replicates to target cluster [OK]

Common Mistakes:

• Reversing source and target clusters
• Thinking topics list is empty
• Assuming data merges instead of copies

4. You configured MirrorMaker 2 but no data is replicating. Which is the most likely error in this config?

{
  "clusters": {
    "source": {"bootstrap.servers": "source.kafka:9092"},
    "target": {"bootstrap.servers": "target.kafka:9092"}
  },
  "mirrors": [{"source.cluster": "source", "target.cluster": "target", "topics": []}]
}

medium

A. Bootstrap servers are incorrectly formatted

B. MirrorMaker 2 requires only one cluster in config

C. Cluster names must be numeric, not strings

D. Empty topics list means no topics are selected for replication

Solution

1. Step 1: Check topics list in mirror config

   The topics list is empty [], so no topics are selected to replicate.

2. Step 2: Validate other config parts

   Bootstrap servers are correctly formatted, cluster names as strings are valid, and multiple clusters are allowed.

3. Final Answer:

   Empty topics list means no topics are selected for replication -> Option D

4. Quick Check:

   Empty topics list = no replication [OK]

Hint: Empty topics list disables replication [OK]

Common Mistakes:

• Assuming bootstrap.servers format is wrong
• Thinking cluster names must be numbers
• Believing MirrorMaker 2 supports only one cluster

5. You want to set up geo-replication between three Kafka clusters (US, EU, Asia) so that each cluster has all topics from the others. Which strategy should you use?

hard

A. Set up MirrorMaker 2 with a full mesh replication where each cluster replicates to the other two

B. Use a single MirrorMaker 2 instance replicating only from US to EU and Asia

C. Configure MirrorMaker 2 to replicate only the largest topic across clusters

D. Use Kafka Connect Sink to move data between clusters

Solution

1. Step 1: Understand full data availability goal

   Each cluster should have all topics from the others, meaning data must flow both ways between all clusters.

2. Step 2: Choose replication strategy

   A full mesh replication with MirrorMaker 2 means each cluster replicates to the other two, ensuring complete data presence everywhere.

3. Final Answer:

   Set up MirrorMaker 2 with a full mesh replication where each cluster replicates to the other two -> Option A

4. Quick Check:

   Full mesh replication = all clusters have all topics [OK]

Hint: Full mesh replicates all topics between all clusters [OK]

Common Mistakes:

• Using one-way replication only
• Replicating only largest topic, missing others
• Confusing Kafka Connect Sink with cluster replication