Geo-replication strategies in Kafka - Commands & Configuration

Introduction

Geo-replication helps copy data between Kafka clusters in different locations. It keeps data available and consistent even if one site fails or is slow.

When you want to keep Kafka data available across multiple data centers for disaster recovery.

When users in different regions need fast access to local Kafka data copies.

When you want to balance load by replicating topics to clusters closer to users.

When you need to migrate Kafka data from one cluster to another without downtime.

When you want to ensure data durability by having copies in separate geographic locations.

Config File - replicator.properties

replicator.properties

bootstrap.servers=clusterA.kafka.local:9092
replication.policy.class=org.apache.kafka.connect.mirror.MirrorReplicationPolicy
replication.factor=3
topics=important-topic,logs
source.cluster.alias=clusterA
target.cluster.alias=clusterB
emit.heartbeats.enabled=true
heartbeat.interval.seconds=30
sync.topic.acls.enabled=true

This configuration file sets up MirrorMaker 2 for geo-replication between two Kafka clusters.

bootstrap.servers: Connects to the source Kafka cluster.

replication.policy.class: Defines the replication strategy used.

replication.factor: Number of replicas for mirrored topics in the target cluster.

topics: Lists topics to replicate.

source.cluster.alias and target.cluster.alias: Names for source and target clusters.

emit.heartbeats.enabled and heartbeat.interval.seconds: Enable and set frequency for heartbeat messages to monitor replication health.

sync.topic.acls.enabled: Ensures access control lists are synced between clusters.

Commands

Starts the MirrorMaker 2 tool using the configuration file to begin replicating topics from the source to the target Kafka cluster.

Terminal

connect-mirror-maker replicator.properties

Expected Output

INFO Starting MirrorMaker 2

INFO Connecting to source cluster clusterA.kafka.local:9092

INFO Connecting to target cluster clusterB.kafka.local:9092

INFO Replication started for topics: important-topic, logs

INFO Heartbeats enabled every 30 seconds

Lists all topics in the target cluster to verify that the topics have been replicated successfully.

Terminal

kafka-topics --bootstrap-server clusterB.kafka.local:9092 --list

Expected Output

important-topic

logs

→

--bootstrap-server - Specifies the Kafka cluster to connect to.

→

--list - Lists all topics in the cluster.

Checks the status of the MirrorMaker 2 consumer group on the target cluster to monitor replication lag and health.

Terminal

kafka-consumer-groups --bootstrap-server clusterB.kafka.local:9092 --describe --group mirror_maker_2

Expected Output

GROUP           TOPIC           PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG  CONSUMER-ID  HOST  CLIENT-ID

mirror_maker_2  important-topic 0          1500            1500            0    consumer-1   /0    mm2-client

mirror_maker_2  logs            0          3000            3000            0    consumer-1   /0    mm2-client

→

--describe - Shows detailed information about the consumer group.

→

--group - Specifies the consumer group to describe.

Key Concept

If you remember nothing else from geo-replication, remember: MirrorMaker 2 copies topics between Kafka clusters to keep data consistent and available across locations.

Common Mistakes

Not specifying the correct source and target cluster aliases in the configuration.

MirrorMaker 2 won't know which clusters to replicate between, causing replication to fail.

Always set source.cluster.alias and target.cluster.alias correctly to match your Kafka clusters.

Forgetting to include the topics list or using an incorrect topic name.

No topics or wrong topics will be replicated, so data won't appear in the target cluster.

Specify the exact topic names you want to replicate in the topics property.

Not monitoring the consumer group lag for MirrorMaker 2.

Replication might be delayed or stuck without your knowledge, risking data loss or inconsistency.

Regularly check the MirrorMaker 2 consumer group lag using kafka-consumer-groups command.

Summary

Create a MirrorMaker 2 configuration file to define source and target Kafka clusters and topics to replicate.

Start MirrorMaker 2 with the configuration to begin geo-replication.

Verify replicated topics exist on the target cluster and monitor replication health using consumer group status.

Practice

1. What is the main purpose of geo-replication in Kafka?

easy

A. To encrypt Kafka topics for security

B. To compress Kafka messages for faster transmission

C. To copy data between Kafka clusters in different locations for safety and availability

D. To delete old Kafka data automatically

Solution

1. Step 1: Understand geo-replication concept

   Geo-replication means copying data between clusters in different places to keep data safe and accessible.

2. Step 2: Identify the main goal

   The main goal is to keep data safe and close to users worldwide by copying it across locations.

3. Final Answer:

   To copy data between Kafka clusters in different locations for safety and availability -> Option C

4. Quick Check:

   Geo-replication = Data copying across locations [OK]

Hint: Geo-replication means copying data across locations [OK]

Common Mistakes:

• Confusing geo-replication with data compression
• Thinking geo-replication encrypts data
• Assuming geo-replication deletes data

2. Which Kafka tool is commonly used for geo-replication?

easy

A. MirrorMaker 2

B. Kafka Connect Sink

C. Kafka Streams

D. Kafka Producer API

Solution

1. Step 1: Recall Kafka geo-replication tools

   MirrorMaker 2 is designed to replicate data between Kafka clusters in different locations.

2. Step 2: Differentiate from other tools

   Kafka Streams processes data streams, Kafka Connect Sink moves data to external systems, and Producer API sends data to Kafka but does not replicate clusters.

3. Final Answer:

   MirrorMaker 2 -> Option A

4. Quick Check:

   Geo-replication tool = MirrorMaker 2 [OK]

Hint: MirrorMaker 2 is for replicating Kafka clusters [OK]

Common Mistakes:

• Choosing Kafka Streams which is for processing, not replication
• Confusing Kafka Connect Sink with replication tool
• Selecting Producer API which only sends data

3. Given the following MirrorMaker 2 configuration snippet, what is the effect?

{
  "clusters": {
    "A": {"bootstrap.servers": "a.kafka:9092"},
    "B": {"bootstrap.servers": "b.kafka:9092"}
  },
  "mirrors": [{"source.cluster": "A", "target.cluster": "B", "topics": ["orders"]}]
}

medium

A. Data from cluster B is copied to cluster A for topic 'orders'

B. Data from cluster A is copied to cluster B for topic 'orders'

C. Data from both clusters A and B are merged into a new topic 'orders'

D. No data is copied because topics list is empty

Solution

1. Step 1: Analyze source and target clusters

   The config shows source.cluster as A and target.cluster as B, meaning data flows from A to B.

2. Step 2: Check topics list

   The topics list includes "orders", so only the 'orders' topic is replicated from A to B.

3. Final Answer:

   Data from cluster A is copied to cluster B for topic 'orders' -> Option B

4. Quick Check:

   Source A to target B for 'orders' topic [OK]

Hint: Source cluster data replicates to target cluster [OK]

Common Mistakes:

• Reversing source and target clusters
• Thinking topics list is empty
• Assuming data merges instead of copies

4. You configured MirrorMaker 2 but no data is replicating. Which is the most likely error in this config?

{
  "clusters": {
    "source": {"bootstrap.servers": "source.kafka:9092"},
    "target": {"bootstrap.servers": "target.kafka:9092"}
  },
  "mirrors": [{"source.cluster": "source", "target.cluster": "target", "topics": []}]
}

medium

A. Bootstrap servers are incorrectly formatted

B. MirrorMaker 2 requires only one cluster in config

C. Cluster names must be numeric, not strings

D. Empty topics list means no topics are selected for replication

Solution

1. Step 1: Check topics list in mirror config

   The topics list is empty [], so no topics are selected to replicate.

2. Step 2: Validate other config parts

   Bootstrap servers are correctly formatted, cluster names as strings are valid, and multiple clusters are allowed.

3. Final Answer:

   Empty topics list means no topics are selected for replication -> Option D

4. Quick Check:

   Empty topics list = no replication [OK]

Hint: Empty topics list disables replication [OK]

Common Mistakes:

• Assuming bootstrap.servers format is wrong
• Thinking cluster names must be numbers
• Believing MirrorMaker 2 supports only one cluster

5. You want to set up geo-replication between three Kafka clusters (US, EU, Asia) so that each cluster has all topics from the others. Which strategy should you use?

hard

A. Set up MirrorMaker 2 with a full mesh replication where each cluster replicates to the other two

B. Use a single MirrorMaker 2 instance replicating only from US to EU and Asia

C. Configure MirrorMaker 2 to replicate only the largest topic across clusters

D. Use Kafka Connect Sink to move data between clusters

Solution

1. Step 1: Understand full data availability goal

   Each cluster should have all topics from the others, meaning data must flow both ways between all clusters.

2. Step 2: Choose replication strategy

   A full mesh replication with MirrorMaker 2 means each cluster replicates to the other two, ensuring complete data presence everywhere.

3. Final Answer:

   Set up MirrorMaker 2 with a full mesh replication where each cluster replicates to the other two -> Option A

4. Quick Check:

   Full mesh replication = all clusters have all topics [OK]

Hint: Full mesh replicates all topics between all clusters [OK]

Common Mistakes:

• Using one-way replication only
• Replicating only largest topic, missing others
• Confusing Kafka Connect Sink with cluster replication