Cross-datacenter replication in Kafka - Commands & Configuration

Introduction

When you have Kafka clusters in different data centers, you want to copy data between them automatically. Cross-datacenter replication helps keep data in sync across locations, so apps can read and write data locally but still share it globally.

When you want to keep Kafka topics synchronized between two data centers for disaster recovery.

When you have users in different regions and want to reduce latency by replicating data closer to them.

When you want to back up Kafka data from one cluster to another in a different physical location.

When you need to migrate data gradually from one Kafka cluster to another without downtime.

When you want to aggregate data from multiple Kafka clusters into a central cluster for analytics.

Config File - replicator.properties

replicator.properties

bootstrap.servers=source-kafka1:9092,source-kafka2:9092
replication.factor=3
group.id=replicator-group
key.converter=org.apache.kafka.connect.converters.ByteArrayConverter
value.converter=org.apache.kafka.connect.converters.ByteArrayConverter
config.storage.topic=replicator-configs
offset.storage.topic=replicator-offsets
status.storage.topic=replicator-status

# Destination cluster bootstrap servers
producer.bootstrap.servers=dest-kafka1:9092,dest-kafka2:9092

# Topics to replicate
topics=important-topic,logs-topic

# Replication policy
replication.policy.class=org.apache.kafka.connect.mirror.DefaultReplicationPolicy

# Enable replication
tasks.max=1

This configuration file sets up Kafka MirrorMaker 2 for cross-datacenter replication.

bootstrap.servers: Connects to the source Kafka cluster.

producer.bootstrap.servers: Connects to the destination Kafka cluster.

topics: Lists the topics to replicate.

replication.policy.class: Defines how topic names are mapped between clusters.

tasks.max: Number of parallel replication tasks.

Commands

Starts the Kafka MirrorMaker 2 process using the configuration file to begin replicating topics from the source to the destination cluster.

Terminal

connect-mirror-maker replicator.properties

Expected Output

INFO Starting MirrorMaker 2

INFO Connecting to source cluster at source-kafka1:9092,source-kafka2:9092

INFO Connecting to destination cluster at dest-kafka1:9092,dest-kafka2:9092

INFO Replicating topics: important-topic, logs-topic

INFO MirrorMaker 2 started successfully

Lists all topics on the destination Kafka cluster to verify that the topics have been replicated.

Terminal

kafka-topics --bootstrap-server dest-kafka1:9092 --list

Expected Output

important-topic

logs-topic

replicator-configs

replicator-offsets

replicator-status

→

--bootstrap-server - Specifies the Kafka cluster to connect to

→

--list - Lists all topics in the cluster

Shows the status of the replication consumer group on the destination cluster to check replication progress and offsets.

Terminal

kafka-consumer-groups --bootstrap-server dest-kafka1:9092 --describe --group replicator-group

Expected Output

GROUP           TOPIC           PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG  CONSUMER-ID  HOST  CLIENT-ID

replicator-group important-topic 0          12345           12345           0    consumer-1   /127.0.0.1  replicator-1

→

--bootstrap-server - Connects to the Kafka cluster

→

--describe - Shows detailed information about the consumer group

→

--group - Specifies the consumer group to describe

Key Concept

If you remember nothing else from this pattern, remember: Kafka MirrorMaker 2 uses a connector configuration to continuously copy topics between clusters, keeping data in sync across data centers.

Common Mistakes

Not specifying the destination cluster bootstrap servers in the configuration.

Without the destination cluster info, MirrorMaker cannot send data to the target cluster, so replication fails.

Always include producer.bootstrap.servers with the destination Kafka brokers in the config file.

Trying to replicate topics that do not exist on the source cluster.

MirrorMaker cannot replicate non-existent topics, so no data will be copied.

Verify the topics exist on the source cluster before adding them to the topics list.

Running MirrorMaker without proper permissions on source or destination clusters.

Lack of permissions causes connection or write failures, stopping replication.

Ensure the user running MirrorMaker has read access on source and write access on destination clusters.

Summary

Create a MirrorMaker 2 configuration file specifying source and destination Kafka clusters and topics to replicate.

Start MirrorMaker 2 with the configuration to begin cross-datacenter replication.

Verify replication by listing topics and checking consumer group status on the destination cluster.

Practice

1.

What is the main purpose of cross-datacenter replication in Kafka?

easy

A. To copy Kafka topics between clusters for data safety and availability

B. To compress Kafka messages for faster transmission

C. To encrypt Kafka topics for security

D. To delete old Kafka topics automatically

Solution

1. Step 1: Understand the concept of cross-datacenter replication

   Cross-datacenter replication means copying data from one Kafka cluster to another in a different location.

2. Step 2: Identify the main goal of this replication

   The goal is to keep data safe and available even if one cluster fails.

3. Final Answer:

   To copy Kafka topics between clusters for data safety and availability -> Option A

4. Quick Check:

   Replication = Data safety and availability [OK]

Hint: Replication means copying data across clusters for safety [OK]

Common Mistakes:

• Confusing replication with compression
• Thinking replication encrypts data
• Assuming replication deletes data

2.

Which of the following is the correct way to specify the source cluster's bootstrap servers in Kafka MirrorMaker configuration?

consumer.bootstrap.servers = ?

easy

A. "source.kafka.cluster:9092"

B. "target.kafka.cluster:9092"

C. "localhost:2181"

D. "mirror.kafka.cluster:2181"

Solution

1. Step 1: Identify the role of consumer.bootstrap.servers

   This property tells MirrorMaker where to read data from, so it must point to the source Kafka cluster.

2. Step 2: Choose the correct server address for the source cluster

   The source cluster's Kafka broker address is usually in the form host:port, like "source.kafka.cluster:9092".

3. Final Answer:

   "source.kafka.cluster:9092" -> Option A

4. Quick Check:

   consumer.bootstrap.servers = source cluster address [OK]

Hint: Consumer connects to source cluster bootstrap servers [OK]

Common Mistakes:

• Using target cluster address for consumer.bootstrap.servers
• Using ZooKeeper address instead of Kafka broker
• Confusing consumer and producer properties

3.

Given the following MirrorMaker configuration snippet, what will be the value of producer.bootstrap.servers?

consumer.bootstrap.servers=source1.kafka:9092,source2.kafka:9092
producer.bootstrap.servers=target.kafka:9092

medium

A. "source1.kafka:9092,source2.kafka:9092"

B. "target.kafka:9092"

C. "source1.kafka:9092"

D. "localhost:9092"

Solution

1. Step 1: Read the configuration lines carefully

   The snippet shows two properties: consumer.bootstrap.servers and producer.bootstrap.servers with their values.

2. Step 2: Identify the value assigned to producer.bootstrap.servers

   It is explicitly set to "target.kafka:9092" in the second line.

3. Final Answer:

   "target.kafka:9092" -> Option B

4. Quick Check:

   producer.bootstrap.servers = target cluster address [OK]

Hint: Producer connects to target cluster bootstrap servers [OK]

Common Mistakes:

• Mixing consumer and producer server addresses
• Assuming producer uses source cluster address
• Ignoring explicit configuration values

4.

Identify the error in this MirrorMaker consumer configuration:

consumer.bootstrap.servers=target.kafka:9092
consumer.group.id=mirror-group

medium

A. consumer.bootstrap.servers should use port 2181

B. consumer.group.id is missing

C. consumer.bootstrap.servers should point to the source cluster, not target

D. consumer.group.id should be unique per topic

Solution

1. Step 1: Understand the role of consumer.bootstrap.servers

   This property must point to the source Kafka cluster to read data for replication.

2. Step 2: Check the given value

   It incorrectly points to "target.kafka:9092", which is the destination cluster, not the source.

3. Final Answer:

   consumer.bootstrap.servers should point to the source cluster, not target -> Option C

4. Quick Check:

   Consumer connects to source cluster [OK]

Hint: Consumer must connect to source cluster, not target [OK]

Common Mistakes:

• Using target cluster address for consumer.bootstrap.servers
• Confusing ZooKeeper port with Kafka broker port
• Assuming group.id is missing when it is present

5.

You want to replicate only topics starting with logs- from the source cluster to the target cluster using MirrorMaker. Which configuration property should you set to filter these topics?

hard

A. consumer.topic.filter=logs-*

B. producer.topic.filter=logs-*

C. mirror.topic.prefix=logs-

D. consumer.topic.whitelist=logs-*

Solution

1. Step 1: Identify the property to filter topics in MirrorMaker

   MirrorMaker uses consumer.topic.whitelist to specify which topics to consume from the source cluster.

2. Step 2: Match the pattern for topics starting with 'logs-'

   The pattern "logs-*" matches all topics starting with "logs-".

3. Final Answer:

   consumer.topic.whitelist=logs-* -> Option D

4. Quick Check:

   Use consumer.topic.whitelist to filter topics [OK]

Hint: Use consumer.topic.whitelist to select topics by pattern [OK]

Common Mistakes:

• Using producer.topic.filter which does not exist
• Confusing mirror.topic.prefix with filtering
• Using consumer.topic.filter instead of whitelist