MirrorMaker 2 concept in Kafka - Commands & Configuration

Introduction

When you run multiple Kafka clusters in different locations, you need a way to copy data between them automatically. MirrorMaker 2 helps by syncing topics from one Kafka cluster to another, so your data is available in both places without manual copying.

When you want to keep data in sync between a primary Kafka cluster and a backup cluster in another data center.

When you need to migrate data from an old Kafka cluster to a new one without downtime.

When you want to aggregate data from multiple Kafka clusters into a central cluster for analysis.

When you want to create a disaster recovery setup by replicating topics to a remote cluster.

When you want to balance load by mirroring topics to clusters closer to your users.

Config File - connect-mirror-maker.properties

connect-mirror-maker.properties

clusters = source, target

source.bootstrap.servers = source-kafka:9092
target.bootstrap.servers = target-kafka:9092

# Replication policy
replication.policy.class = org.apache.kafka.connect.mirror.DefaultReplicationPolicy

# Topics to replicate
topics = .*  # replicate all topics

# Consumer and producer configs
consumer.auto.offset.reset = earliest
producer.acks = all

# Enable heartbeats and offsets sync
emit.heartbeats.enabled = true
sync.topic.acls.enabled = true

# Group ID for MirrorMaker 2
group.id = mirror-maker-group

# Internal topics replication
internal.topics = __consumer_offsets,__transaction_state

# Number of tasks
tasks.max = 1

This file configures MirrorMaker 2 to connect two Kafka clusters named 'source' and 'target'.

clusters: lists the cluster aliases.

source.bootstrap.servers and target.bootstrap.servers: specify the Kafka servers for each cluster.

replication.policy.class: defines how topic names are mapped during replication.

topics: regex to select which topics to replicate (here all topics).

consumer.auto.offset.reset and producer.acks: set consumer and producer reliability.

emit.heartbeats.enabled and sync.topic.acls.enabled: enable internal syncing features.

group.id: consumer group for MirrorMaker 2 tasks.

internal.topics: internal Kafka topics to replicate for consistency.

tasks.max: number of parallel tasks to run.

Commands

This command starts MirrorMaker 2 using the configuration file to begin replicating topics from the source Kafka cluster to the target cluster.

Terminal

connect-mirror-maker.sh connect-mirror-maker.properties

Expected Output

INFO Starting MirrorMaker 2 with config connect-mirror-maker.properties

INFO MirrorMaker 2 started successfully

INFO Replicating topics: .*

This command lists all topics on the target Kafka cluster to verify that topics from the source cluster are being replicated.

Terminal

kafka-topics.sh --bootstrap-server target-kafka:9092 --list

Expected Output

my-topic-1

my-topic-2

__consumer_offsets

__transaction_state

→

--bootstrap-server - Specifies the Kafka server to connect to

→

--list - Lists all topics on the cluster

This command shows the consumer group status of MirrorMaker 2 on the target cluster to check replication progress and offsets.

Terminal

kafka-consumer-groups.sh --bootstrap-server target-kafka:9092 --describe --group mirror-maker-group

Expected Output

GROUP             TOPIC           PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG  CONSUMER-ID  HOST  CLIENT-ID

mirror-maker-group my-topic-1      0          100             100             0    consumer-1   /127.0.0.1  mm2-client

→

--bootstrap-server - Specifies the Kafka server to connect to

→

--describe - Shows detailed info about the consumer group

→

--group - Specifies the consumer group to describe

Key Concept

If you remember nothing else from this pattern, remember: MirrorMaker 2 automatically copies Kafka topics between clusters to keep data in sync without manual effort.

Common Mistakes

Not setting the correct bootstrap servers for source and target clusters in the config file.

MirrorMaker 2 cannot connect to the Kafka clusters and replication fails silently or with errors.

Double-check and set the correct 'source.bootstrap.servers' and 'target.bootstrap.servers' with reachable Kafka broker addresses.

Using an incorrect or too restrictive regex for the 'topics' setting, causing no topics to replicate.

MirrorMaker 2 will not replicate any topics if the regex does not match existing topic names.

Use '.*' to replicate all topics or a regex that matches the desired topic names.

Not verifying replication by listing topics or checking consumer group status on the target cluster.

You might think replication is working when it is not, leading to data loss or stale data.

Run 'kafka-topics.sh --list' and 'kafka-consumer-groups.sh --describe' on the target cluster to confirm replication.

Summary

Create a MirrorMaker 2 config file specifying source and target Kafka clusters and replication settings.

Start MirrorMaker 2 with the config file to begin replicating topics automatically.

Verify replication by listing topics and checking consumer group status on the target cluster.