MirrorMaker 2 concept in Kafka - Commands & Configuration

Introduction

When you run multiple Kafka clusters in different locations, you need a way to copy data between them automatically. MirrorMaker 2 helps by syncing topics from one Kafka cluster to another, so your data is available in both places without manual copying.

When you want to keep data in sync between a primary Kafka cluster and a backup cluster in another data center.

When you need to migrate data from an old Kafka cluster to a new one without downtime.

When you want to aggregate data from multiple Kafka clusters into a central cluster for analysis.

When you want to create a disaster recovery setup by replicating topics to a remote cluster.

When you want to balance load by mirroring topics to clusters closer to your users.

Config File - connect-mirror-maker.properties

connect-mirror-maker.properties

clusters = source, target

source.bootstrap.servers = source-kafka:9092
target.bootstrap.servers = target-kafka:9092

# Replication policy
replication.policy.class = org.apache.kafka.connect.mirror.DefaultReplicationPolicy

# Topics to replicate
topics = .*  # replicate all topics

# Consumer and producer configs
consumer.auto.offset.reset = earliest
producer.acks = all

# Enable heartbeats and offsets sync
emit.heartbeats.enabled = true
sync.topic.acls.enabled = true

# Group ID for MirrorMaker 2
group.id = mirror-maker-group

# Internal topics replication
internal.topics = __consumer_offsets,__transaction_state

# Number of tasks
tasks.max = 1

This file configures MirrorMaker 2 to connect two Kafka clusters named 'source' and 'target'.

clusters: lists the cluster aliases.

source.bootstrap.servers and target.bootstrap.servers: specify the Kafka servers for each cluster.

replication.policy.class: defines how topic names are mapped during replication.

topics: regex to select which topics to replicate (here all topics).

consumer.auto.offset.reset and producer.acks: set consumer and producer reliability.

emit.heartbeats.enabled and sync.topic.acls.enabled: enable internal syncing features.

group.id: consumer group for MirrorMaker 2 tasks.

internal.topics: internal Kafka topics to replicate for consistency.

tasks.max: number of parallel tasks to run.

Commands

This command starts MirrorMaker 2 using the configuration file to begin replicating topics from the source Kafka cluster to the target cluster.

Terminal

connect-mirror-maker.sh connect-mirror-maker.properties

Expected Output

INFO Starting MirrorMaker 2 with config connect-mirror-maker.properties

INFO MirrorMaker 2 started successfully

INFO Replicating topics: .*

This command lists all topics on the target Kafka cluster to verify that topics from the source cluster are being replicated.

Terminal

kafka-topics.sh --bootstrap-server target-kafka:9092 --list

Expected Output

my-topic-1

my-topic-2

__consumer_offsets

__transaction_state

→

--bootstrap-server - Specifies the Kafka server to connect to

→

--list - Lists all topics on the cluster

This command shows the consumer group status of MirrorMaker 2 on the target cluster to check replication progress and offsets.

Terminal

kafka-consumer-groups.sh --bootstrap-server target-kafka:9092 --describe --group mirror-maker-group

Expected Output

GROUP             TOPIC           PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG  CONSUMER-ID  HOST  CLIENT-ID

mirror-maker-group my-topic-1      0          100             100             0    consumer-1   /127.0.0.1  mm2-client

→

--bootstrap-server - Specifies the Kafka server to connect to

→

--describe - Shows detailed info about the consumer group

→

--group - Specifies the consumer group to describe

Key Concept

If you remember nothing else from this pattern, remember: MirrorMaker 2 automatically copies Kafka topics between clusters to keep data in sync without manual effort.

Common Mistakes

Not setting the correct bootstrap servers for source and target clusters in the config file.

MirrorMaker 2 cannot connect to the Kafka clusters and replication fails silently or with errors.

Double-check and set the correct 'source.bootstrap.servers' and 'target.bootstrap.servers' with reachable Kafka broker addresses.

Using an incorrect or too restrictive regex for the 'topics' setting, causing no topics to replicate.

MirrorMaker 2 will not replicate any topics if the regex does not match existing topic names.

Use '.*' to replicate all topics or a regex that matches the desired topic names.

Not verifying replication by listing topics or checking consumer group status on the target cluster.

You might think replication is working when it is not, leading to data loss or stale data.

Run 'kafka-topics.sh --list' and 'kafka-consumer-groups.sh --describe' on the target cluster to confirm replication.

Summary

Create a MirrorMaker 2 config file specifying source and target Kafka clusters and replication settings.

Start MirrorMaker 2 with the config file to begin replicating topics automatically.

Verify replication by listing topics and checking consumer group status on the target cluster.

Practice

1. What is the main purpose of Kafka MirrorMaker 2?

easy

A. To copy data between Kafka clusters for backup and availability

B. To create new Kafka topics automatically

C. To monitor Kafka cluster health

D. To compress Kafka messages for storage

Solution

1. Step 1: Understand MirrorMaker 2 role

   MirrorMaker 2 is designed to replicate data from one Kafka cluster to another to ensure data safety and availability.

2. Step 2: Compare options with MirrorMaker 2 function

   Only To copy data between Kafka clusters for backup and availability describes copying data between clusters, which matches MirrorMaker 2's purpose.

3. Final Answer:

   To copy data between Kafka clusters for backup and availability -> Option A

4. Quick Check:

   MirrorMaker 2 purpose = Data replication [OK]

Hint: MirrorMaker 2 copies data between clusters for safety [OK]

Common Mistakes:

• Confusing MirrorMaker 2 with monitoring tools
• Thinking it creates topics automatically
• Assuming it compresses messages

2. Which configuration is required to specify the source Kafka cluster in MirrorMaker 2?

easy

A. producer.bootstrap.servers

B. replica.bootstrap.servers

C. mirror.bootstrap.servers

D. consumer.bootstrap.servers

Solution

1. Step 1: Identify source cluster connection

   MirrorMaker 2 uses consumer configs to read from the source Kafka cluster.

2. Step 2: Match config key to source cluster

   The correct config key for source cluster is consumer.bootstrap.servers, which tells MirrorMaker 2 where to consume data from.

3. Final Answer:

   consumer.bootstrap.servers -> Option D

4. Quick Check:

   Source cluster = consumer.bootstrap.servers [OK]

Hint: Source cluster uses consumer.bootstrap.servers config [OK]

Common Mistakes:

• Using producer.bootstrap.servers for source cluster
• Confusing config keys with mirror.bootstrap.servers
• Assuming replica.bootstrap.servers is valid

3. Given this MirrorMaker 2 config snippet:

{
  "source.cluster.bootstrap.servers": "source:9092",
  "target.cluster.bootstrap.servers": "target:9092",
  "topics": "topicA,topicB",
  "num.streams": 3
}

How many parallel streams will MirrorMaker 2 use to copy data?

medium

A. 3

B. 2

C. 1

D. 4

Solution

1. Step 1: Locate num.streams setting

   The config sets "num.streams" to 3, which controls how many parallel streams MirrorMaker 2 uses.

2. Step 2: Understand effect of num.streams

   More streams mean more parallelism in copying data, so MirrorMaker 2 will use 3 streams here.

3. Final Answer:

   3 -> Option A

4. Quick Check:

   num.streams = 3 [OK]

Hint: Check num.streams value for parallelism count [OK]

Common Mistakes:

• Assuming default 1 stream if not read carefully
• Confusing topics count with streams count
• Ignoring num.streams config

4. You configured MirrorMaker 2 with:

topics = "topic1,topic2"
num.streams = 0

What is the problem with this configuration?

medium

A. topics must be a single topic, not multiple

B. num.streams cannot be zero; it must be at least 1

C. topics should be configured as a list, not a string

D. num.streams must be an even number

Solution

1. Step 1: Check num.streams value validity

   num.streams controls parallel streams and must be at least 1 to work properly.

2. Step 2: Identify error in config

   Setting num.streams to 0 disables streams, causing MirrorMaker 2 to fail or not copy data.

3. Final Answer:

   num.streams cannot be zero; it must be at least 1 -> Option B

4. Quick Check:

   num.streams ≥ 1 required [OK]

Hint: num.streams must be ≥ 1, zero disables copying [OK]

Common Mistakes:

• Thinking topics must be single only
• Assuming num.streams can be zero
• Believing num.streams must be even

5. You want MirrorMaker 2 to replicate only topics starting with "logs-" from source to target cluster. Which config setting achieves this?

hard

A. "topics": "logs-*"

B. "topics": "logs-"

C. "topics.regex": "logs-.*"

D. "topics.regex": "^logs$"

Solution

1. Step 1: Understand topic filtering in MirrorMaker 2

   MirrorMaker 2 supports filtering topics by regex using the "topics.regex" config.

2. Step 2: Match regex to desired pattern

   To match all topics starting with "logs-", the regex "logs-.*" correctly matches any topic beginning with "logs-".

3. Step 3: Compare options

   "topics": "logs-*" uses "topics" with wildcard which is not supported; "topics": "logs-" matches exact topic name; "topics.regex": "^logs$" matches only "logs" exactly.

4. Final Answer:

   "topics.regex": "logs-.*" -> Option C

5. Quick Check:

   Use topics.regex with proper regex for filtering [OK]

Hint: Use topics.regex with regex pattern to filter topics [OK]

Common Mistakes:

• Using wildcard * in topics instead of regex
• Confusing exact match with regex match
• Using incorrect regex that matches wrong topics