MirrorMaker 2 concept in Kafka - Mini Project: Build & Apply

Kafka MirrorMaker 2 Basic Setup

📖 Scenario: You work for a company that has two Kafka clusters in different data centers. You want to keep the data in these clusters synchronized automatically.Kafka MirrorMaker 2 is a tool that helps copy data between Kafka clusters, so they stay in sync.

🎯 Goal: Set up a simple MirrorMaker 2 configuration to replicate topics from a source Kafka cluster to a target Kafka cluster.

📋 What You'll Learn

Create a MirrorMaker 2 properties file with source and target cluster details

Configure replication of all topics

Start MirrorMaker 2 with the created configuration

Verify that data is being mirrored from source to target

💡 Why This Matters

🌍 Real World

Companies use MirrorMaker 2 to keep Kafka data synchronized across data centers for disaster recovery and data locality.

💼 Career

Kafka administrators and DevOps engineers often configure MirrorMaker 2 to ensure data availability and consistency across distributed systems.

1

Create MirrorMaker 2 configuration file

Create a file named mm2.properties with these exact lines:
clusters = source, target
source.bootstrap.servers = localhost:9092
target.bootstrap.servers = localhost:9093

Kafka

# Create mm2.properties file with cluster names and bootstrap servers
# Your code here

Hint

Define two clusters named source and target with their bootstrap servers.

2

Configure topic replication

Add these exact lines to mm2.properties to replicate all topics:
source->target.enabled = true
source->target.topics = .*

Kafka

clusters = source, target
source.bootstrap.servers = localhost:9092
target.bootstrap.servers = localhost:9093
# Enable replication from source to target for all topics
# Your code here

Hint

Enable replication and use .* to match all topics.

3

Start MirrorMaker 2

Run this exact command in your terminal to start MirrorMaker 2:
connect-mirror-maker mm2.properties

Kafka

clusters = source, target
source.bootstrap.servers = localhost:9092
target.bootstrap.servers = localhost:9093
source->target.enabled = true
source->target.topics = .*
# Run the MirrorMaker 2 start command below
# Your code here

Hint

Use the connect-mirror-maker command with the mm2.properties file.

4

Verify data replication

Write this exact command to check topics on the target cluster:
kafka-topics --bootstrap-server localhost:9093 --list

Kafka

clusters = source, target
source.bootstrap.servers = localhost:9092
target.bootstrap.servers = localhost:9093
source->target.enabled = true
source->target.topics = .*
connect-mirror-maker mm2.properties
# Check topics on target cluster
# Your code here

Hint

Use kafka-topics with the target cluster's bootstrap server to list topics.

Practice

1. What is the main purpose of Kafka MirrorMaker 2?

easy

A. To copy data between Kafka clusters for backup and availability

B. To create new Kafka topics automatically

C. To monitor Kafka cluster health

D. To compress Kafka messages for storage

Solution

1. Step 1: Understand MirrorMaker 2 role

   MirrorMaker 2 is designed to replicate data from one Kafka cluster to another to ensure data safety and availability.

2. Step 2: Compare options with MirrorMaker 2 function

   Only To copy data between Kafka clusters for backup and availability describes copying data between clusters, which matches MirrorMaker 2's purpose.

3. Final Answer:

   To copy data between Kafka clusters for backup and availability -> Option A

4. Quick Check:

   MirrorMaker 2 purpose = Data replication [OK]

Hint: MirrorMaker 2 copies data between clusters for safety [OK]

Common Mistakes:

• Confusing MirrorMaker 2 with monitoring tools
• Thinking it creates topics automatically
• Assuming it compresses messages

2. Which configuration is required to specify the source Kafka cluster in MirrorMaker 2?

easy

A. producer.bootstrap.servers

B. replica.bootstrap.servers

C. mirror.bootstrap.servers

D. consumer.bootstrap.servers

Solution

1. Step 1: Identify source cluster connection

   MirrorMaker 2 uses consumer configs to read from the source Kafka cluster.

2. Step 2: Match config key to source cluster

   The correct config key for source cluster is consumer.bootstrap.servers, which tells MirrorMaker 2 where to consume data from.

3. Final Answer:

   consumer.bootstrap.servers -> Option D

4. Quick Check:

   Source cluster = consumer.bootstrap.servers [OK]

Hint: Source cluster uses consumer.bootstrap.servers config [OK]

Common Mistakes:

• Using producer.bootstrap.servers for source cluster
• Confusing config keys with mirror.bootstrap.servers
• Assuming replica.bootstrap.servers is valid

3. Given this MirrorMaker 2 config snippet:

{
  "source.cluster.bootstrap.servers": "source:9092",
  "target.cluster.bootstrap.servers": "target:9092",
  "topics": "topicA,topicB",
  "num.streams": 3
}

How many parallel streams will MirrorMaker 2 use to copy data?

medium

A. 3

B. 2

C. 1

D. 4

Solution

1. Step 1: Locate num.streams setting

   The config sets "num.streams" to 3, which controls how many parallel streams MirrorMaker 2 uses.

2. Step 2: Understand effect of num.streams

   More streams mean more parallelism in copying data, so MirrorMaker 2 will use 3 streams here.

3. Final Answer:

   3 -> Option A

4. Quick Check:

   num.streams = 3 [OK]

Hint: Check num.streams value for parallelism count [OK]

Common Mistakes:

• Assuming default 1 stream if not read carefully
• Confusing topics count with streams count
• Ignoring num.streams config

4. You configured MirrorMaker 2 with:

topics = "topic1,topic2"
num.streams = 0

What is the problem with this configuration?

medium

A. topics must be a single topic, not multiple

B. num.streams cannot be zero; it must be at least 1

C. topics should be configured as a list, not a string

D. num.streams must be an even number

Solution

1. Step 1: Check num.streams value validity

   num.streams controls parallel streams and must be at least 1 to work properly.

2. Step 2: Identify error in config

   Setting num.streams to 0 disables streams, causing MirrorMaker 2 to fail or not copy data.

3. Final Answer:

   num.streams cannot be zero; it must be at least 1 -> Option B

4. Quick Check:

   num.streams ≥ 1 required [OK]

Hint: num.streams must be ≥ 1, zero disables copying [OK]

Common Mistakes:

• Thinking topics must be single only
• Assuming num.streams can be zero
• Believing num.streams must be even

5. You want MirrorMaker 2 to replicate only topics starting with "logs-" from source to target cluster. Which config setting achieves this?

hard

A. "topics": "logs-*"

B. "topics": "logs-"

C. "topics.regex": "logs-.*"

D. "topics.regex": "^logs$"

Solution

1. Step 1: Understand topic filtering in MirrorMaker 2

   MirrorMaker 2 supports filtering topics by regex using the "topics.regex" config.

2. Step 2: Match regex to desired pattern

   To match all topics starting with "logs-", the regex "logs-.*" correctly matches any topic beginning with "logs-".

3. Step 3: Compare options

   "topics": "logs-*" uses "topics" with wildcard which is not supported; "topics": "logs-" matches exact topic name; "topics.regex": "^logs$" matches only "logs" exactly.

4. Final Answer:

   "topics.regex": "logs-.*" -> Option C

5. Quick Check:

   Use topics.regex with proper regex for filtering [OK]

Hint: Use topics.regex with regex pattern to filter topics [OK]

Common Mistakes:

• Using wildcard * in topics instead of regex
• Confusing exact match with regex match
• Using incorrect regex that matches wrong topics