MirrorMaker 2 concept in Kafka - Step-by-Step Execution

Process Flow - MirrorMaker 2 concept

Start MirrorMaker 2

↓

Connect to Source Cluster

↓

Read Topics from Source

↓

Apply Filters and Transformations

↓

Produce to Target Cluster

↓

Monitor Replication Status

↓

Handle Failures and Retries

↓

End

MirrorMaker 2 connects two Kafka clusters, reads topics from the source, optionally transforms data, and writes it to the target cluster while monitoring and handling errors.

Execution Sample

Kafka

connect source_cluster
read topic 'orders'
filter messages with key 'user'
produce to target_cluster
monitor replication
handle errors

This sequence shows MirrorMaker 2 connecting to a source cluster, reading a topic, filtering messages, sending them to a target cluster, and monitoring the process.

Process Table

Step	Action	Details	Result
1	Connect to source cluster	Connect to kafka-source.example.com	Connection established
2	Read topic	Read messages from 'orders' topic	Messages fetched
3	Filter messages	Keep messages with key 'user'	Filtered messages ready
4	Produce to target cluster	Send filtered messages to kafka-target.example.com	Messages produced
5	Monitor replication	Check offsets and lag	Replication healthy
6	Handle errors	Retry on failure	Errors handled or logged
7	End	Replication complete or stopped	Process ends

💡 Replication ends when stopped manually or on unrecoverable error

Status Tracker

Variable	Start	After Step 2	After Step 3	After Step 4	After Step 5	Final
connection_status	disconnected	connected	connected	connected	connected	disconnected
messages_fetched	0	1000	1000	1000	1000	0
messages_filtered	0	0	600	600	600	0
messages_produced	0	0	0	600	600	600
replication_health	unknown	unknown	unknown	unknown	healthy	stopped

Key Moments - 3 Insights

Why do we filter messages after reading them from the source?

Filtering happens after reading to select only relevant messages for replication, as shown in step 3 of the execution_table where messages are filtered by key.

What happens if the connection to the target cluster fails during production?

MirrorMaker 2 retries sending messages and handles errors as shown in step 6, ensuring replication continues or logs failures.

When does the replication process stop?

Replication stops manually or on unrecoverable errors as noted in the exit_note and step 7 of the execution_table.

Visual Quiz - 3 Questions

Look at the execution_table, what is the number of messages filtered after step 3?

A 600

B 1000

C 0

D 400

💡 Hint

Check the 'messages_filtered' variable in variable_tracker after step 3

At which step does MirrorMaker 2 check the replication health?

A Step 4

B Step 2

C Step 5

D Step 6

💡 Hint

Look at the 'Monitor replication' action in the execution_table

If the connection to the source cluster fails at step 1, what happens next?

A Replication proceeds without source connection

B Process cannot continue and stops

C Messages are filtered anyway

D Messages are produced to target cluster

💡 Hint

Refer to step 1 in execution_table where connection is established; failure means no further steps

Concept Snapshot

MirrorMaker 2 replicates Kafka topics between clusters.
It connects to source, reads topics, filters or transforms messages.
Then it produces messages to the target cluster.
It monitors replication health and handles errors.
Replication stops manually or on unrecoverable errors.

Full Transcript

MirrorMaker 2 is a tool that copies data from one Kafka cluster to another. It starts by connecting to the source cluster, then reads messages from specified topics. After reading, it can filter or transform messages before sending them to the target cluster. During this process, it monitors the replication status to ensure data is copied correctly. If errors happen, it retries or logs them. The process continues until stopped manually or if a serious error occurs.

Practice

1. What is the main purpose of Kafka MirrorMaker 2?

easy

A. To copy data between Kafka clusters for backup and availability

B. To create new Kafka topics automatically

C. To monitor Kafka cluster health

D. To compress Kafka messages for storage

Solution

1. Step 1: Understand MirrorMaker 2 role

   MirrorMaker 2 is designed to replicate data from one Kafka cluster to another to ensure data safety and availability.

2. Step 2: Compare options with MirrorMaker 2 function

   Only To copy data between Kafka clusters for backup and availability describes copying data between clusters, which matches MirrorMaker 2's purpose.

3. Final Answer:

   To copy data between Kafka clusters for backup and availability -> Option A

4. Quick Check:

   MirrorMaker 2 purpose = Data replication [OK]

Hint: MirrorMaker 2 copies data between clusters for safety [OK]

Common Mistakes:

• Confusing MirrorMaker 2 with monitoring tools
• Thinking it creates topics automatically
• Assuming it compresses messages

2. Which configuration is required to specify the source Kafka cluster in MirrorMaker 2?

easy

A. producer.bootstrap.servers

B. replica.bootstrap.servers

C. mirror.bootstrap.servers

D. consumer.bootstrap.servers

Solution

1. Step 1: Identify source cluster connection

   MirrorMaker 2 uses consumer configs to read from the source Kafka cluster.

2. Step 2: Match config key to source cluster

   The correct config key for source cluster is consumer.bootstrap.servers, which tells MirrorMaker 2 where to consume data from.

3. Final Answer:

   consumer.bootstrap.servers -> Option D

4. Quick Check:

   Source cluster = consumer.bootstrap.servers [OK]

Hint: Source cluster uses consumer.bootstrap.servers config [OK]

Common Mistakes:

• Using producer.bootstrap.servers for source cluster
• Confusing config keys with mirror.bootstrap.servers
• Assuming replica.bootstrap.servers is valid

3. Given this MirrorMaker 2 config snippet:

{
  "source.cluster.bootstrap.servers": "source:9092",
  "target.cluster.bootstrap.servers": "target:9092",
  "topics": "topicA,topicB",
  "num.streams": 3
}

How many parallel streams will MirrorMaker 2 use to copy data?

medium

A. 3

B. 2

C. 1

D. 4

Solution

1. Step 1: Locate num.streams setting

   The config sets "num.streams" to 3, which controls how many parallel streams MirrorMaker 2 uses.

2. Step 2: Understand effect of num.streams

   More streams mean more parallelism in copying data, so MirrorMaker 2 will use 3 streams here.

3. Final Answer:

   3 -> Option A

4. Quick Check:

   num.streams = 3 [OK]

Hint: Check num.streams value for parallelism count [OK]

Common Mistakes:

• Assuming default 1 stream if not read carefully
• Confusing topics count with streams count
• Ignoring num.streams config

4. You configured MirrorMaker 2 with:

topics = "topic1,topic2"
num.streams = 0

What is the problem with this configuration?

medium

A. topics must be a single topic, not multiple

B. num.streams cannot be zero; it must be at least 1

C. topics should be configured as a list, not a string

D. num.streams must be an even number

Solution

1. Step 1: Check num.streams value validity

   num.streams controls parallel streams and must be at least 1 to work properly.

2. Step 2: Identify error in config

   Setting num.streams to 0 disables streams, causing MirrorMaker 2 to fail or not copy data.

3. Final Answer:

   num.streams cannot be zero; it must be at least 1 -> Option B

4. Quick Check:

   num.streams ≥ 1 required [OK]

Hint: num.streams must be ≥ 1, zero disables copying [OK]

Common Mistakes:

• Thinking topics must be single only
• Assuming num.streams can be zero
• Believing num.streams must be even

5. You want MirrorMaker 2 to replicate only topics starting with "logs-" from source to target cluster. Which config setting achieves this?

hard

A. "topics": "logs-*"

B. "topics": "logs-"

C. "topics.regex": "logs-.*"

D. "topics.regex": "^logs$"

Solution

1. Step 1: Understand topic filtering in MirrorMaker 2

   MirrorMaker 2 supports filtering topics by regex using the "topics.regex" config.

2. Step 2: Match regex to desired pattern

   To match all topics starting with "logs-", the regex "logs-.*" correctly matches any topic beginning with "logs-".

3. Step 3: Compare options

   "topics": "logs-*" uses "topics" with wildcard which is not supported; "topics": "logs-" matches exact topic name; "topics.regex": "^logs$" matches only "logs" exactly.

4. Final Answer:

   "topics.regex": "logs-.*" -> Option C

5. Quick Check:

   Use topics.regex with proper regex for filtering [OK]

Hint: Use topics.regex with regex pattern to filter topics [OK]

Common Mistakes:

• Using wildcard * in topics instead of regex
• Confusing exact match with regex match
• Using incorrect regex that matches wrong topics