Cross-datacenter replication in Kafka - Step-by-Step Execution

Process Flow - Cross-datacenter replication

Produce message in DC1

↓

Message stored in DC1 Kafka cluster

↓

Replication process reads message

↓

Send message over network to DC2

↓

Message stored in DC2 Kafka cluster

↓

Consumer reads message from DC2

Messages produced in one data center are stored locally, then replicated over the network to another data center's Kafka cluster for consumption.

Execution Sample

Kafka

producer.send('topic', 'message1')
# Message stored in DC1
replicator.replicate('topic')
# Message sent to DC2
consumer.read('topic')
# Consumer reads replicated message

This code simulates producing a message in one data center, replicating it to another, and consuming it there.

Process Table

Step	Action	Source DC	Message State	Replication Status	Consumer Output
1	Produce message 'message1'	DC1	Stored in DC1	Not started	None
2	Start replication	DC1	Stored in DC1	Sending to DC2	None
3	Message received in DC2	DC2	Stored in DC2	Completed	None
4	Consumer reads message	DC2	Stored in DC2	Completed	'message1' read
5	No new messages	DC1 & DC2	No change	Idle	No output

💡 Replication completes after message is stored in DC2 and consumed.

Status Tracker

Variable	Start	After Step 1	After Step 2	After Step 3	After Step 4	Final
message_storage_DC1	empty	message1	message1	message1	message1	message1
message_storage_DC2	empty	empty	empty	message1	message1	message1
replication_status	idle	idle	sending	completed	completed	idle
consumer_output	none	none	none	none	message1 read	none

Key Moments - 3 Insights

Why does the message appear in DC2 only after replication starts?

Because the message is first stored locally in DC1 (see Step 1), and only after replication begins (Step 2) is it sent and stored in DC2 (Step 3).

Can the consumer read the message before replication completes?

No, the consumer in DC2 reads the message only after it is stored there (Step 4), so before replication completes, there is no message to read.

What happens if no new messages are produced?

Replication stays idle (Step 5), and consumers have no new messages to read, so no output occurs.

Visual Quiz - 3 Questions

Look at the execution table, what is the replication status at Step 3?

A Completed

B Sending

C Idle

D Not started

💡 Hint

Check the 'Replication Status' column at Step 3 in the execution_table.

At which step does the consumer read the replicated message?

A Step 2

B Step 3

C Step 4

D Step 5

💡 Hint

Look at the 'Consumer Output' column in the execution_table to find when 'message1 read' appears.

If the producer sends a second message before replication starts, what changes in the variable_tracker?

A message_storage_DC2 will have two messages immediately

B message_storage_DC1 will have two messages after Step 1

C replication_status will be 'completed' at Step 1

D consumer_output will show both messages read at Step 2

💡 Hint

Check how message_storage_DC1 changes after Step 1 in variable_tracker.

Concept Snapshot

Cross-datacenter replication in Kafka:
- Messages produced in one data center (DC1) are stored locally.
- A replication process sends messages over the network to another data center (DC2).
- Messages are stored in DC2's Kafka cluster.
- Consumers in DC2 read replicated messages.
- Replication ensures data availability across locations.

Full Transcript

Cross-datacenter replication means sending Kafka messages from one data center to another. First, a message is produced and stored in the first data center's Kafka cluster. Then, a replication process reads this message and sends it over the network to the second data center. The message is stored there, and consumers in the second data center can read it. This process ensures that data is available in multiple locations for reliability and faster access. The execution table shows each step: producing, replicating, storing, and consuming. Variables track message storage and replication status. Key moments clarify when messages appear and can be read. The quiz tests understanding of replication status and message flow.

Practice

1.

What is the main purpose of cross-datacenter replication in Kafka?

easy

A. To copy Kafka topics between clusters for data safety and availability

B. To compress Kafka messages for faster transmission

C. To encrypt Kafka topics for security

D. To delete old Kafka topics automatically

Solution

1. Step 1: Understand the concept of cross-datacenter replication

   Cross-datacenter replication means copying data from one Kafka cluster to another in a different location.

2. Step 2: Identify the main goal of this replication

   The goal is to keep data safe and available even if one cluster fails.

3. Final Answer:

   To copy Kafka topics between clusters for data safety and availability -> Option A

4. Quick Check:

   Replication = Data safety and availability [OK]

Hint: Replication means copying data across clusters for safety [OK]

Common Mistakes:

• Confusing replication with compression
• Thinking replication encrypts data
• Assuming replication deletes data

2.

Which of the following is the correct way to specify the source cluster's bootstrap servers in Kafka MirrorMaker configuration?

consumer.bootstrap.servers = ?

easy

A. "source.kafka.cluster:9092"

B. "target.kafka.cluster:9092"

C. "localhost:2181"

D. "mirror.kafka.cluster:2181"

Solution

1. Step 1: Identify the role of consumer.bootstrap.servers

   This property tells MirrorMaker where to read data from, so it must point to the source Kafka cluster.

2. Step 2: Choose the correct server address for the source cluster

   The source cluster's Kafka broker address is usually in the form host:port, like "source.kafka.cluster:9092".

3. Final Answer:

   "source.kafka.cluster:9092" -> Option A

4. Quick Check:

   consumer.bootstrap.servers = source cluster address [OK]

Hint: Consumer connects to source cluster bootstrap servers [OK]

Common Mistakes:

• Using target cluster address for consumer.bootstrap.servers
• Using ZooKeeper address instead of Kafka broker
• Confusing consumer and producer properties

3.

Given the following MirrorMaker configuration snippet, what will be the value of producer.bootstrap.servers?

consumer.bootstrap.servers=source1.kafka:9092,source2.kafka:9092
producer.bootstrap.servers=target.kafka:9092

medium

A. "source1.kafka:9092,source2.kafka:9092"

B. "target.kafka:9092"

C. "source1.kafka:9092"

D. "localhost:9092"

Solution

1. Step 1: Read the configuration lines carefully

   The snippet shows two properties: consumer.bootstrap.servers and producer.bootstrap.servers with their values.

2. Step 2: Identify the value assigned to producer.bootstrap.servers

   It is explicitly set to "target.kafka:9092" in the second line.

3. Final Answer:

   "target.kafka:9092" -> Option B

4. Quick Check:

   producer.bootstrap.servers = target cluster address [OK]

Hint: Producer connects to target cluster bootstrap servers [OK]

Common Mistakes:

• Mixing consumer and producer server addresses
• Assuming producer uses source cluster address
• Ignoring explicit configuration values

4.

Identify the error in this MirrorMaker consumer configuration:

consumer.bootstrap.servers=target.kafka:9092
consumer.group.id=mirror-group

medium

A. consumer.bootstrap.servers should use port 2181

B. consumer.group.id is missing

C. consumer.bootstrap.servers should point to the source cluster, not target

D. consumer.group.id should be unique per topic

Solution

1. Step 1: Understand the role of consumer.bootstrap.servers

   This property must point to the source Kafka cluster to read data for replication.

2. Step 2: Check the given value

   It incorrectly points to "target.kafka:9092", which is the destination cluster, not the source.

3. Final Answer:

   consumer.bootstrap.servers should point to the source cluster, not target -> Option C

4. Quick Check:

   Consumer connects to source cluster [OK]

Hint: Consumer must connect to source cluster, not target [OK]

Common Mistakes:

• Using target cluster address for consumer.bootstrap.servers
• Confusing ZooKeeper port with Kafka broker port
• Assuming group.id is missing when it is present

5.

You want to replicate only topics starting with logs- from the source cluster to the target cluster using MirrorMaker. Which configuration property should you set to filter these topics?

hard

A. consumer.topic.filter=logs-*

B. producer.topic.filter=logs-*

C. mirror.topic.prefix=logs-

D. consumer.topic.whitelist=logs-*

Solution

1. Step 1: Identify the property to filter topics in MirrorMaker

   MirrorMaker uses consumer.topic.whitelist to specify which topics to consume from the source cluster.

2. Step 2: Match the pattern for topics starting with 'logs-'

   The pattern "logs-*" matches all topics starting with "logs-".

3. Final Answer:

   consumer.topic.whitelist=logs-* -> Option D

4. Quick Check:

   Use consumer.topic.whitelist to filter topics [OK]

Hint: Use consumer.topic.whitelist to select topics by pattern [OK]

Common Mistakes:

• Using producer.topic.filter which does not exist
• Confusing mirror.topic.prefix with filtering
• Using consumer.topic.filter instead of whitelist