Geo-replication strategies in Kafka - Step-by-Step Execution

Process Flow - Geo-replication strategies

Start: Data produced in Region A

↓

Local Kafka Cluster in Region A

↓

Replication Setup

↓

Replicate to Region B

↓

Kafka Cluster B

↓

Consumers read locally in each region

↓

End

Data is produced in one region and replicated asynchronously to Kafka clusters in other regions, allowing local consumers to read data with low latency.

Execution Sample

Kafka

producer.send(topic, data)
// Data stored in Region A cluster
replicator.replicate(topic, regionB)
replicator.replicate(topic, regionC)
consumerB.consume(topic)
consumerC.consume(topic)

Shows data produced in one region, replicated to other regions, and consumed locally.

Process Table

Step	Action	Source Region	Target Region	Result
1	Produce data	Region A	Region A	Data stored in Region A cluster
2	Replicate data	Region A	Region B	Data copied to Region B cluster
3	Replicate data	Region A	Region C	Data copied to Region C cluster
4	Consume data	Region B	Region B	Consumer reads data locally in Region B
5	Consume data	Region C	Region C	Consumer reads data locally in Region C
6	End	-	-	Replication and consumption complete

💡 All data replicated and consumed in target regions

Status Tracker

Variable	Start	After Step 1	After Step 2	After Step 3	After Step 4	After Step 5	Final
Data in Region A cluster	empty	data1	data1	data1	data1	data1	data1
Data in Region B cluster	empty	empty	data1	data1	data1	data1	data1
Data in Region C cluster	empty	empty	empty	data1	data1	data1	data1
Consumer B read	none	none	none	none	data1	data1	data1
Consumer C read	none	none	none	none	none	data1	data1

Key Moments - 3 Insights

Why does data appear in Region B only after Step 2?

Because replication happens asynchronously after data is produced in Region A, as shown in execution_table row 2.

Can consumers in Region B read data before replication completes?

No, consumers read data only after replication finishes, as shown in execution_table row 4.

Is data produced separately in each region?

No, data is produced once in Region A and then replicated to other regions, as shown in execution_table rows 1-3.

Visual Quiz - 3 Questions

Look at the execution_table, at which step does Region C receive the data?

A Step 2

B Step 3

C Step 4

D Step 5

💡 Hint

Check the 'Target Region' and 'Result' columns in execution_table rows.

According to variable_tracker, what is the state of 'Data in Region B cluster' after Step 1?

A none

B data1

C empty

D undefined

💡 Hint

Look at the 'Data in Region B cluster' row and 'After Step 1' column in variable_tracker.

If replication to Region C is delayed, which step's result changes in execution_table?

A Step 3

B Step 4

C Step 5

D Step 2

💡 Hint

Replication to Region C happens at Step 3 in execution_table.

Concept Snapshot

Geo-replication in Kafka:
- Data produced in one region's Kafka cluster
- Asynchronous replication to other regional clusters
- Consumers read data locally for low latency
- Replication ensures data availability across regions
- Replication lag may cause temporary data delay

Full Transcript

Geo-replication strategies in Kafka involve producing data in one region and replicating it asynchronously to Kafka clusters in other regions. This allows consumers in each region to read data locally, reducing latency. The process starts with data production in the source region, followed by replication steps to target regions. Consumers then read the replicated data from their local clusters. Replication is asynchronous, so data appears in target regions after some delay. This strategy improves availability and fault tolerance across geographic locations.

Practice

1. What is the main purpose of geo-replication in Kafka?

easy

A. To encrypt Kafka topics for security

B. To compress Kafka messages for faster transmission

C. To copy data between Kafka clusters in different locations for safety and availability

D. To delete old Kafka data automatically

Solution

1. Step 1: Understand geo-replication concept

   Geo-replication means copying data between clusters in different places to keep data safe and accessible.

2. Step 2: Identify the main goal

   The main goal is to keep data safe and close to users worldwide by copying it across locations.

3. Final Answer:

   To copy data between Kafka clusters in different locations for safety and availability -> Option C

4. Quick Check:

   Geo-replication = Data copying across locations [OK]

Hint: Geo-replication means copying data across locations [OK]

Common Mistakes:

• Confusing geo-replication with data compression
• Thinking geo-replication encrypts data
• Assuming geo-replication deletes data

2. Which Kafka tool is commonly used for geo-replication?

easy

A. MirrorMaker 2

B. Kafka Connect Sink

C. Kafka Streams

D. Kafka Producer API

Solution

1. Step 1: Recall Kafka geo-replication tools

   MirrorMaker 2 is designed to replicate data between Kafka clusters in different locations.

2. Step 2: Differentiate from other tools

   Kafka Streams processes data streams, Kafka Connect Sink moves data to external systems, and Producer API sends data to Kafka but does not replicate clusters.

3. Final Answer:

   MirrorMaker 2 -> Option A

4. Quick Check:

   Geo-replication tool = MirrorMaker 2 [OK]

Hint: MirrorMaker 2 is for replicating Kafka clusters [OK]

Common Mistakes:

• Choosing Kafka Streams which is for processing, not replication
• Confusing Kafka Connect Sink with replication tool
• Selecting Producer API which only sends data

3. Given the following MirrorMaker 2 configuration snippet, what is the effect?

{
  "clusters": {
    "A": {"bootstrap.servers": "a.kafka:9092"},
    "B": {"bootstrap.servers": "b.kafka:9092"}
  },
  "mirrors": [{"source.cluster": "A", "target.cluster": "B", "topics": ["orders"]}]
}

medium

A. Data from cluster B is copied to cluster A for topic 'orders'

B. Data from cluster A is copied to cluster B for topic 'orders'

C. Data from both clusters A and B are merged into a new topic 'orders'

D. No data is copied because topics list is empty

Solution

1. Step 1: Analyze source and target clusters

   The config shows source.cluster as A and target.cluster as B, meaning data flows from A to B.

2. Step 2: Check topics list

   The topics list includes "orders", so only the 'orders' topic is replicated from A to B.

3. Final Answer:

   Data from cluster A is copied to cluster B for topic 'orders' -> Option B

4. Quick Check:

   Source A to target B for 'orders' topic [OK]

Hint: Source cluster data replicates to target cluster [OK]

Common Mistakes:

• Reversing source and target clusters
• Thinking topics list is empty
• Assuming data merges instead of copies

4. You configured MirrorMaker 2 but no data is replicating. Which is the most likely error in this config?

{
  "clusters": {
    "source": {"bootstrap.servers": "source.kafka:9092"},
    "target": {"bootstrap.servers": "target.kafka:9092"}
  },
  "mirrors": [{"source.cluster": "source", "target.cluster": "target", "topics": []}]
}

medium

A. Bootstrap servers are incorrectly formatted

B. MirrorMaker 2 requires only one cluster in config

C. Cluster names must be numeric, not strings

D. Empty topics list means no topics are selected for replication

Solution

1. Step 1: Check topics list in mirror config

   The topics list is empty [], so no topics are selected to replicate.

2. Step 2: Validate other config parts

   Bootstrap servers are correctly formatted, cluster names as strings are valid, and multiple clusters are allowed.

3. Final Answer:

   Empty topics list means no topics are selected for replication -> Option D

4. Quick Check:

   Empty topics list = no replication [OK]

Hint: Empty topics list disables replication [OK]

Common Mistakes:

• Assuming bootstrap.servers format is wrong
• Thinking cluster names must be numbers
• Believing MirrorMaker 2 supports only one cluster

5. You want to set up geo-replication between three Kafka clusters (US, EU, Asia) so that each cluster has all topics from the others. Which strategy should you use?

hard

A. Set up MirrorMaker 2 with a full mesh replication where each cluster replicates to the other two

B. Use a single MirrorMaker 2 instance replicating only from US to EU and Asia

C. Configure MirrorMaker 2 to replicate only the largest topic across clusters

D. Use Kafka Connect Sink to move data between clusters

Solution

1. Step 1: Understand full data availability goal

   Each cluster should have all topics from the others, meaning data must flow both ways between all clusters.

2. Step 2: Choose replication strategy

   A full mesh replication with MirrorMaker 2 means each cluster replicates to the other two, ensuring complete data presence everywhere.

3. Final Answer:

   Set up MirrorMaker 2 with a full mesh replication where each cluster replicates to the other two -> Option A

4. Quick Check:

   Full mesh replication = all clusters have all topics [OK]

Hint: Full mesh replicates all topics between all clusters [OK]

Common Mistakes:

• Using one-way replication only
• Replicating only largest topic, missing others
• Confusing Kafka Connect Sink with cluster replication