Cross-datacenter replication in Kafka - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

What is the output of this Kafka MirrorMaker2 configuration snippet?

Given the following MirrorMaker2 configuration snippet, what will be the value of source.cluster.bootstrap.servers after parsing?

Kafka

clusters = {
  "source": {
    "bootstrap.servers": "source1:9092,source2:9092"
  },
  "target": {
    "bootstrap.servers": "target1:9092"
  }
}

source_bootstrap = clusters["source"]["bootstrap.servers"]
print(source_bootstrap)

A "target1:9092"

B "source1:9092,source2:9092"

C "source1:9092"

D KeyError

💡 Hint

Look at how the dictionary keys are accessed for the source cluster.

The configuration dictionary has a key 'source' which contains 'bootstrap.servers' with the value 'source1:9092,source2:9092'. Accessing clusters['source']['bootstrap.servers'] returns this string.

🧠 Conceptual

intermediate

Which option best describes the role of MirrorMaker2 in cross-datacenter replication?

Select the correct description of MirrorMaker2's function in Kafka cross-datacenter replication.

A It replicates Kafka topics from one cluster to another to ensure data availability across data centers.

B It compresses Kafka messages to reduce network bandwidth usage within a single cluster.

C It manages Kafka consumer group offsets locally without replication.

D It provides a user interface for Kafka topic configuration.

💡 Hint

Think about what cross-datacenter replication means for Kafka data.

MirrorMaker2 is designed to replicate topics between Kafka clusters in different data centers to improve availability and disaster recovery.

🔧 Debug

advanced

What error does this MirrorMaker2 connector configuration cause?

Consider this MirrorMaker2 connector config snippet: ``` "source.cluster": "clusterA", "target.cluster": "clusterB", "topics": "^important-.*", "tasks.max": "two" ``` What error will occur when starting the connector?

A ValueError: invalid literal for int() with base 10: 'two'

B No error, connector starts successfully

C KeyError: 'tasks.max' not found

D TypeError: unsupported operand type(s) for +: 'int' and 'str'

💡 Hint

Check the expected data type for 'tasks.max' in Kafka connector configs.

'tasks.max' expects an integer value. Providing the string 'two' causes a ValueError when the system tries to convert it to int.

📝 Syntax

advanced

Which option correctly defines a Kafka Connect replication policy class in MirrorMaker2?

Choose the correct Java class name syntax for a custom replication policy in MirrorMaker2 configuration.

A com.example.kafka.replication.customReplicationPolicy

B com.example.kafka.replication.CustomReplicationPolicy()

C com.example.kafka.replication.CustomReplicationPolicy

D com.example.kafka.replication.Custom_Replication_Policy

💡 Hint

Java class names follow a specific naming convention and do not include parentheses.

Java class names use PascalCase without underscores and do not include parentheses when referenced in configs.

🚀 Application

expert

How many topics will be replicated with this MirrorMaker2 topic regex?

If the source Kafka cluster has topics: ["orders", "orders-2023", "payments", "payments-2023", "logs", "logs-archive"], and MirrorMaker2 is configured with: ``` topics = "^(orders|payments)-2023$" ``` How many topics will MirrorMaker2 replicate?

A 3

B 1

C 4

D 2

💡 Hint

Check which topic names exactly match the regex pattern.

The regex matches topics that start with 'orders' or 'payments' and end exactly with '-2023'. Only 'orders-2023' and 'payments-2023' match.

Practice

1.

What is the main purpose of cross-datacenter replication in Kafka?

easy

A. To copy Kafka topics between clusters for data safety and availability

B. To compress Kafka messages for faster transmission

C. To encrypt Kafka topics for security

D. To delete old Kafka topics automatically

Solution

1. Step 1: Understand the concept of cross-datacenter replication

   Cross-datacenter replication means copying data from one Kafka cluster to another in a different location.

2. Step 2: Identify the main goal of this replication

   The goal is to keep data safe and available even if one cluster fails.

3. Final Answer:

   To copy Kafka topics between clusters for data safety and availability -> Option A

4. Quick Check:

   Replication = Data safety and availability [OK]

Hint: Replication means copying data across clusters for safety [OK]

Common Mistakes:

• Confusing replication with compression
• Thinking replication encrypts data
• Assuming replication deletes data

2.

Which of the following is the correct way to specify the source cluster's bootstrap servers in Kafka MirrorMaker configuration?

consumer.bootstrap.servers = ?

easy

A. "source.kafka.cluster:9092"

B. "target.kafka.cluster:9092"

C. "localhost:2181"

D. "mirror.kafka.cluster:2181"

Solution

1. Step 1: Identify the role of consumer.bootstrap.servers

   This property tells MirrorMaker where to read data from, so it must point to the source Kafka cluster.

2. Step 2: Choose the correct server address for the source cluster

   The source cluster's Kafka broker address is usually in the form host:port, like "source.kafka.cluster:9092".

3. Final Answer:

   "source.kafka.cluster:9092" -> Option A

4. Quick Check:

   consumer.bootstrap.servers = source cluster address [OK]

Hint: Consumer connects to source cluster bootstrap servers [OK]

Common Mistakes:

• Using target cluster address for consumer.bootstrap.servers
• Using ZooKeeper address instead of Kafka broker
• Confusing consumer and producer properties

3.

Given the following MirrorMaker configuration snippet, what will be the value of producer.bootstrap.servers?

consumer.bootstrap.servers=source1.kafka:9092,source2.kafka:9092
producer.bootstrap.servers=target.kafka:9092

medium

A. "source1.kafka:9092,source2.kafka:9092"

B. "target.kafka:9092"

C. "source1.kafka:9092"

D. "localhost:9092"

Solution

1. Step 1: Read the configuration lines carefully

   The snippet shows two properties: consumer.bootstrap.servers and producer.bootstrap.servers with their values.

2. Step 2: Identify the value assigned to producer.bootstrap.servers

   It is explicitly set to "target.kafka:9092" in the second line.

3. Final Answer:

   "target.kafka:9092" -> Option B

4. Quick Check:

   producer.bootstrap.servers = target cluster address [OK]

Hint: Producer connects to target cluster bootstrap servers [OK]

Common Mistakes:

• Mixing consumer and producer server addresses
• Assuming producer uses source cluster address
• Ignoring explicit configuration values

4.

Identify the error in this MirrorMaker consumer configuration:

consumer.bootstrap.servers=target.kafka:9092
consumer.group.id=mirror-group

medium

A. consumer.bootstrap.servers should use port 2181

B. consumer.group.id is missing

C. consumer.bootstrap.servers should point to the source cluster, not target

D. consumer.group.id should be unique per topic

Solution

1. Step 1: Understand the role of consumer.bootstrap.servers

   This property must point to the source Kafka cluster to read data for replication.

2. Step 2: Check the given value

   It incorrectly points to "target.kafka:9092", which is the destination cluster, not the source.

3. Final Answer:

   consumer.bootstrap.servers should point to the source cluster, not target -> Option C

4. Quick Check:

   Consumer connects to source cluster [OK]

Hint: Consumer must connect to source cluster, not target [OK]

Common Mistakes:

• Using target cluster address for consumer.bootstrap.servers
• Confusing ZooKeeper port with Kafka broker port
• Assuming group.id is missing when it is present

5.

You want to replicate only topics starting with logs- from the source cluster to the target cluster using MirrorMaker. Which configuration property should you set to filter these topics?

hard

A. consumer.topic.filter=logs-*

B. producer.topic.filter=logs-*

C. mirror.topic.prefix=logs-

D. consumer.topic.whitelist=logs-*

Solution

1. Step 1: Identify the property to filter topics in MirrorMaker

   MirrorMaker uses consumer.topic.whitelist to specify which topics to consume from the source cluster.

2. Step 2: Match the pattern for topics starting with 'logs-'

   The pattern "logs-*" matches all topics starting with "logs-".

3. Final Answer:

   consumer.topic.whitelist=logs-* -> Option D

4. Quick Check:

   Use consumer.topic.whitelist to filter topics [OK]

Hint: Use consumer.topic.whitelist to select topics by pattern [OK]

Common Mistakes:

• Using producer.topic.filter which does not exist
• Confusing mirror.topic.prefix with filtering
• Using consumer.topic.filter instead of whitelist