Disaster recovery planning in Kafka - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

Kafka Consumer Group Offset Behavior After Broker Failure

Consider a Kafka consumer group consuming from a topic with 3 partitions. The consumer commits offsets automatically every 5 seconds. If one broker hosting a partition leader fails and the leader is moved to another broker, what will be the output of the following code snippet that prints the last committed offset for each partition after the failover?

Kafka

from kafka import KafkaConsumer

consumer = KafkaConsumer(
    'my_topic',
    group_id='my_group',
    bootstrap_servers=['broker1:9092', 'broker2:9092', 'broker3:9092'],
    enable_auto_commit=True,
    auto_commit_interval_ms=5000
)

# Simulate reading messages
for _ in range(10):
    msg = next(consumer)
    print(f'Partition: {msg.partition}, Offset: {msg.offset}')

# After broker failure and leader election
committed = {partition: consumer.committed(partition) for partition in consumer.assignment()}
print(committed)

A A runtime error because the consumer cannot fetch committed offsets after broker failure.

B A dictionary with the last committed offsets for all partitions, reflecting the offsets before the broker failure.

C A dictionary with offsets reset to zero for all partitions due to leader re-election.

D An empty dictionary because the consumer lost all committed offsets after the broker failure.

💡 Hint

Think about how Kafka stores committed offsets in __consumer_offsets topic and how leader election affects offset availability.

Kafka stores committed offsets in an internal topic replicated across brokers. When a broker fails and a new leader is elected, the committed offsets remain available and unchanged. Therefore, the consumer can retrieve the last committed offsets as before the failure.

🧠 Conceptual

intermediate

Understanding Kafka Replication Factor Impact on Disaster Recovery

In Kafka, what is the primary benefit of setting a replication factor greater than 1 for a topic in the context of disaster recovery?

A It enables consumers to read messages faster by reading from multiple replicas simultaneously.

B It increases the throughput of message production by parallelizing writes.

C It allows messages to be stored on multiple brokers, ensuring data availability if one broker fails.

D It automatically backs up messages to an external storage system.

💡 Hint

Think about how Kafka handles broker failures and data loss prevention.

Replication factor controls how many copies of each partition exist on different brokers. Having multiple copies ensures that if one broker fails, another broker has the data, preventing data loss and enabling recovery.

🔧 Debug

advanced

Identify the Cause of Data Loss in Kafka After Broker Crash

A Kafka topic has replication factor 3 and min.insync.replicas set to 2. Producers use acks='all'. After a broker crash, some messages are missing from the topic. Which code snippet below best explains why data loss occurred?

Kafka

producer = KafkaProducer(
    bootstrap_servers=['broker1:9092', 'broker2:9092', 'broker3:9092'],
    acks='all',
    retries=0
)

# Sending messages
producer.send('my_topic', b'message1')
producer.flush()

A Retries set to 0 causes producer to not retry sending messages if a broker is down, leading to data loss.

B Acks='all' causes the producer to wait indefinitely, causing timeout and message loss.

C min.insync.replicas=2 means only one replica must acknowledge, so data loss is expected.

D Producer flush() is asynchronous and does not guarantee message delivery.

💡 Hint

Consider how producer retries affect message delivery guarantees.

With retries=0, if a broker is down during send, the producer does not retry sending the message, causing message loss even though acks='all' and min.insync.replicas=2 are set.

📝 Syntax

advanced

Identify the Syntax Error in Kafka Topic Configuration Code

Which of the following Kafka topic creation code snippets contains a syntax error that would prevent the topic from being created?

A admin_client.create_topics([NewTopic(name='my_topic', num_partitions=3, replication_factor=2)])

B admin_client.create_topics([NewTopic(name='my_topic', num_partitions=3)])

C admin_client.create_topics([NewTopic(name='my_topic', num_partitions=3, replication_factor=2, config={'cleanup.policy':'compact'})])

D admin_client.create_topics([NewTopic(name='my_topic', num_partitions=3, replication_factor='2')])

💡 Hint

Check the data types of the parameters passed to NewTopic.

The replication_factor parameter must be an integer. Passing it as a string causes a type error and prevents topic creation.

🚀 Application

expert

Designing a Kafka Disaster Recovery Plan for Multi-Region Deployment

You are tasked with designing a disaster recovery plan for a Kafka deployment spanning two geographic regions. Which approach below best ensures minimal data loss and quick recovery if one region fails?

A Use MirrorMaker 2 to replicate topics asynchronously between regions with monitoring and failover automation.

B Use a single Kafka cluster spanning both regions with no replication between them.

C Set replication factor to 1 and rely on local backups in each region for recovery.

D Manually export and import topic data daily between regions using scripts.

💡 Hint

Consider how Kafka supports cross-region replication and automated failover.

MirrorMaker 2 is designed for replicating Kafka topics between clusters asynchronously, enabling disaster recovery with minimal data loss and automated failover.