Disaster recovery planning in Kafka - Step-by-Step Execution

Process Flow - Disaster recovery planning

Identify critical data

↓

Set up replication

↓

Configure backup storage

↓

Monitor replication health

↓

Detect disaster event

↓

Failover to backup cluster

↓

Restore data if needed

↓

Resume normal operations

This flow shows the key steps in disaster recovery for Kafka: identifying data, replicating it, monitoring, detecting failure, failing over, restoring, and resuming.

Execution Sample

Kafka

producer.send(topic, data)
# Data is replicated to backup cluster
if disaster_detected:
    failover_to_backup()
    restore_data()
consumer.consume()

This code sends data to Kafka, checks for disaster, fails over if needed, restores data, and continues consuming.

Process Table

Step	Action	Condition	Result	Notes
1	Send data to primary cluster	N/A	Data stored and replicated	Producer sends messages
2	Monitor replication	Replication healthy?	Yes	No issues detected
3	Detect disaster	Disaster occurred?	No	Continue normal operation
4	Send data to primary cluster	N/A	Data stored and replicated	More messages sent
5	Detect disaster	Disaster occurred?	Yes	Failover triggered
6	Failover to backup cluster	N/A	Backup cluster active	Switch consumers to backup
7	Restore data if needed	Data missing?	Data restored	Backup cluster synced
8	Resume operations	N/A	Normal operation on backup	Consumers consume from backup
9	Monitor replication	Replication healthy?	Yes	Backup cluster stable
10	End	No more disaster	System stable	Recovery complete

💡 Disaster detected at step 5 triggers failover; recovery completes at step 10.

Status Tracker

Variable	Start	After Step 1	After Step 5	After Step 7	Final
data_location	primary cluster	primary cluster	failover triggered	backup cluster restored	backup cluster active
replication_status	not started	healthy	disrupted	restored	healthy
consumer_target	primary cluster	primary cluster	switching	backup cluster	backup cluster

Key Moments - 3 Insights

Why do we need to monitor replication before disaster?

Monitoring ensures data is safely copied to backup before disaster happens, as shown in steps 2 and 9 in the execution_table.

What triggers the failover to the backup cluster?

Failover triggers when disaster is detected (step 5), switching consumers to backup to keep data available.

Why restore data after failover if replication exists?

Sometimes data may be missing or incomplete on backup; restoring ensures full data availability (step 7).

Visual Quiz - 3 Questions

Look at the execution_table, at which step is disaster detected?

A Step 7

B Step 3

C Step 5

D Step 9

💡 Hint

Check the 'Condition' and 'Result' columns for disaster detection in execution_table rows.

According to variable_tracker, where is the consumer_target after step 7?

A backup cluster

B failover triggered

C primary cluster

D not started

💡 Hint

Look at the 'consumer_target' row and the 'After Step 7' column in variable_tracker.

If replication_status was unhealthy at step 2, what would likely happen next?

A Failover immediately

B Trigger disaster detection

C Attempt to restore data

D Continue normal operation

💡 Hint

Consider the importance of replication health in execution_table steps 2 and 3.

Concept Snapshot

Disaster recovery in Kafka:
- Identify critical data and replicate it to backup cluster
- Monitor replication health continuously
- Detect disaster events to trigger failover
- Failover switches consumers to backup cluster
- Restore missing data if needed
- Resume normal operations on backup

Full Transcript

Disaster recovery planning in Kafka involves setting up replication of critical data to a backup cluster. The system monitors replication health to ensure data safety. When a disaster occurs, failover switches operations to the backup cluster. Data is restored if missing, and normal operations resume on the backup. This process ensures data availability and system resilience.

Practice

1. What is the main purpose of disaster recovery planning in Kafka?

easy

A. To keep Kafka data safe and services running after failures

B. To improve Kafka message throughput

C. To reduce Kafka cluster size

D. To speed up Kafka consumer processing

Solution

1. Step 1: Understand disaster recovery goal

   Disaster recovery planning focuses on protecting data and maintaining service availability during failures.

2. Step 2: Match goal to options

   Only To keep Kafka data safe and services running after failures mentions keeping data safe and services running after problems, which matches the goal.

3. Final Answer:

   To keep Kafka data safe and services running after failures -> Option A

4. Quick Check:

   Disaster recovery = data safety and uptime [OK]

Hint: Disaster recovery means data safety and uptime [OK]

Common Mistakes:

• Confusing disaster recovery with performance tuning
• Thinking it reduces cluster size
• Assuming it speeds up consumers

2. Which Kafka feature is used to replicate data to a backup cluster for disaster recovery?

easy

A. Kafka Connect

B. Kafka Streams

C. MirrorMaker

D. Kafka Producer

Solution

1. Step 1: Identify replication tool for backup clusters

   MirrorMaker is designed to copy data between Kafka clusters for backup and disaster recovery.

2. Step 2: Compare options

   Kafka Connect is for data integration, Kafka Streams for processing, and Producer for sending data, so only MirrorMaker fits replication to backup clusters.

3. Final Answer:

   MirrorMaker -> Option C

4. Quick Check:

   Replication to backup = MirrorMaker [OK]

Hint: MirrorMaker copies data between clusters [OK]

Common Mistakes:

• Choosing Kafka Connect which is for integration, not replication
• Confusing Kafka Streams with replication
• Thinking Kafka Producer replicates data

3. Given the following MirrorMaker command, what is its purpose?

bin/kafka-mirror-maker.sh --consumer.config consumer.properties --producer.config producer.properties --whitelist='topic1,topic2'

medium

A. Consume messages from topic1 and topic2 without replication

B. Delete topic1 and topic2 from the target cluster

C. Produce messages to topic1 and topic2 only on source cluster

D. Replicate only topic1 and topic2 from source to target cluster

Solution

1. Step 1: Understand MirrorMaker command components

   The command uses consumer and producer configs and a whitelist of topics to replicate.

2. Step 2: Interpret whitelist usage

   The whitelist='topic1,topic2' means only these topics are replicated from source to target cluster.

3. Final Answer:

   Replicate only topic1 and topic2 from source to target cluster -> Option D

4. Quick Check:

   Whitelist limits topics replicated [OK]

Hint: Whitelist limits topics MirrorMaker replicates [OK]

Common Mistakes:

• Thinking it only consumes without replication
• Assuming it produces only on source cluster
• Believing it deletes topics

4. You wrote a MirrorMaker config but replication fails. Which is a likely cause?

bootstrap.servers=source:9092
group.id=mirror-group
whitelist=topicA,topicB

What is wrong?

medium

A. The whitelist should be 'whitelist='topicA,topicB' with quotes

B. Missing producer bootstrap.servers config for target cluster

C. group.id should be 'producer-group' instead of 'mirror-group'

D. bootstrap.servers should point to target cluster, not source

Solution

1. Step 1: Check MirrorMaker config requirements

   MirrorMaker needs both consumer (source) and producer (target) bootstrap.servers configs to replicate data.

2. Step 2: Identify missing config

   The given config only shows source bootstrap.servers; missing producer config causes replication failure.

3. Final Answer:

   Missing producer bootstrap.servers config for target cluster -> Option B

4. Quick Check:

   Both consumer and producer configs needed [OK]

Hint: Both consumer and producer configs are required [OK]

Common Mistakes:

• Thinking quotes are mandatory around whitelist
• Confusing group.id with producer group
• Setting bootstrap.servers to target cluster only

5. You want to ensure minimal data loss during a disaster. Which combination is best for Kafka disaster recovery?

hard

A. Use topic replication factor 3 and MirrorMaker to backup cluster

B. Use only MirrorMaker without replication factor increase

C. Use replication factor 1 and rely on manual backups

D. Use Kafka Streams to replicate data to backup cluster

Solution

1. Step 1: Understand replication factor role

   A replication factor of 3 means data is copied to 3 brokers, reducing data loss risk.

2. Step 2: Combine with MirrorMaker backup

   MirrorMaker copies data to a separate cluster, adding disaster recovery protection.

3. Final Answer:

   Use topic replication factor 3 and MirrorMaker to backup cluster -> Option A

4. Quick Check:

   Replication + MirrorMaker = minimal data loss [OK]

Hint: Combine replication factor and MirrorMaker for best safety [OK]

Common Mistakes:

• Relying only on MirrorMaker without replication
• Using replication factor 1 which risks data loss
• Confusing Kafka Streams with replication tool