Disaster recovery planning in Kafka - Commands & Configuration

Introduction

Disaster recovery planning helps you prepare for unexpected failures in your Kafka system. It ensures your data and services can be restored quickly after problems like hardware failure or data corruption.

When you want to protect your Kafka data from accidental deletion or corruption.

When you need to recover Kafka topics and messages after a server crash.

When you want to replicate Kafka data to another data center for backup.

When you want to automate restoring Kafka configurations and topics after failure.

When you want to minimize downtime and data loss during Kafka outages.

Config File - server.properties

server.properties

broker.id=1
log.dirs=/var/lib/kafka-logs
zookeeper.connect=localhost:2181
# Enable topic auto-creation
auto.create.topics.enable=true
# Enable log retention for disaster recovery
log.retention.hours=168
# Enable replication for fault tolerance
default.replication.factor=3
min.insync.replicas=2
# Enable log cleaner for compacted topics
log.cleaner.enable=true
# Configure listeners
listeners=PLAINTEXT://:9092
# Configure inter-broker communication
inter.broker.listener.name=PLAINTEXT

This configuration file sets up a Kafka broker with key settings for disaster recovery:

• log.retention.hours=168: Keeps logs for 7 days to allow recovery from recent data loss.
• default.replication.factor=3: Ensures each topic partition is copied to 3 brokers for fault tolerance.
• min.insync.replicas=2: Requires at least 2 replicas to acknowledge writes, preventing data loss.
• log.cleaner.enable=true: Enables log compaction for topics where only the latest value per key is kept, useful for recovery.

Commands

This command creates a Kafka topic named 'disaster-recovery-test' with 3 partitions and replication factor 3 to ensure data is copied across brokers for recovery.

Terminal

kafka-topics.sh --create --topic disaster-recovery-test --bootstrap-server localhost:9092 --partitions 3 --replication-factor 3

Expected Output

Created topic disaster-recovery-test.

→

--partitions - Number of partitions for parallelism and fault tolerance

→

--replication-factor - Number of copies of each partition for data safety

This command starts a producer to send messages to the 'disaster-recovery-test' topic to simulate data input.

Terminal

kafka-console-producer.sh --topic disaster-recovery-test --bootstrap-server localhost:9092

Expected Output

No output (command runs silently)

→

--topic - Specifies the topic to send messages to

This command reads the first 5 messages from the 'disaster-recovery-test' topic to verify data is stored and replicated correctly.

Terminal

kafka-console-consumer.sh --topic disaster-recovery-test --bootstrap-server localhost:9092 --from-beginning --max-messages 5

Expected Output

message1

message2

message3

message4

message5

→

--from-beginning - Reads messages from the start of the topic

→

--max-messages - Limits the number of messages to read

This command starts a partition reassignment to move partitions to different brokers as part of recovery or maintenance.

Terminal

kafka-reassign-partitions.sh --zookeeper localhost:2181 --reassignment-json-file reassignment.json --execute

Expected Output

Starting partition reassignment for topic disaster-recovery-test.

Reassignment started successfully.

→

--execute - Executes the reassignment described in the JSON file

This command verifies the status of the partition reassignment to ensure it completed successfully.

Terminal

kafka-reassign-partitions.sh --zookeeper localhost:2181 --verify

Expected Output

Reassignment of partitions is complete.

→

--verify - Checks the progress and completion of reassignment

Key Concept

If you remember nothing else from disaster recovery planning in Kafka, remember: replication and log retention settings are your first line of defense against data loss.

Common Mistakes

Setting replication factor to 1 for topics

This means there is only one copy of data, so if that broker fails, data is lost.

Always set replication factor to at least 2 or 3 for important topics to ensure copies exist.

Not verifying partition reassignment status

You might think reassignment finished but it could be stuck, causing uneven load or data loss risk.

Always run the verify command after reassignment to confirm success.

Not enabling log retention or setting it too low

Kafka deletes old data too soon, making recovery impossible for recent messages.

Set log retention to a reasonable time (e.g., 7 days) to allow recovery window.

Summary

Create topics with replication factor 3 to keep multiple copies of data.

Use producers and consumers to test data flow and verify replication.

Use partition reassignment commands to move data safely during recovery.

Verify reassignment completion to ensure cluster health.

Configure log retention and replication in server.properties for disaster readiness.

Practice

1. What is the main purpose of disaster recovery planning in Kafka?

easy

A. To keep Kafka data safe and services running after failures

B. To improve Kafka message throughput

C. To reduce Kafka cluster size

D. To speed up Kafka consumer processing

Solution

1. Step 1: Understand disaster recovery goal

   Disaster recovery planning focuses on protecting data and maintaining service availability during failures.

2. Step 2: Match goal to options

   Only To keep Kafka data safe and services running after failures mentions keeping data safe and services running after problems, which matches the goal.

3. Final Answer:

   To keep Kafka data safe and services running after failures -> Option A

4. Quick Check:

   Disaster recovery = data safety and uptime [OK]

Hint: Disaster recovery means data safety and uptime [OK]

Common Mistakes:

• Confusing disaster recovery with performance tuning
• Thinking it reduces cluster size
• Assuming it speeds up consumers

2. Which Kafka feature is used to replicate data to a backup cluster for disaster recovery?

easy

A. Kafka Connect

B. Kafka Streams

C. MirrorMaker

D. Kafka Producer

Solution

1. Step 1: Identify replication tool for backup clusters

   MirrorMaker is designed to copy data between Kafka clusters for backup and disaster recovery.

2. Step 2: Compare options

   Kafka Connect is for data integration, Kafka Streams for processing, and Producer for sending data, so only MirrorMaker fits replication to backup clusters.

3. Final Answer:

   MirrorMaker -> Option C

4. Quick Check:

   Replication to backup = MirrorMaker [OK]

Hint: MirrorMaker copies data between clusters [OK]

Common Mistakes:

• Choosing Kafka Connect which is for integration, not replication
• Confusing Kafka Streams with replication
• Thinking Kafka Producer replicates data

3. Given the following MirrorMaker command, what is its purpose?

bin/kafka-mirror-maker.sh --consumer.config consumer.properties --producer.config producer.properties --whitelist='topic1,topic2'

medium

A. Consume messages from topic1 and topic2 without replication

B. Delete topic1 and topic2 from the target cluster

C. Produce messages to topic1 and topic2 only on source cluster

D. Replicate only topic1 and topic2 from source to target cluster

Solution

1. Step 1: Understand MirrorMaker command components

   The command uses consumer and producer configs and a whitelist of topics to replicate.

2. Step 2: Interpret whitelist usage

   The whitelist='topic1,topic2' means only these topics are replicated from source to target cluster.

3. Final Answer:

   Replicate only topic1 and topic2 from source to target cluster -> Option D

4. Quick Check:

   Whitelist limits topics replicated [OK]

Hint: Whitelist limits topics MirrorMaker replicates [OK]

Common Mistakes:

• Thinking it only consumes without replication
• Assuming it produces only on source cluster
• Believing it deletes topics

4. You wrote a MirrorMaker config but replication fails. Which is a likely cause?

bootstrap.servers=source:9092
group.id=mirror-group
whitelist=topicA,topicB

What is wrong?

medium

A. The whitelist should be 'whitelist='topicA,topicB' with quotes

B. Missing producer bootstrap.servers config for target cluster

C. group.id should be 'producer-group' instead of 'mirror-group'

D. bootstrap.servers should point to target cluster, not source

Solution

1. Step 1: Check MirrorMaker config requirements

   MirrorMaker needs both consumer (source) and producer (target) bootstrap.servers configs to replicate data.

2. Step 2: Identify missing config

   The given config only shows source bootstrap.servers; missing producer config causes replication failure.

3. Final Answer:

   Missing producer bootstrap.servers config for target cluster -> Option B

4. Quick Check:

   Both consumer and producer configs needed [OK]

Hint: Both consumer and producer configs are required [OK]

Common Mistakes:

• Thinking quotes are mandatory around whitelist
• Confusing group.id with producer group
• Setting bootstrap.servers to target cluster only

5. You want to ensure minimal data loss during a disaster. Which combination is best for Kafka disaster recovery?

hard

A. Use topic replication factor 3 and MirrorMaker to backup cluster

B. Use only MirrorMaker without replication factor increase

C. Use replication factor 1 and rely on manual backups

D. Use Kafka Streams to replicate data to backup cluster

Solution

1. Step 1: Understand replication factor role

   A replication factor of 3 means data is copied to 3 brokers, reducing data loss risk.

2. Step 2: Combine with MirrorMaker backup

   MirrorMaker copies data to a separate cluster, adding disaster recovery protection.

3. Final Answer:

   Use topic replication factor 3 and MirrorMaker to backup cluster -> Option A

4. Quick Check:

   Replication + MirrorMaker = minimal data loss [OK]

Hint: Combine replication factor and MirrorMaker for best safety [OK]

Common Mistakes:

• Relying only on MirrorMaker without replication
• Using replication factor 1 which risks data loss
• Confusing Kafka Streams with replication tool