Why distributed architecture ensures reliability in Kafka - Why It Works

Introduction

Distributed architecture splits work across many machines. This helps keep systems running even if some parts fail. It makes services more reliable and available.

When you want your messaging system to keep working even if one server crashes

When you need to handle large amounts of data without losing messages

When you want to spread workload across multiple servers to avoid overload

When you want to add or remove servers without stopping the whole system

When you want to ensure data is safe by storing copies on different machines

Commands

This command creates a topic named 'my-topic' with 3 partitions and 2 copies (replicas) of each partition for reliability.

Terminal

kafka-topics.sh --create --topic my-topic --bootstrap-server localhost:9092 --partitions 3 --replication-factor 2

Expected Output

Created topic my-topic.

→

--partitions - Number of partitions to split the topic into

→

--replication-factor - Number of copies of each partition for fault tolerance

→

--bootstrap-server - Kafka server address to connect to

This command shows details about 'my-topic', including partitions and their replicas to verify reliability setup.

Terminal

kafka-topics.sh --describe --topic my-topic --bootstrap-server localhost:9092

Expected Output

Topic: my-topic	PartitionCount: 3	ReplicationFactor: 2	Configs: 

	Topic: my-topic	Partition: 0	Leader: 1	Replicas: 1,2	Isr: 1,2

	Topic: my-topic	Partition: 1	Leader: 2	Replicas: 2,3	Isr: 2,3

	Topic: my-topic	Partition: 2	Leader: 3	Replicas: 3,1	Isr: 3,1

→

--describe - Show detailed info about the topic

→

--topic - Specify the topic name

→

--bootstrap-server - Kafka server address to connect to

Key Concept

If you remember nothing else from this pattern, remember: distributing data and copies across servers keeps your system reliable even if some servers fail.

Common Mistakes

Creating a topic with replication-factor 1

No copies means if one server fails, data is lost and system is unreliable.

Always set replication-factor to at least 2 to have copies for fault tolerance.

Setting partitions to 1 for high load topics

Single partition limits parallel processing and can cause bottlenecks.

Use multiple partitions to spread load and improve performance.

Summary

Create Kafka topics with multiple partitions and replication for reliability.

Use replication-factor to keep copies of data on different servers.

Describe topics to verify partitions and replicas are set correctly.