Why Kafka exists - Why It Works

Introduction

Kafka exists to help different parts of a system talk to each other by sending messages quickly and reliably. It solves the problem of moving data between apps without losing it or slowing down the system.

When you want to collect data from many sources and process it later without losing any messages

When you need to connect different apps so they can work together by sharing information instantly

When you want to handle a large stream of events, like user clicks or sensor data, in real time

When you want to build a system that keeps working even if some parts fail temporarily

When you want to store messages safely so they can be read multiple times or by many apps

Commands

This command creates a new topic named 'example-topic' with 3 partitions to allow parallel processing and a replication factor of 1 for data safety.

Terminal

kafka-topics.sh --create --topic example-topic --bootstrap-server localhost:9092 --partitions 3 --replication-factor 1

Expected Output

Created topic example-topic.

→

--topic - Name of the topic to create

→

--partitions - Number of partitions for parallelism

→

--replication-factor - Number of copies of data for fault tolerance

This command starts a simple producer that sends messages to the 'example-topic'. You can type messages and press enter to send them.

Terminal

kafka-console-producer.sh --topic example-topic --bootstrap-server localhost:9092

Expected Output

No output (command runs silently)

→

--topic - Topic to send messages to

→

--bootstrap-server - Kafka server address

This command starts a consumer that reads all messages from the beginning of 'example-topic', showing how Kafka stores messages for later use.

Terminal

kafka-console-consumer.sh --topic example-topic --bootstrap-server localhost:9092 --from-beginning

Expected Output

Hello

World

Kafka is great

→

--topic - Topic to read messages from

→

--from-beginning - Read all messages from the start

Key Concept

If you remember nothing else, remember: Kafka exists to move data between apps quickly, safely, and without losing messages.

Common Mistakes

Creating a topic without enough partitions

This limits how many consumers can read in parallel, slowing down processing.

Create topics with multiple partitions to allow parallel processing.

Not setting replication factor

Without replication, data can be lost if a server fails.

Set replication factor to at least 2 for data safety.

Starting consumer without --from-beginning when you want all messages

The consumer will only see new messages, missing old ones.

Use --from-beginning to read all messages from the start.

Summary

Kafka helps apps send and receive messages quickly and safely.

Create topics with partitions and replication for performance and safety.

Use producers to send messages and consumers to read them, optionally from the start.