Why Kafka exists - Why It Works This Way

Overview - Why Kafka exists

What is it?

Kafka is a system that helps different parts of software talk to each other by sending messages quickly and reliably. It stores these messages so they can be read later, even if the receiver is busy or offline. Kafka is designed to handle a huge amount of messages without slowing down. It works like a middleman that keeps data flowing smoothly between systems.

Why it matters

Without Kafka, software systems would struggle to share information in real time, causing delays and lost data. Imagine a busy post office that can't keep track of letters or delivers them late. Kafka solves this by organizing and storing messages so they don't get lost and can be processed quickly. This helps businesses react faster and keep their services running smoothly.

Where it fits

Before learning Kafka, you should understand basic messaging concepts and how software components communicate. After Kafka, you can explore advanced topics like stream processing, event-driven architecture, and real-time analytics. Kafka fits in the journey between simple message queues and complex data processing pipelines.

Mental Model

Core Idea

Kafka exists to reliably move and store streams of messages between software systems at high speed and scale.

Think of it like...

Kafka is like a busy train station where many trains (messages) arrive and depart on time, carrying passengers (data) to different destinations (systems) without losing anyone along the way.

┌───────────────┐       ┌───────────────┐       ┌───────────────┐
│  Producers    │──────▶│    Kafka      │──────▶│  Consumers    │
│ (Message      │       │ (Message Hub) │       │ (Message      │
│  Senders)     │       │               │       │  Receivers)   │
└───────────────┘       └───────────────┘       └───────────────┘

Build-Up - 6 Steps

1

Foundation

What is a Message Broker

Concept: Introduce the idea of a message broker as a middleman that passes messages between software parts.

A message broker is like a mail sorter. It receives messages from one place and delivers them to another. This helps software parts communicate without needing to know about each other directly.

Result

You understand the basic role of a system that moves messages between software components.

Knowing what a message broker does helps you see why Kafka is needed to organize and deliver messages reliably.

2

Foundation

Challenges in Data Communication

Concept: Explain common problems when software systems share data, like delays and lost messages.

When many systems talk, messages can get lost if the receiver is busy or offline. Also, sending too many messages at once can slow things down. Without a good system, data can arrive late or not at all.

Result

You see why simple message passing can fail in busy or complex systems.

Understanding these problems shows why a robust system like Kafka is necessary.

3

Intermediate

Kafka’s Role as a Distributed Log

🤔Before reading on: do you think Kafka stores messages temporarily or permanently? Commit to your answer.

Concept: Kafka stores messages in a log that keeps data in order and allows multiple readers.

Kafka saves messages in a sequence called a log. This log keeps messages safe and lets many consumers read them at their own pace. Unlike simple queues, Kafka doesn’t delete messages immediately after reading.

Result

You understand Kafka’s unique way of storing messages for reliability and flexibility.

Knowing Kafka’s log storage explains how it supports multiple consumers and replaying data.

4

Intermediate

Handling High Volume and Speed

🤔Before reading on: do you think Kafka can handle millions of messages per second? Commit to your answer.

Concept: Kafka is designed to process huge amounts of messages quickly without losing data.

Kafka uses efficient storage and network methods to handle millions of messages per second. It spreads data across many servers to balance load and avoid slowdowns.

Result

You see how Kafka supports large-scale, fast data flows in real systems.

Understanding Kafka’s design for speed and scale shows why it suits big data and real-time needs.

5

Advanced

Fault Tolerance and Data Durability

🤔Before reading on: do you think Kafka loses messages if a server crashes? Commit to your answer.

Concept: Kafka keeps copies of data on multiple servers to prevent loss during failures.

Kafka replicates messages across servers. If one fails, others keep the data safe. This ensures messages are not lost and systems can recover quickly.

Result

You understand how Kafka protects data and keeps systems reliable.

Knowing Kafka’s replication prevents data loss and downtime in production.

6

Expert

Kafka’s Impact on Modern Architectures

🤔Before reading on: do you think Kafka is only for messaging or also for data processing? Commit to your answer.

Concept: Kafka enables event-driven and real-time data processing beyond simple messaging.

Kafka is not just a message mover; it powers systems that react instantly to data changes. It integrates with tools that process streams of data live, enabling new ways to build software.

Result

You see Kafka’s role as a foundation for modern, reactive software architectures.

Understanding Kafka’s broader impact reveals why it transformed how companies build data-driven applications.

Under the Hood

Kafka works by writing messages to disk in an append-only log format, partitioned across multiple servers. Each message is assigned an offset, allowing consumers to track their read position independently. Kafka uses replication to copy data across brokers, ensuring fault tolerance. Producers send messages to topics, which are divided into partitions for parallelism. Consumers pull messages at their own pace, enabling flexible processing.

Why designed this way?

Kafka was designed to handle large-scale, real-time data streams with high throughput and durability. Traditional message queues deleted messages after consumption, limiting replay and multiple consumers. Kafka’s log-based design allows multiple consumers to read independently and replay data. Replication and partitioning address reliability and scalability, meeting the needs of modern distributed systems.

┌───────────────┐          ┌───────────────┐          ┌───────────────┐
│   Producer    │─────────▶│   Kafka Broker│─────────▶│   Consumer    │
│ (Sends data)  │          │ (Stores logs) │          │ (Reads data)  │
└───────────────┘          └───────────────┘          └───────────────┘
          │                        │                         ▲
          │                        │                         │
          │                        ▼                         │
          │                ┌───────────────┐               │
          │                │ Replication & │───────────────┘
          │                │ Partitioning  │
          │                └───────────────┘

Myth Busters - 4 Common Misconceptions

Quick: Does Kafka delete messages immediately after a consumer reads them? Commit yes or no.

Common Belief: Kafka deletes messages as soon as a consumer reads them, like a normal queue.

Reality: Kafka retains messages for a configured time or size limit, allowing multiple consumers to read independently and replay messages.

Why it matters: Assuming immediate deletion leads to design mistakes where data replay or multiple consumers are needed but impossible.

Quick: Is Kafka only useful for small-scale systems? Commit yes or no.

Common Belief: Kafka is only for small projects or simple messaging needs.

Reality: Kafka is built for large-scale, high-throughput systems and is widely used in big data and real-time applications.

Why it matters: Underestimating Kafka’s scale can cause missed opportunities for building robust, scalable systems.

Quick: Does Kafka guarantee message order across all consumers? Commit yes or no.

Common Belief: Kafka guarantees global message order for all consumers.

Reality: Kafka guarantees order only within each partition, not across all partitions or consumers.

Why it matters: Misunderstanding ordering can cause bugs in systems that assume global order where it does not exist.

Quick: Can Kafka replace all databases for storing data? Commit yes or no.

Common Belief: Kafka can be used as a full database replacement for all data storage needs.

Reality: Kafka is designed for streaming and messaging, not as a general-purpose database with complex queries or transactions.

Why it matters: Using Kafka as a database leads to poor performance and missing features needed for data management.

Expert Zone

1

Kafka’s partitioning strategy affects load balancing and consumer parallelism, requiring careful topic design.

2

The choice of retention policies balances storage cost and data availability, impacting replay and recovery.

3

Kafka’s exactly-once semantics require specific configurations and understanding of producer and consumer behavior.

When NOT to use

Kafka is not suitable for low-latency request-response patterns or small-scale simple messaging. Alternatives like RabbitMQ or traditional message queues may be better for those cases. Also, Kafka is not a replacement for transactional databases or complex query engines.

Production Patterns

In production, Kafka is used for event sourcing, log aggregation, real-time analytics, and as the backbone of microservices communication. Companies use Kafka Connect to integrate with databases and Kafka Streams for processing data in motion.

Connections

Event-Driven Architecture

Kafka is a foundational technology enabling event-driven systems by delivering events reliably.

Understanding Kafka helps grasp how software can react instantly to events, improving responsiveness and scalability.

Distributed Systems

Kafka is a distributed system that manages data across multiple servers for fault tolerance and scalability.

Knowing Kafka deepens understanding of distributed coordination, replication, and partitioning challenges.

Railway Signaling Systems

Kafka’s message flow and ordering resemble how railway signals control train movements safely and efficiently.

Seeing Kafka like a signaling system highlights the importance of order, timing, and fault tolerance in complex networks.

Common Pitfalls

#1 Assuming Kafka deletes messages immediately after consumption.

Wrong approach:

Setting retention.ms to 0 or very low, expecting messages to vanish after reading.

Correct approach:

Configure retention.ms to a suitable time to keep messages for replay and multiple consumers.

Root cause:

Misunderstanding Kafka’s log retention model versus traditional queue behavior.

#2 Using a single partition for a high-throughput topic.

Wrong approach:

Creating a topic with only one partition for all messages.

Correct approach:

Create multiple partitions to allow parallel processing and better scalability.

Root cause:

Not realizing partitions enable Kafka’s horizontal scaling and consumer parallelism.

#3 Expecting global message order across partitions.

Wrong approach:

Designing consumers assuming all messages are strictly ordered globally.

Correct approach:

Design consumers to handle ordering within partitions only or implement ordering logic if needed.

Root cause:

Confusing partition-level ordering with global ordering guarantees.

Key Takeaways

Kafka exists to move and store messages reliably between software systems at large scale and speed.

It uses a log-based storage model that allows multiple consumers to read messages independently and replay data.

Kafka’s design solves common problems like message loss, slowdowns, and system failures with replication and partitioning.

Understanding Kafka’s role helps build modern, event-driven, and real-time data processing systems.

Misunderstanding Kafka’s retention, ordering, or scale can lead to design mistakes and system failures.

Practice

1. Why does Kafka exist in software systems?

easy

A. To help programs send and receive messages reliably and quickly

B. To replace databases for storing user profiles

C. To create user interfaces for web applications

D. To compile code faster

Solution

1. Step 1: Understand Kafka's main purpose

   Kafka is designed to handle messaging between programs efficiently and reliably.

2. Step 2: Compare options with Kafka's role

   The other options describe unrelated tasks like databases, UI, or compiling, which Kafka does not do.

3. Final Answer:

   To help programs send and receive messages reliably and quickly -> Option A

4. Quick Check:

   Kafka = Reliable fast messaging [OK]

Hint: Kafka is about messaging, not databases or UI [OK]

Common Mistakes:

• Confusing Kafka with a database
• Thinking Kafka builds user interfaces
• Assuming Kafka compiles code

2. Which of the following is the correct way to describe Kafka's data storage?

easy

A. Kafka stores messages in tables like a database

B. Kafka stores messages in files on a user's desktop

C. Kafka stores messages only temporarily in memory

D. Kafka stores messages in topics for multiple programs to use

Solution

1. Step 1: Recall Kafka's storage method

   Kafka organizes messages into topics, which multiple programs can read from or write to.

2. Step 2: Evaluate each option

   Kafka stores messages in tables like a database is incorrect because Kafka does not use tables like a database. Kafka stores messages in files on a user's desktop is wrong because Kafka does not store messages on a user's desktop. Kafka stores messages only temporarily in memory is incorrect because Kafka stores messages durably, not just temporarily in memory.

3. Final Answer:

   Kafka stores messages in topics for multiple programs to use -> Option D

4. Quick Check:

   Kafka storage = Topics for sharing [OK]

Hint: Remember Kafka uses topics, not tables or temporary memory [OK]

Common Mistakes:

• Thinking Kafka stores data like a traditional database
• Assuming Kafka stores data only temporarily
• Confusing Kafka storage location with local files

3. Consider this scenario: A program sends messages to a Kafka topic, and multiple other programs read from it. What is the main benefit of this setup?

medium

A. Messages are lost if one program fails

B. Multiple programs can read the same messages independently

C. Only one program can read messages at a time

D. Messages are deleted immediately after being read

Solution

1. Step 1: Understand Kafka's message consumption model

   Kafka allows multiple programs to read the same messages independently from a topic.

2. Step 2: Analyze each option

   Messages are lost if one program fails is false because Kafka ensures message durability. Only one program can read messages at a time is incorrect because Kafka supports multiple consumers reading simultaneously. Messages are deleted immediately after being read is wrong because Kafka retains messages for a configured time, not deleting immediately after reading.

3. Final Answer:

   Multiple programs can read the same messages independently -> Option B

4. Quick Check:

   Kafka supports multiple independent readers [OK]

Hint: Kafka lets many programs read same messages independently [OK]

Common Mistakes:

• Believing messages disappear after one read
• Thinking only one consumer can read at a time
• Assuming message loss on failure

4. A developer wrote code to send messages to Kafka but notices messages are not received by consumers. What could be a likely error?

medium

A. Kafka does not support multiple consumers

B. Kafka deletes messages immediately after sending

C. The topic name used by producer and consumer do not match

D. Kafka requires messages to be sent as files

Solution

1. Step 1: Identify common communication errors in Kafka

   If producer and consumer use different topic names, messages won't be received.

2. Step 2: Evaluate other options

   Kafka does not support multiple consumers is false; Kafka supports multiple consumers. Kafka deletes messages immediately after sending is incorrect; Kafka retains messages for a set time. Kafka requires messages to be sent as files is wrong; Kafka messages are sent as data, not files.

3. Final Answer:

   The topic name used by producer and consumer do not match -> Option C

4. Quick Check:

   Matching topic names needed for communication [OK]

Hint: Check topic names match between sender and receiver [OK]

Common Mistakes:

• Assuming Kafka deletes messages immediately
• Thinking Kafka only supports one consumer
• Believing messages must be files

5. You want to build a real-time system where many services process the same data stream independently and reliably. Why is Kafka a good choice?

hard

A. Kafka allows multiple consumers to read the same data without losing messages

B. Kafka stores data only temporarily, so it is fast but unreliable

C. Kafka requires all consumers to read messages in order and together

D. Kafka is designed only for batch processing, not real-time data

Solution

1. Step 1: Understand Kafka's support for real-time data and multiple consumers

   Kafka stores messages durably and allows many consumers to read independently, making it reliable for real-time systems.

2. Step 2: Analyze incorrect options

   Kafka stores data only temporarily, so it is fast but unreliable is false because Kafka stores data durably, not just temporarily. Kafka requires all consumers to read messages in order and together is incorrect; consumers can read independently and at different speeds. Kafka is designed only for batch processing, not real-time data is wrong; Kafka is designed for real-time streaming, not just batch.

3. Final Answer:

   Kafka allows multiple consumers to read the same data without losing messages -> Option A

4. Quick Check:

   Kafka = Reliable real-time multi-consumer streaming [OK]

Hint: Kafka supports reliable real-time multi-reader streams [OK]

Common Mistakes:

• Thinking Kafka only supports batch processing
• Assuming Kafka loses messages quickly
• Believing consumers must read together