First message (produce and consume) in Kafka - Deep Dive

Overview - First message (produce and consume)

What is it?

Kafka is a system that lets programs send and receive messages quickly and reliably. Producing means sending a message to Kafka, and consuming means reading that message from Kafka. This process helps different parts of a system talk to each other without being directly connected. It works like a message post office that stores and forwards messages.

Why it matters

Without producing and consuming messages, programs would have to wait for each other or be tightly linked, making systems slow and fragile. Kafka solves this by allowing messages to be sent and received independently, so systems can work faster and keep running even if parts fail. This makes apps more reliable and scalable, which is important for real-time data and big systems.

Where it fits

Before learning this, you should understand basic messaging concepts and how data flows between systems. After this, you can learn about Kafka topics, partitions, offsets, and how to handle message failures and scaling in production.

Mental Model

Core Idea

Producing sends messages into Kafka, and consuming reads them out, enabling independent communication between systems.

Think of it like...

It's like sending a letter to a mailbox (produce) and someone later picking it up and reading it (consume), without needing to be there at the same time.

┌─────────────┐      ┌─────────────┐      ┌─────────────┐
│ Producer    │─────▶│ Kafka Topic │─────▶│ Consumer    │
└─────────────┘      └─────────────┘      └─────────────┘

Build-Up - 6 Steps

1

Foundation

What is Kafka messaging

Concept: Kafka is a system that stores and forwards messages between programs.

Kafka acts like a post office for messages. Producers send messages to Kafka, which stores them in topics. Consumers then read messages from these topics when they want.

Result

You understand Kafka as a middleman that holds messages until consumers read them.

Understanding Kafka as a message broker helps you see why producing and consuming are separate steps.

2

Foundation

Basic produce and consume commands

Concept: You can send and receive messages using simple Kafka commands.

To produce a message: kafka-console-producer --broker-list localhost:9092 --topic test Type your message and press enter. To consume messages: kafka-console-consumer --bootstrap-server localhost:9092 --topic test --from-beginning This shows all messages from the start.

Result

You can send a message to Kafka and read it back from the topic.

Knowing these commands lets you try producing and consuming messages hands-on.

3

Intermediate

Message flow and offsets

🤔Before reading on: do you think consumers read messages in order or randomly? Commit to your answer.

Concept: Messages in Kafka topics have positions called offsets that track order and reading progress.

Each message in a Kafka topic has an offset number starting at 0. Consumers read messages in order by offset. Kafka remembers which offset each consumer has read, so it can continue from there next time.

Result

You see that Kafka keeps track of message order and consumer progress using offsets.

Understanding offsets explains how Kafka ensures consumers don't miss or repeat messages.

4

Intermediate

Producer and consumer configuration basics

🤔Before reading on: do you think producers must wait for consumers to be ready before sending messages? Commit to your answer.

Concept: Producers and consumers have settings that control how messages are sent and received, like delivery guarantees and reading behavior.

Producers can set options like 'acks' to control how many Kafka servers confirm a message before it's considered sent. Consumers can set 'auto.offset.reset' to decide what to do if no offset is found (start at earliest or latest message). These settings affect reliability and message flow.

Result

You learn how to adjust producer and consumer behavior for your needs.

Knowing configuration options helps you balance speed, reliability, and message delivery guarantees.

5

Advanced

Handling first message in real apps

🤔Before reading on: do you think consuming always starts from the latest message or from the beginning? Commit to your answer.

Concept: In real systems, consuming the first message requires setting the right offset and handling cases where no messages exist yet.

When a consumer starts, it may find no offset saved. Setting 'auto.offset.reset=earliest' makes it read from the first message. If the topic is empty, the consumer waits for new messages. This ensures the first message is not missed even if the consumer starts late.

Result

You understand how to make sure your consumer reads the very first message produced.

Knowing how to handle offsets and empty topics prevents missing important initial messages.

6

Expert

Internal message storage and delivery guarantees

🤔Before reading on: do you think Kafka deletes messages immediately after consumption? Commit to your answer.

Concept: Kafka stores messages durably and delivers them based on configured guarantees, independent of consumption state.

Kafka writes messages to disk in partitions and keeps them for a configured time or size limit. Messages are not deleted when consumed; they remain until retention expires. This allows multiple consumers to read independently and supports replaying messages.

Result

You see Kafka as a durable log, not just a queue that deletes messages after reading.

Understanding Kafka's storage model explains why consumers can read messages at different times without loss.

Under the Hood

Kafka stores messages in partitions on disk as an ordered log. Producers append messages to the end. Consumers track offsets to know which message to read next. Kafka uses a commit log design, which means messages are immutable and stored sequentially for fast reads and writes. This design supports high throughput and fault tolerance.

Why designed this way?

Kafka was designed to handle large volumes of data with low latency and high reliability. Using a commit log allows simple, fast writes and easy recovery after failures. Keeping messages for a retention period enables multiple consumers to read independently and replay messages if needed. Alternatives like traditional queues delete messages on consumption, limiting flexibility.

┌───────────────┐
│ Producer      │
└──────┬────────┘
       │ Append message
       ▼
┌─────────────────────┐
│ Kafka Partition Log  │
│ (Ordered, Immutable) │
└──────┬──────────────┘
       │ Read by offset
       ▼
┌───────────────┐
│ Consumer      │
└───────────────┘

Myth Busters - 4 Common Misconceptions

Quick: Do you think Kafka deletes messages immediately after a consumer reads them? Commit yes or no.

Common Belief: Kafka removes messages as soon as a consumer reads them to save space.

Reality: Kafka keeps messages for a configured retention time or size, regardless of consumption.

Why it matters: Assuming messages disappear after reading can cause data loss if consumers restart or need to replay messages.

Quick: Do you think a producer must wait for a consumer to be ready before sending messages? Commit yes or no.

Common Belief: Producers can only send messages if consumers are actively listening.

Reality: Producers send messages to Kafka independently; consumers can read them later at their own pace.

Why it matters: Believing this limits system design and prevents using Kafka as a decoupling layer.

Quick: Do you think all consumers share the same reading position in Kafka? Commit yes or no.

Common Belief: All consumers read messages from the same offset, so they see the same messages at the same time.

Reality: Each consumer group tracks its own offset, allowing independent reading progress.

Why it matters: Misunderstanding this can cause confusion about message delivery and duplicate processing.

Quick: Do you think Kafka guarantees message order across all topics and partitions? Commit yes or no.

Common Belief: Kafka always delivers messages in the exact order they were produced across the entire topic.

Reality: Kafka guarantees order only within a single partition, not across multiple partitions.

Why it matters: Expecting global order can lead to bugs when processing messages from multiple partitions.

Expert Zone

1

Producers can batch messages to improve throughput but must balance latency and memory use.

2

Consumers can commit offsets manually or automatically, affecting message processing guarantees.

3

Kafka's idempotent producer feature prevents duplicate messages during retries, important for exactly-once delivery.

When NOT to use

Kafka is not ideal for low-latency request-response patterns or small-scale messaging. Alternatives like RabbitMQ or MQTT may be better for simple queueing or IoT scenarios.

Production Patterns

In production, teams use consumer groups to scale reading, configure retention policies for storage, and monitor offsets to detect lag. They also handle failures with retries and dead-letter queues.

Connections

Message Queueing

Kafka is a type of message queue but uses a log-based design instead of traditional queue semantics.

Understanding Kafka as a durable log helps differentiate it from simple queues and explains its replay and scaling abilities.

Event Sourcing

Kafka's log storage aligns with event sourcing, where all changes are stored as a sequence of events.

Knowing Kafka supports event sourcing clarifies how systems can rebuild state from message history.

Library Book Lending

Like Kafka messages, library books are kept for a time and can be borrowed independently by many readers.

This shows how Kafka allows multiple consumers to read the same data independently without removing it.

Common Pitfalls

#1 Starting consumer without setting offset reset policy causes missing first messages.

Wrong approach:

kafka-console-consumer --bootstrap-server localhost:9092 --topic test

Correct approach:

kafka-console-consumer --bootstrap-server localhost:9092 --topic test --from-beginning

Root cause:

Default offset reset is 'latest', so consumer skips existing messages if no offset is saved.

#2 Assuming messages are deleted after consumption leads to data loss.

Wrong approach:

Relying on Kafka to remove messages immediately after consumer reads them.

Correct approach:

Configure retention policies explicitly and manage consumer offsets properly.

Root cause:

Misunderstanding Kafka's retention model causes incorrect assumptions about message availability.

#3 Producing messages without waiting for acknowledgments risks message loss.

Wrong approach:

kafka-console-producer --broker-list localhost:9092 --topic test (send messages without acks)

Correct approach:

Configure producer with 'acks=all' to ensure messages are fully committed.

Root cause:

Ignoring producer acks sacrifices delivery guarantees for speed.

Key Takeaways

Producing and consuming in Kafka are separate steps that let systems communicate asynchronously.

Kafka stores messages durably in an ordered log, allowing consumers to read at their own pace.

Offsets track consumer progress and ensure messages are read in order within partitions.

Proper configuration of producers and consumers is key to reliable message delivery.

Understanding Kafka's storage and delivery model prevents common mistakes and enables powerful real-world use.

Practice

1. What is the main purpose of producing a message in Kafka?

easy

A. To send data to a Kafka topic

B. To read data from a Kafka topic

C. To delete messages from a Kafka topic

D. To create a new Kafka topic

Solution

1. Step 1: Understand Kafka producing

   Producing means sending messages to a Kafka topic as bytes.

2. Step 2: Differentiate producing from consuming

   Consuming reads messages, but producing sends them.

3. Final Answer:

   To send data to a Kafka topic -> Option A

4. Quick Check:

   Producing = Sending messages [OK]

Hint: Producing means sending messages to Kafka topics [OK]

Common Mistakes:

• Confusing producing with consuming
• Thinking producing deletes messages
• Assuming producing creates topics

2. Which of the following is the correct way to create a Kafka producer in Python using the kafka-python library?

easy

A. producer = KafkaConsumer(topic='mytopic')

B. producer = KafkaConsumer(bootstrap_servers='localhost:9092')

C. producer = KafkaProducer(topic='mytopic')

D. producer = KafkaProducer(bootstrap_servers='localhost:9092')

Solution

1. Step 1: Identify Kafka producer creation syntax

   KafkaProducer is created with bootstrap_servers parameter specifying Kafka server address.

2. Step 2: Eliminate consumer and wrong parameters

   KafkaConsumer is for consuming, not producing. Topic is not a parameter for producer creation.

3. Final Answer:

   producer = KafkaProducer(bootstrap_servers='localhost:9092') -> Option D

4. Quick Check:

   Producer uses KafkaProducer with bootstrap_servers [OK]

Hint: Producer uses KafkaProducer with bootstrap_servers [OK]

Common Mistakes:

• Using KafkaConsumer instead of KafkaProducer
• Passing topic during producer creation
• Missing bootstrap_servers parameter

3. Consider this Python code snippet using kafka-python:

from kafka import KafkaProducer, KafkaConsumer

producer = KafkaProducer(bootstrap_servers='localhost:9092')
producer.send('test-topic', b'Hello Kafka')
producer.flush()

consumer = KafkaConsumer('test-topic', bootstrap_servers='localhost:9092', auto_offset_reset='earliest', consumer_timeout_ms=1000)
for msg in consumer:
    print(msg.value.decode())

What will be the output when this code runs?

medium

A. Error: topic not found

B. Hello Kafka

C. No output, consumer times out

D. b'Hello Kafka'

Solution

1. Step 1: Analyze producer sending message

   Producer sends bytes message b'Hello Kafka' to 'test-topic' and flushes to ensure delivery.

2. Step 2: Analyze consumer reading message

   Consumer reads from 'test-topic' from earliest offset and decodes bytes to string before printing.

3. Final Answer:

   Hello Kafka -> Option B

4. Quick Check:

   Consumer prints decoded message string [OK]

Hint: Consumer decodes bytes to string before printing [OK]

Common Mistakes:

• Expecting raw bytes output
• Assuming consumer misses message due to timeout
• Thinking topic does not exist

4. You wrote this code to consume messages but it raises an error:

consumer = KafkaConsumer('my-topic')
for message in consumer:
    print(message.value.decode())

What is the most likely cause of the error?

medium

A. Using decode() on message.value which is already a string

B. Not calling consumer.poll() before iterating

C. Missing bootstrap_servers parameter when creating KafkaConsumer

D. Topic name should be a list, not a string

Solution

1. Step 1: Check KafkaConsumer creation requirements

   KafkaConsumer requires bootstrap_servers parameter to connect to Kafka cluster.

2. Step 2: Validate other options

   decode() is needed because message.value is bytes; consumer.poll() is optional; topic can be string or list.

3. Final Answer:

   Missing bootstrap_servers parameter when creating KafkaConsumer -> Option C

4. Quick Check:

   KafkaConsumer needs bootstrap_servers [OK]

Hint: Always specify bootstrap_servers when creating KafkaConsumer [OK]

Common Mistakes:

• Omitting bootstrap_servers parameter
• Assuming message.value is string without decoding
• Thinking poll() is mandatory before iteration

5. You want to produce and consume the first message in a Kafka topic named 'orders'. Which sequence of steps correctly achieves this in Python using kafka-python?

hard

A. Create KafkaProducer with bootstrap_servers, send message as bytes, flush; create KafkaConsumer with bootstrap_servers and auto_offset_reset='earliest', then read and decode messages

B. Create KafkaConsumer first, then KafkaProducer; send message as string; consumer reads without decoding

C. Create KafkaProducer without bootstrap_servers, send string message; create KafkaConsumer with bootstrap_servers, read messages as bytes

D. Create KafkaProducer with topic parameter, send message; create KafkaConsumer without bootstrap_servers, read and decode messages

Solution

1. Step 1: Properly create KafkaProducer and send message

   KafkaProducer must be created with bootstrap_servers; messages sent as bytes and flushed to ensure delivery.

2. Step 2: Properly create KafkaConsumer and read messages

   KafkaConsumer needs bootstrap_servers and auto_offset_reset='earliest' to read from the first message; decode bytes to string when reading.

3. Final Answer:

   Create KafkaProducer with bootstrap_servers, send message as bytes, flush; create KafkaConsumer with bootstrap_servers and auto_offset_reset='earliest', then read and decode messages -> Option A

4. Quick Check:

   Correct producer and consumer setup with bytes and decoding [OK]

Hint: Producer sends bytes and flushes; consumer reads earliest and decodes [OK]

Common Mistakes:

• Creating producer or consumer without bootstrap_servers
• Sending string instead of bytes
• Not setting auto_offset_reset to 'earliest'
• Not decoding message bytes before printing