Disk I/O optimization in Kafka - Step-by-Step Execution

Process Flow - Disk I/O optimization

Start: Kafka Broker receives message

↓

Write message to page cache

↓

Batch multiple writes in memory

↓

Flush batch to disk asynchronously

↓

Disk writes complete

↓

Acknowledge message write

↓

End

Kafka batches messages in memory and writes them asynchronously to disk to reduce slow disk I/O operations.

Execution Sample

Kafka

producer.send(record)
// Broker batches messages
// Writes batch to disk asynchronously
// Acknowledges producer

This code shows Kafka producer sending a message that the broker batches and writes to disk asynchronously.

Process Table

Step	Action	State Change	Disk I/O	Output
1	Producer sends message	Message added to broker batch	No disk write yet	Message queued
2	Batch reaches size/time limit	Batch ready to flush	No disk write yet	Batch prepared
3	Flush batch asynchronously	Batch sent to OS page cache	Disk write started asynchronously	Flush started
4	Disk write completes	Batch persisted on disk	Disk write done	Acknowledgement sent
5	Producer receives ack	Producer notified of success	No disk write	Send complete
6	New messages arrive	Repeat batching process	No disk write yet	New batch queued

💡 Process repeats continuously as messages arrive and batches flush asynchronously to optimize disk I/O.

Status Tracker

Variable	Start	After Step 1	After Step 2	After Step 3	After Step 4	After Step 5
Batch	empty	1 message	batch full	flushing	flushed	empty
Disk Write	idle	idle	idle	writing	done	idle
Producer Ack	none	none	none	none	sent	received

Key Moments - 3 Insights

Why doesn't Kafka write each message immediately to disk?

Kafka batches messages in memory first (see Step 2 in execution_table) to reduce slow disk writes and improve throughput.

What does asynchronous disk write mean in Kafka's context?

It means Kafka tells the OS to write data to disk but continues processing without waiting (Step 3), improving performance.

When does the producer get notified that the message is saved?

After the disk write completes (Step 4), Kafka sends an acknowledgement to the producer (Step 5).

Visual Quiz - 3 Questions

Look at the execution_table, at which step does Kafka start writing the batch to disk?

A Step 2

B Step 3

C Step 4

D Step 5

💡 Hint

Check the 'Disk I/O' column in execution_table for when 'Disk write started asynchronously' occurs.

According to variable_tracker, what is the state of 'Batch' after Step 4?

A flushing

B 1 message

C flushed

D batch full

💡 Hint

Look at the 'Batch' row and the column 'After Step 4' in variable_tracker.

If Kafka wrote each message immediately without batching, how would the 'Disk Write' state change in execution_table?

A Disk write would happen more frequently and slower

B Disk write would happen less frequently

C Disk write would be asynchronous

D Disk write would be skipped

💡 Hint

Consider the purpose of batching shown in execution_table and key_moments about reducing disk I/O.

Concept Snapshot

Disk I/O optimization in Kafka:
- Messages batch in memory before disk write
- Batches flush asynchronously to OS page cache
- Reduces slow disk operations
- Producer gets ack after disk write completes
- Improves throughput and latency

Full Transcript

Kafka optimizes disk input/output by batching messages in memory and writing them asynchronously to disk. When a producer sends a message, Kafka adds it to a batch in memory. Once the batch reaches a size or time limit, Kafka flushes it asynchronously to the OS page cache, which then writes to disk. This asynchronous write allows Kafka to continue processing without waiting for slow disk operations. After the disk write completes, Kafka acknowledges the producer that the message is safely stored. This process repeats continuously, improving performance by reducing frequent slow disk writes.