Producer throughput optimization in Kafka - Step-by-Step Execution

Process Flow - Producer throughput optimization

Start Producer

↓

Set batch.size

↓

Set linger.ms

↓

Enable compression

↓

Send messages in batches

↓

Wait linger.ms or batch full

↓

Send batch to broker

↓

Repeat for next messages

↓

End Producer

The producer collects messages into batches, waits briefly to fill batches, compresses data, then sends batches to the broker to improve throughput.

Execution Sample

Kafka

props = {
  'batch_size': 16384,
  'linger_ms': 5,
  'compression_type': 'snappy'
}
producer = KafkaProducer(**props)
producer.send('topic', b'message')

This code sets producer properties to batch messages, wait 5 ms before sending, compress data, and then sends a message.

Process Table

Step	Action	Batch Size (bytes)	Linger Time (ms)	Compression	Batch Status	Message Sent
1	Initialize producer with batch.size=16384, linger.ms=5, compression=snappy	16384	5	snappy	Empty	No
2	Send first message (100 bytes)	16384	5	snappy	Batch has 100 bytes	No
3	Wait for linger.ms (5 ms) to fill batch	16384	5	snappy	Batch has 100 bytes	No
4	Linger time reached, send batch	16384	5	snappy	Batch sent with 100 bytes	Yes
5	Send next message (200 bytes)	16384	5	snappy	Batch has 200 bytes	No
6	Batch not full, wait linger.ms	16384	5	snappy	Batch has 200 bytes	No
7	Linger time reached, send batch	16384	5	snappy	Batch sent with 200 bytes	Yes
8	Send messages rapidly to fill batch	16384	5	snappy	Batch full (16384 bytes)	No
9	Batch full, send immediately	16384	5	snappy	Batch sent full	Yes
10	No more messages, producer closes	16384	5	snappy	No batch	No

💡 Producer stops after sending all messages and closing.

Status Tracker

Variable	Start	After Step 2	After Step 4	After Step 5	After Step 7	After Step 8	Final
batch_size	16384	16384	16384	16384	16384	16384	16384
linger_ms	5	5	5	5	5	5	5
compression_type	snappy	snappy	snappy	snappy	snappy	snappy	snappy
batch_bytes	0	100	0	200	0	16384	0
messages_sent	0	0	1	1	2	2	3

Key Moments - 3 Insights

Why does the producer wait for linger.ms before sending a batch even if the batch is not full?

The producer waits linger.ms to allow more messages to accumulate in the batch, improving throughput by sending larger batches. See execution_table rows 3 and 6 where batch is sent after linger.ms even if not full.

What happens when the batch size limit is reached before linger.ms?

When batch size reaches the configured limit, the producer sends the batch immediately without waiting for linger.ms. This is shown in execution_table row 9.

How does compression affect the batch sending process?

Compression reduces the size of the batch data sent to the broker, improving network efficiency. The compression type is set at initialization (step 1) and applies to all batches sent.

Visual Quiz - 3 Questions

Look at the execution_table, what is the batch_bytes value after step 2?

A 100

B 0

C 16384

D 200

💡 Hint

Check the 'Batch Status' column at step 2 in execution_table.

At which step does the producer send a batch because the batch is full?

A Step 7

B Step 9

C Step 4

D Step 6

💡 Hint

Look for 'Batch full' in the 'Batch Status' column in execution_table.

If linger.ms was set to 0, how would the sending behavior change?

A Batch size would increase automatically.

B Messages would wait longer before sending.

C Messages would be sent immediately without waiting to batch.

D Compression would be disabled.

💡 Hint

Refer to the role of linger.ms in execution_table steps 3 and 6.

Concept Snapshot

Producer throughput optimization:
- batch.size sets max batch bytes
- linger.ms sets max wait time before sending
- compression.type reduces data size
- Producer batches messages to send fewer, larger requests
- Batches send when full or linger.ms expires
- Improves network and broker efficiency

Full Transcript

This visual trace shows how a Kafka producer optimizes throughput by batching messages. The producer is configured with batch.size 16384 bytes, linger.ms 5 milliseconds, and compression type snappy. When messages arrive, they are added to a batch. The producer waits up to linger.ms to fill the batch before sending. If the batch fills before linger.ms, it sends immediately. Compression reduces data size sent. The execution table tracks batch size, linger time, compression, batch status, and when messages are sent. Variable tracking shows batch bytes and messages sent over steps. Key moments clarify why linger.ms waits help throughput, how batch size triggers immediate send, and the role of compression. The quiz tests understanding of batch size after steps, when full batch sends occur, and effects of changing linger.ms. The snapshot summarizes key settings and behavior for throughput optimization.