Producer throughput optimization in Kafka - Step-by-Step Execution

Process Flow - Producer throughput optimization

Start Producer

↓

Set batch.size

↓

Set linger.ms

↓

Enable compression

↓

Send messages in batches

↓

Wait linger.ms or batch full

↓

Send batch to broker

↓

Repeat for next messages

↓

End Producer

The producer collects messages into batches, waits briefly to fill batches, compresses data, then sends batches to the broker to improve throughput.

Execution Sample

Kafka

props = {
  'batch_size': 16384,
  'linger_ms': 5,
  'compression_type': 'snappy'
}
producer = KafkaProducer(**props)
producer.send('topic', b'message')

This code sets producer properties to batch messages, wait 5 ms before sending, compress data, and then sends a message.

Process Table

Step	Action	Batch Size (bytes)	Linger Time (ms)	Compression	Batch Status	Message Sent
1	Initialize producer with batch.size=16384, linger.ms=5, compression=snappy	16384	5	snappy	Empty	No
2	Send first message (100 bytes)	16384	5	snappy	Batch has 100 bytes	No
3	Wait for linger.ms (5 ms) to fill batch	16384	5	snappy	Batch has 100 bytes	No
4	Linger time reached, send batch	16384	5	snappy	Batch sent with 100 bytes	Yes
5	Send next message (200 bytes)	16384	5	snappy	Batch has 200 bytes	No
6	Batch not full, wait linger.ms	16384	5	snappy	Batch has 200 bytes	No
7	Linger time reached, send batch	16384	5	snappy	Batch sent with 200 bytes	Yes
8	Send messages rapidly to fill batch	16384	5	snappy	Batch full (16384 bytes)	No
9	Batch full, send immediately	16384	5	snappy	Batch sent full	Yes
10	No more messages, producer closes	16384	5	snappy	No batch	No

💡 Producer stops after sending all messages and closing.

Status Tracker

Variable	Start	After Step 2	After Step 4	After Step 5	After Step 7	After Step 8	Final
batch_size	16384	16384	16384	16384	16384	16384	16384
linger_ms	5	5	5	5	5	5	5
compression_type	snappy	snappy	snappy	snappy	snappy	snappy	snappy
batch_bytes	0	100	0	200	0	16384	0
messages_sent	0	0	1	1	2	2	3

Key Moments - 3 Insights

Why does the producer wait for linger.ms before sending a batch even if the batch is not full?

The producer waits linger.ms to allow more messages to accumulate in the batch, improving throughput by sending larger batches. See execution_table rows 3 and 6 where batch is sent after linger.ms even if not full.

What happens when the batch size limit is reached before linger.ms?

When batch size reaches the configured limit, the producer sends the batch immediately without waiting for linger.ms. This is shown in execution_table row 9.

How does compression affect the batch sending process?

Compression reduces the size of the batch data sent to the broker, improving network efficiency. The compression type is set at initialization (step 1) and applies to all batches sent.

Visual Quiz - 3 Questions

Look at the execution_table, what is the batch_bytes value after step 2?

A 100

B 0

C 16384

D 200

💡 Hint

Check the 'Batch Status' column at step 2 in execution_table.

At which step does the producer send a batch because the batch is full?

A Step 7

B Step 9

C Step 4

D Step 6

💡 Hint

Look for 'Batch full' in the 'Batch Status' column in execution_table.

If linger.ms was set to 0, how would the sending behavior change?

A Batch size would increase automatically.

B Messages would wait longer before sending.

C Messages would be sent immediately without waiting to batch.

D Compression would be disabled.

💡 Hint

Refer to the role of linger.ms in execution_table steps 3 and 6.

Concept Snapshot

Producer throughput optimization:
- batch.size sets max batch bytes
- linger.ms sets max wait time before sending
- compression.type reduces data size
- Producer batches messages to send fewer, larger requests
- Batches send when full or linger.ms expires
- Improves network and broker efficiency

Full Transcript

This visual trace shows how a Kafka producer optimizes throughput by batching messages. The producer is configured with batch.size 16384 bytes, linger.ms 5 milliseconds, and compression type snappy. When messages arrive, they are added to a batch. The producer waits up to linger.ms to fill the batch before sending. If the batch fills before linger.ms, it sends immediately. Compression reduces data size sent. The execution table tracks batch size, linger time, compression, batch status, and when messages are sent. Variable tracking shows batch bytes and messages sent over steps. Key moments clarify why linger.ms waits help throughput, how batch size triggers immediate send, and the role of compression. The quiz tests understanding of batch size after steps, when full batch sends occur, and effects of changing linger.ms. The snapshot summarizes key settings and behavior for throughput optimization.

Practice

1. What is the main benefit of increasing the batch.size setting in a Kafka producer?

easy

A. It guarantees message delivery to all replicas.

B. It compresses messages to save disk space.

C. It allows sending more messages together, improving throughput.

D. It reduces the number of partitions in a topic.

Solution

1. Step 1: Understand batch.size role

   The batch.size controls how many messages the producer groups before sending.

2. Step 2: Effect on throughput

   Grouping more messages reduces network calls, increasing throughput.

3. Final Answer:

   It allows sending more messages together, improving throughput. -> Option C

4. Quick Check:

   batch.size increases throughput [OK]

Hint: More batch size means fewer sends, better throughput [OK]

Common Mistakes:

• Confusing batch.size with message compression
• Thinking batch.size controls message durability
• Assuming batch.size changes topic partitions

2. Which of the following is the correct way to set the linger time to 100 milliseconds in Kafka producer configuration?

easy

A. "linger.ms = 100"

B. "linger_time = 100"

C. "lingerMs: 100"

D. "linger = 100ms"

Solution

1. Step 1: Identify correct property name

   The Kafka producer uses linger.ms to set linger time in milliseconds.

2. Step 2: Confirm correct syntax

   The correct syntax is linger.ms = 100, using dot notation and milliseconds.

3. Final Answer:

   "linger.ms = 100" -> Option A

4. Quick Check:

   linger.ms sets linger time [OK]

Hint: Use exact property names with dots, like linger.ms [OK]

Common Mistakes:

• Using underscores instead of dots
• Adding units like 'ms' in the value
• Using camelCase or colon syntax incorrectly

3. Given this Kafka producer config snippet:

props.put("batch.size", 16384);
props.put("linger.ms", 5);
props.put("acks", "all");

What is the expected effect on message sending?

medium

A. Messages batch up to 16KB or wait 5ms before sending, with full acknowledgment.

B. Messages are compressed automatically before sending.

C. Messages are sent immediately without batching.

D. Producer waits for only one replica acknowledgment.

Solution

1. Step 1: Analyze batch.size and linger.ms

   Batch size 16384 bytes means messages batch up to 16KB or linger 5ms before sending.

2. Step 2: Understand acks setting

   acks=all means producer waits for all replicas to confirm, ensuring durability.

3. Final Answer:

   Messages batch up to 16KB or wait 5ms before sending, with full acknowledgment. -> Option A

4. Quick Check:

   batch.size + linger.ms + acks=all = Messages batch up to 16KB or wait 5ms before sending, with full acknowledgment. [OK]

Hint: Batch size + linger.ms controls send timing, acks controls durability [OK]

Common Mistakes:

• Assuming messages send immediately ignoring linger.ms
• Confusing acks=all with acks=1
• Thinking compression happens automatically without config

4. You have this Kafka producer config:

props.put("batch.size", 32768);
props.put("linger.ms", 0);
props.put("acks", "0");

What is the main problem with this setup if you want high throughput and reliability?

medium

A. linger.ms=0 disables batching, reducing throughput.

B. acks=0 means no acknowledgment, risking message loss.

C. batch.size is too large and causes errors.

D. The config syntax is invalid and will cause exceptions.

Solution

1. Step 1: Check acks setting impact

   acks=0 means producer does not wait for any acknowledgment, risking message loss.

2. Step 2: Evaluate throughput vs reliability

   While batch.size is large and linger.ms=0 disables waiting, the main reliability issue is acks=0.

3. Final Answer:

   acks=0 means no acknowledgment, risking message loss. -> Option B

4. Quick Check:

   acks=0 risks loss despite throughput [OK]

Hint: acks=0 disables reliability, not good for safe throughput [OK]

Common Mistakes:

• Thinking linger.ms=0 disables batching completely
• Assuming batch.size too large causes errors
• Confusing config syntax as invalid

5. You want to optimize a Kafka producer for maximum throughput but still ensure messages are not lost. Which combination of settings is best?

hard

A. batch.size=512, linger.ms=50, acks=0, compression.type=lz4

B. batch.size=1024, linger.ms=0, acks=0, compression.type=none

C. batch.size=16384, linger.ms=0, acks=1, compression.type=gzip

D. batch.size=65536, linger.ms=20, acks=all, compression.type=snappy

Solution

1. Step 1: Maximize throughput with batching and linger

   Large batch.size (65536) and linger.ms (20ms) allow grouping many messages before sending.

2. Step 2: Ensure reliability and compression

   acks=all ensures all replicas confirm, preventing loss. Compression (snappy) reduces data size, improving throughput.

3. Final Answer:

   batch.size=65536, linger.ms=20, acks=all, compression.type=snappy -> Option D

4. Quick Check:

   Large batch + linger + acks=all + compression = batch.size=65536, linger.ms=20, acks=all, compression.type=snappy [OK]

Hint: Big batch + linger + acks=all + compression = best throughput + safety [OK]

Common Mistakes:

• Choosing acks=0 losing reliability
• Using too small batch sizes
• Ignoring compression benefits