Producer throughput optimization in Kafka - Commands & Configuration

Introduction

Sending messages quickly to Kafka can be tricky. If your producer sends messages one by one, it slows down. Optimizing throughput means sending many messages efficiently without losing data.

When your app needs to send thousands of messages per second to Kafka.

When you notice your producer is slow and can't keep up with incoming data.

When you want to reduce network overhead by sending messages in batches.

When you want to balance speed and reliability in message delivery.

When you want to tune Kafka producer settings to handle high traffic smoothly.

Config File - producer.properties

producer.properties

bootstrap.servers=localhost:9092
acks=1
batch.size=32768
linger.ms=10
compression.type=snappy
buffer.memory=67108864
max.in.flight.requests.per.connection=5
retries=3

bootstrap.servers: Kafka server address to connect.
acks=1: Wait for leader to acknowledge for faster response.
batch.size: Size in bytes to batch messages before sending.
linger.ms: Time to wait to batch more messages before sending.
compression.type: Compress messages to reduce size.
buffer.memory: Memory for buffering messages.
max.in.flight.requests.per.connection: Number of unacknowledged requests allowed.
retries: Number of retry attempts on failure.

Commands

Starts the Kafka producer using the optimized settings from producer.properties to send messages to 'example-topic'.

Terminal

kafka-console-producer --broker-list localhost:9092 --topic example-topic --producer.config producer.properties

Expected Output

> This is a test message

> Another message

→

--broker-list - Specifies Kafka server addresses to connect.

→

--topic - Specifies the topic to send messages to.

→

--producer.config - Loads producer configuration from a file.

Runs a performance test to send 100,000 messages of 100 bytes each with optimized producer settings to measure throughput.

Terminal

kafka-run-class kafka.tools.ProducerPerformance --topic example-topic --num-records 100000 --record-size 100 --throughput -1 --producer-props bootstrap.servers=localhost:9092 acks=1 batch.size=32768 linger.ms=10 compression.type=snappy

Expected Output

100000 records sent, 100000 records acknowledged, 0 errors

Throughput: 50000 records/sec

Average latency: 5 ms

→

--num-records - Number of messages to send.

→

--record-size - Size of each message in bytes.

→

--throughput - Messages per second to send; -1 means unlimited.

Checks the consumer group lag to ensure messages are being consumed timely after high throughput production.

Terminal

kafka-consumer-groups --bootstrap-server localhost:9092 --describe --group example-group

Expected Output

TOPIC          PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG  CONSUMER-ID  HOST  CLIENT-ID

example-topic  0          100000          100000          0    consumer-1   /127.0.0.1  consumer-1

→

--bootstrap-server - Kafka server address.

→

--describe - Shows detailed info about the consumer group.

→

--group - Specifies the consumer group to check.

Key Concept

Batching messages and tuning producer settings like linger.ms and batch.size greatly improve Kafka producer throughput.

Common Mistakes

Setting linger.ms to 0 to send messages immediately.

This disables batching, causing many small requests and lowering throughput.

Set linger.ms to a small positive value like 10 ms to allow batching of messages.

Using a very small batch.size like 1 KB.

Small batches increase network overhead and reduce throughput.

Use a larger batch.size like 32 KB or more to send bigger batches efficiently.

Not enabling compression on messages.

Uncompressed messages use more bandwidth and slow down throughput.

Enable compression.type like 'snappy' to reduce message size and improve speed.

Summary

Configure producer.properties with batch.size, linger.ms, and compression.type for better throughput.

Use kafka-console-producer with the config file to send messages efficiently.

Run kafka-run-class ProducerPerformance to test and measure producer throughput.

Check consumer lag with kafka-consumer-groups to ensure consumers keep up.

Practice

1. What is the main benefit of increasing the batch.size setting in a Kafka producer?

easy

A. It guarantees message delivery to all replicas.

B. It compresses messages to save disk space.

C. It allows sending more messages together, improving throughput.

D. It reduces the number of partitions in a topic.

Solution

1. Step 1: Understand batch.size role

   The batch.size controls how many messages the producer groups before sending.

2. Step 2: Effect on throughput

   Grouping more messages reduces network calls, increasing throughput.

3. Final Answer:

   It allows sending more messages together, improving throughput. -> Option C

4. Quick Check:

   batch.size increases throughput [OK]

Hint: More batch size means fewer sends, better throughput [OK]

Common Mistakes:

• Confusing batch.size with message compression
• Thinking batch.size controls message durability
• Assuming batch.size changes topic partitions

2. Which of the following is the correct way to set the linger time to 100 milliseconds in Kafka producer configuration?

easy

A. "linger.ms = 100"

B. "linger_time = 100"

C. "lingerMs: 100"

D. "linger = 100ms"

Solution

1. Step 1: Identify correct property name

   The Kafka producer uses linger.ms to set linger time in milliseconds.

2. Step 2: Confirm correct syntax

   The correct syntax is linger.ms = 100, using dot notation and milliseconds.

3. Final Answer:

   "linger.ms = 100" -> Option A

4. Quick Check:

   linger.ms sets linger time [OK]

Hint: Use exact property names with dots, like linger.ms [OK]

Common Mistakes:

• Using underscores instead of dots
• Adding units like 'ms' in the value
• Using camelCase or colon syntax incorrectly

3. Given this Kafka producer config snippet:

props.put("batch.size", 16384);
props.put("linger.ms", 5);
props.put("acks", "all");

What is the expected effect on message sending?

medium

A. Messages batch up to 16KB or wait 5ms before sending, with full acknowledgment.

B. Messages are compressed automatically before sending.

C. Messages are sent immediately without batching.

D. Producer waits for only one replica acknowledgment.

Solution

1. Step 1: Analyze batch.size and linger.ms

   Batch size 16384 bytes means messages batch up to 16KB or linger 5ms before sending.

2. Step 2: Understand acks setting

   acks=all means producer waits for all replicas to confirm, ensuring durability.

3. Final Answer:

   Messages batch up to 16KB or wait 5ms before sending, with full acknowledgment. -> Option A

4. Quick Check:

   batch.size + linger.ms + acks=all = Messages batch up to 16KB or wait 5ms before sending, with full acknowledgment. [OK]

Hint: Batch size + linger.ms controls send timing, acks controls durability [OK]

Common Mistakes:

• Assuming messages send immediately ignoring linger.ms
• Confusing acks=all with acks=1
• Thinking compression happens automatically without config

4. You have this Kafka producer config:

props.put("batch.size", 32768);
props.put("linger.ms", 0);
props.put("acks", "0");

What is the main problem with this setup if you want high throughput and reliability?

medium

A. linger.ms=0 disables batching, reducing throughput.

B. acks=0 means no acknowledgment, risking message loss.

C. batch.size is too large and causes errors.

D. The config syntax is invalid and will cause exceptions.

Solution

1. Step 1: Check acks setting impact

   acks=0 means producer does not wait for any acknowledgment, risking message loss.

2. Step 2: Evaluate throughput vs reliability

   While batch.size is large and linger.ms=0 disables waiting, the main reliability issue is acks=0.

3. Final Answer:

   acks=0 means no acknowledgment, risking message loss. -> Option B

4. Quick Check:

   acks=0 risks loss despite throughput [OK]

Hint: acks=0 disables reliability, not good for safe throughput [OK]

Common Mistakes:

• Thinking linger.ms=0 disables batching completely
• Assuming batch.size too large causes errors
• Confusing config syntax as invalid

5. You want to optimize a Kafka producer for maximum throughput but still ensure messages are not lost. Which combination of settings is best?

hard

A. batch.size=512, linger.ms=50, acks=0, compression.type=lz4

B. batch.size=1024, linger.ms=0, acks=0, compression.type=none

C. batch.size=16384, linger.ms=0, acks=1, compression.type=gzip

D. batch.size=65536, linger.ms=20, acks=all, compression.type=snappy

Solution

1. Step 1: Maximize throughput with batching and linger

   Large batch.size (65536) and linger.ms (20ms) allow grouping many messages before sending.

2. Step 2: Ensure reliability and compression

   acks=all ensures all replicas confirm, preventing loss. Compression (snappy) reduces data size, improving throughput.

3. Final Answer:

   batch.size=65536, linger.ms=20, acks=all, compression.type=snappy -> Option D

4. Quick Check:

   Large batch + linger + acks=all + compression = batch.size=65536, linger.ms=20, acks=all, compression.type=snappy [OK]

Hint: Big batch + linger + acks=all + compression = best throughput + safety [OK]

Common Mistakes:

• Choosing acks=0 losing reliability
• Using too small batch sizes
• Ignoring compression benefits