Consumer throughput optimization in Kafka - Commands & Configuration

Introduction

When many messages arrive quickly in Kafka, consumers can get slow or overwhelmed. Optimizing consumer throughput helps process messages faster and keeps your app responsive.

When your Kafka consumer lags behind and can't keep up with incoming messages.

When you want to reduce the time it takes to process batches of messages.

When you need to balance load across multiple consumer instances efficiently.

When you want to tune your consumer to handle bursts of traffic without crashing.

When you want to improve resource use by adjusting how many messages your consumer fetches at once.

Commands

Starts a Kafka consumer that fetches up to 500 messages per poll to increase throughput by processing larger batches.

Terminal

kafka-console-consumer --bootstrap-server localhost:9092 --topic example-topic --group example-group --max-poll-records 500

Expected Output

This consumer will print messages from 'example-topic' as they arrive, up to 500 at a time.

→

--max-poll-records - Limits the number of records returned in a single poll to improve batch processing efficiency.

Checks the current lag and partition assignment of the consumer group to verify if throughput improvements reduce lag.

Terminal

kafka-consumer-groups --bootstrap-server localhost:9092 --describe --group example-group

Expected Output

TOPIC           PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG  CONSUMER-ID                                    HOST            CLIENT-ID

example-topic   0          15000           15000           0    consumer-1-12345-67890                        /127.0.0.1      consumer-1

→

--describe - Shows detailed information about consumer group status.

Configures the consumer to wait up to 500ms to fetch at least 1MB of data before returning, improving throughput by reducing frequent small fetches.

Terminal

kafka-console-consumer --bootstrap-server localhost:9092 --topic example-topic --group example-group --fetch-min-bytes 1048576 --fetch-max-wait-ms 500

Expected Output

This consumer will print messages from 'example-topic' in larger chunks, waiting briefly to gather more data.

→

--fetch-min-bytes - Sets minimum amount of data the server should return for a fetch request.

→

--fetch-max-wait-ms - Sets maximum wait time to accumulate fetch.min.bytes of data.

Key Concept

If you remember nothing else from this pattern, remember: tuning batch size and fetch timing controls how fast your Kafka consumer processes messages.

Common Mistakes

Setting max.poll.records too high without enough processing power.

The consumer may get overwhelmed and slow down or crash because it can't handle large batches quickly.

Increase max.poll.records gradually and monitor resource use to find a balance.

Not adjusting fetch.min.bytes and fetch.max.wait.ms together.

The consumer might fetch too little data too often, reducing throughput benefits.

Set fetch.min.bytes to a meaningful size and fetch.max.wait.ms to a small delay to batch messages efficiently.

Ignoring consumer lag after tuning throughput settings.

Lag indicates the consumer is still behind, so throughput improvements may not be effective.

Use kafka-consumer-groups to monitor lag and adjust settings accordingly.

Summary

Use --max-poll-records to control how many messages the consumer processes per poll.

Adjust --fetch-min-bytes and --fetch-max-wait-ms to batch message fetching efficiently.

Monitor consumer lag with kafka-consumer-groups to verify throughput improvements.