Consumer throughput optimization in Kafka - Cheat Sheet & Quick Revision

Recall & Review

beginner

What is consumer throughput in Kafka?

Consumer throughput is the rate at which a Kafka consumer reads and processes messages from Kafka topics.

intermediate

How does increasing the number of consumer threads affect throughput?

Increasing consumer threads can improve throughput by parallelizing message processing, but too many threads may cause contention and reduce performance.

intermediate

What role does 'fetch.min.bytes' play in consumer throughput?

'fetch.min.bytes' sets the minimum amount of data the consumer fetches in one request, helping to batch messages and improve throughput by reducing network calls.

intermediate

Why is 'max.poll.records' important for throughput optimization?

'max.poll.records' controls how many records a consumer fetches per poll. Increasing it can improve throughput by processing more messages at once, but may increase processing latency.

advanced

How does enabling consumer 'async commit' affect throughput?

Async commit allows the consumer to commit offsets without blocking message processing, increasing throughput by reducing wait times.

Which Kafka consumer setting controls the maximum number of records returned in a single poll?

A max.poll.records

B fetch.min.bytes

C session.timeout.ms

D auto.offset.reset

'max.poll.records' sets the maximum number of records returned in one poll call.

What happens if 'fetch.min.bytes' is set too high?

A Consumer fetches smaller batches, reducing throughput

B Consumer waits longer to fetch enough data, increasing latency

C Consumer stops fetching data

D Consumer commits offsets more frequently

A high 'fetch.min.bytes' means the consumer waits to accumulate enough data before fetching, which can increase latency.

Why might increasing the number of consumer threads not always improve throughput?

A Because of thread contention and resource limits

B Because Kafka limits the number of threads

C Because threads slow down network speed

D Because more threads cause data loss

Too many threads can cause contention for CPU and memory, reducing overall performance.

What is the benefit of asynchronous offset commits?

A They increase message size

B They guarantee exactly-once processing

C They reduce blocking and improve throughput

D They disable auto commit

Async commits allow the consumer to continue processing without waiting for the commit to complete, improving throughput.

Which setting helps batch messages to reduce network calls?

A auto.commit.interval.ms

B max.poll.records

C heartbeat.interval.ms

D fetch.min.bytes

'fetch.min.bytes' controls the minimum batch size fetched, reducing network overhead.

Explain how adjusting 'max.poll.records' and 'fetch.min.bytes' can optimize Kafka consumer throughput.

Think about how batching messages reduces network calls and processing overhead.

Describe the impact of consumer thread count on throughput and potential downsides.

Consider how too many cooks spoil the broth.