Consumer throughput optimization in Kafka - Cheat Sheet & Quick Revision

Recall & Review

beginner

What is consumer throughput in Kafka?

Consumer throughput is the rate at which a Kafka consumer reads and processes messages from Kafka topics.

intermediate

How does increasing the number of consumer threads affect throughput?

Increasing consumer threads can improve throughput by parallelizing message processing, but too many threads may cause contention and reduce performance.

intermediate

What role does 'fetch.min.bytes' play in consumer throughput?

'fetch.min.bytes' sets the minimum amount of data the consumer fetches in one request, helping to batch messages and improve throughput by reducing network calls.

intermediate

Why is 'max.poll.records' important for throughput optimization?

'max.poll.records' controls how many records a consumer fetches per poll. Increasing it can improve throughput by processing more messages at once, but may increase processing latency.

advanced

How does enabling consumer 'async commit' affect throughput?

Async commit allows the consumer to commit offsets without blocking message processing, increasing throughput by reducing wait times.

Which Kafka consumer setting controls the maximum number of records returned in a single poll?

A max.poll.records

B fetch.min.bytes

C session.timeout.ms

D auto.offset.reset

'max.poll.records' sets the maximum number of records returned in one poll call.

What happens if 'fetch.min.bytes' is set too high?

A Consumer fetches smaller batches, reducing throughput

B Consumer waits longer to fetch enough data, increasing latency

C Consumer stops fetching data

D Consumer commits offsets more frequently

A high 'fetch.min.bytes' means the consumer waits to accumulate enough data before fetching, which can increase latency.

Why might increasing the number of consumer threads not always improve throughput?

A Because of thread contention and resource limits

B Because Kafka limits the number of threads

C Because threads slow down network speed

D Because more threads cause data loss

Too many threads can cause contention for CPU and memory, reducing overall performance.

What is the benefit of asynchronous offset commits?

A They increase message size

B They guarantee exactly-once processing

C They reduce blocking and improve throughput

D They disable auto commit

Async commits allow the consumer to continue processing without waiting for the commit to complete, improving throughput.

Which setting helps batch messages to reduce network calls?

A auto.commit.interval.ms

B max.poll.records

C heartbeat.interval.ms

D fetch.min.bytes

'fetch.min.bytes' controls the minimum batch size fetched, reducing network overhead.

Explain how adjusting 'max.poll.records' and 'fetch.min.bytes' can optimize Kafka consumer throughput.

Think about how batching messages reduces network calls and processing overhead.

Describe the impact of consumer thread count on throughput and potential downsides.

Consider how too many cooks spoil the broth.

Practice

1. What is the primary purpose of increasing max.poll.records in a Kafka consumer configuration?

easy

A. To decrease the consumer's memory usage

B. To increase the number of records fetched in one poll, improving throughput

C. To reduce the number of partitions assigned to the consumer

D. To increase the consumer's connection timeout

Solution

1. Step 1: Understand max.poll.records role

   This setting controls how many records the consumer fetches in a single poll call.

2. Step 2: Effect on throughput

   Increasing this value allows the consumer to process more records per poll, which can improve throughput by reducing overhead.

3. Final Answer:

   To increase the number of records fetched in one poll, improving throughput -> Option B

4. Quick Check:

   max.poll.records controls batch size [OK]

Hint: More records per poll means higher throughput [OK]

Common Mistakes:

• Confusing max.poll.records with timeout settings
• Thinking it reduces memory usage
• Assuming it changes partition assignment

2. Which of the following is the correct way to set fetch.min.bytes to 50000 in a Kafka consumer properties file?

easy

A. fetch.min.bytes=50000

B. fetch.min.bytes: 50000

C. "fetch.min.bytes" = 50000

D. fetch_min_bytes=50000

Solution

1. Step 1: Identify Kafka properties file syntax

   Kafka consumer properties use key=value pairs without quotes or underscores.

2. Step 2: Match correct syntax

   fetch.min.bytes=50000 uses the correct key with dot notation and equals sign, matching Kafka config style.

3. Final Answer:

   fetch.min.bytes=50000 -> Option A

4. Quick Check:

   Kafka config uses key=value format [OK]

Hint: Use dot notation and equals sign for Kafka configs [OK]

Common Mistakes:

• Using colon instead of equals sign
• Adding quotes around keys
• Replacing dots with underscores

3. Given the following Kafka consumer configuration snippet:

props.put("max.poll.records", "1000");
props.put("fetch.min.bytes", "1048576");

What is the expected effect on consumer throughput and latency?

medium

A. Lower throughput and lower latency

B. Higher throughput and lower latency

C. No change in throughput or latency

D. Higher throughput with potentially increased latency

Solution

1. Step 1: Analyze max.poll.records setting

   Setting it to 1000 means the consumer fetches up to 1000 records per poll, increasing batch size and throughput.

2. Step 2: Analyze fetch.min.bytes setting

   Setting it to 1MB (1048576 bytes) means the consumer waits to fetch at least 1MB of data, which can increase latency waiting for enough data.

3. Final Answer:

   Higher throughput with potentially increased latency -> Option D

4. Quick Check:

   Large batch size = higher throughput, bigger fetch.min.bytes = higher latency [OK]

Hint: Large batch size boosts throughput but may add latency [OK]

Common Mistakes:

• Assuming latency always decreases with throughput
• Ignoring fetch.min.bytes impact
• Confusing max.poll.records with fetch.max.bytes

4. You notice your Kafka consumer is lagging and not processing messages fast enough. Which of the following configuration mistakes could cause this issue?

medium

A. Setting max.poll.records too low, like 1

B. Setting fetch.min.bytes to 1

C. Setting max.poll.interval.ms to a very high value

D. Setting enable.auto.commit to true

Solution

1. Step 1: Understand impact of low max.poll.records

   Setting it to 1 means the consumer fetches only one record per poll, severely limiting throughput and causing lag.

2. Step 2: Evaluate other options

   Setting fetch.min.bytes to 1 reduces wait time, high max.poll.interval.ms delays rebalance but doesn't reduce throughput, and auto commit true is normal behavior.

3. Final Answer:

   Setting max.poll.records too low, like 1 -> Option A

4. Quick Check:

   Too small batch size causes lag [OK]

Hint: Too small max.poll.records limits throughput [OK]

Common Mistakes:

• Confusing fetch.min.bytes effect
• Thinking auto commit affects speed directly
• Ignoring batch size impact

5. You want to optimize a Kafka consumer for high throughput but low latency. Which combination of settings is best?

hard

A. Set both max.poll.records and fetch.min.bytes to high values

B. Set max.poll.records low (e.g., 10) and fetch.min.bytes high (e.g., 1048576)

C. Set max.poll.records high (e.g., 1000) and fetch.min.bytes low (e.g., 1)

D. Set both max.poll.records and fetch.min.bytes to low values

Solution

1. Step 1: Understand throughput and latency tradeoff

   High max.poll.records allows processing many records per poll, increasing throughput.

2. Step 2: Understand fetch.min.bytes effect on latency

   Low fetch.min.bytes means the consumer fetches data quickly without waiting for large batches, reducing latency.

3. Step 3: Combine settings for desired effect

   High batch size with low fetch size balances high throughput and low latency.

4. Final Answer:

   Set max.poll.records high (e.g., 1000) and fetch.min.bytes low (e.g., 1) -> Option C

5. Quick Check:

   High batch + low fetch.min.bytes = throughput + low latency [OK]

Hint: High max.poll.records + low fetch.min.bytes = best balance [OK]

Common Mistakes:

• Setting both values high causes latency spikes
• Setting both low reduces throughput
• Ignoring fetch.min.bytes impact on latency