Consumer throughput optimization in Kafka - Step-by-Step Execution

Process Flow - Consumer throughput optimization

Start Consumer

↓

Fetch batch of messages

↓

Process messages in batch

↓

Commit offsets

↓

Adjust batch size or concurrency

↓

Repeat fetching next batch

↓

Stop Consumer when done

The consumer fetches messages in batches, processes them, commits offsets, and adjusts batch size or concurrency to optimize throughput.

Execution Sample

Kafka

consumer = KafkaConsumer('topic', group_id='group')
for batch in consumer.poll(timeout_ms=1000, max_records=10).values():
    process(batch)
    consumer.commit()

This code fetches messages in batches of up to 10, processes them, then commits offsets to optimize throughput.

Process Table

Step	Action	Batch Size	Messages Fetched	Processing	Offset Commit	Throughput Effect
1	Start Consumer	-	-	-	-	Consumer ready
2	Fetch batch	10	10 messages	No	No	Batch fetch begins
3	Process batch	10	10 messages	Processing 10 messages	No	Processing throughput high
4	Commit offsets	10	10 messages	Processed	Offsets committed	Offsets saved
5	Adjust batch size	10 -> 20	-	-	-	Increase batch size to improve throughput
6	Fetch batch	20	20 messages	No	No	Larger batch fetch
7	Process batch	20	20 messages	Processing 20 messages	No	Higher throughput
8	Commit offsets	20	20 messages	Processed	Offsets committed	Offsets saved
9	Stop Consumer	-	-	-	-	Consumer stopped

💡 Consumer stops after processing all messages or on manual stop.

Status Tracker

Variable	Start	After Step 2	After Step 5	After Step 6	Final
batch_size	10	10	20	20	20
messages_fetched	0	10	10	20	20
offset_committed	False	False	False	True	True

Key Moments - 3 Insights

Why do we increase the batch size after committing offsets?

Increasing batch size after committing offsets (see step 5) helps fetch more messages at once, improving throughput by reducing fetch overhead.

What happens if we commit offsets before processing messages?

Committing offsets before processing risks losing messages if processing fails. The table shows offsets committed only after processing (steps 4 and 8).

Why do we process messages in batches instead of one by one?

Batch processing reduces overhead and improves throughput by handling multiple messages together, as shown in steps 3 and 7.

Visual Quiz - 3 Questions

Look at the execution table, what is the batch size at step 6?

A 10

B 20

C 5

D 15

💡 Hint

Check the 'Batch Size' column at step 6 in the execution table.

At which step are offsets committed for the first time?

A Step 4

B Step 3

C Step 2

D Step 5

💡 Hint

Look at the 'Offset Commit' column to find when offsets are first committed.

If batch size was not increased at step 5, how would throughput be affected?

A Throughput would decrease

B Throughput would increase

C Throughput would stay the same

D Consumer would stop

💡 Hint

Refer to the 'Throughput Effect' column comparing steps before and after batch size change.

Concept Snapshot

Consumer throughput optimization:
- Fetch messages in batches
- Process batch before committing offsets
- Commit offsets after processing
- Adjust batch size or concurrency to improve throughput
- Repeat until done

Full Transcript

This visual trace shows how a Kafka consumer optimizes throughput by fetching messages in batches, processing them, and committing offsets after processing. The consumer starts with a batch size of 10, processes and commits offsets, then increases batch size to 20 to improve throughput. Variables like batch size and messages fetched change accordingly. Key moments include why batch size increases after committing offsets and why committing offsets before processing is risky. The quiz tests understanding of batch size changes, offset commits, and throughput effects.

Practice

1. What is the primary purpose of increasing max.poll.records in a Kafka consumer configuration?

easy

A. To decrease the consumer's memory usage

B. To increase the number of records fetched in one poll, improving throughput

C. To reduce the number of partitions assigned to the consumer

D. To increase the consumer's connection timeout

Solution

1. Step 1: Understand max.poll.records role

   This setting controls how many records the consumer fetches in a single poll call.

2. Step 2: Effect on throughput

   Increasing this value allows the consumer to process more records per poll, which can improve throughput by reducing overhead.

3. Final Answer:

   To increase the number of records fetched in one poll, improving throughput -> Option B

4. Quick Check:

   max.poll.records controls batch size [OK]

Hint: More records per poll means higher throughput [OK]

Common Mistakes:

• Confusing max.poll.records with timeout settings
• Thinking it reduces memory usage
• Assuming it changes partition assignment

2. Which of the following is the correct way to set fetch.min.bytes to 50000 in a Kafka consumer properties file?

easy

A. fetch.min.bytes=50000

B. fetch.min.bytes: 50000

C. "fetch.min.bytes" = 50000

D. fetch_min_bytes=50000

Solution

1. Step 1: Identify Kafka properties file syntax

   Kafka consumer properties use key=value pairs without quotes or underscores.

2. Step 2: Match correct syntax

   fetch.min.bytes=50000 uses the correct key with dot notation and equals sign, matching Kafka config style.

3. Final Answer:

   fetch.min.bytes=50000 -> Option A

4. Quick Check:

   Kafka config uses key=value format [OK]

Hint: Use dot notation and equals sign for Kafka configs [OK]

Common Mistakes:

• Using colon instead of equals sign
• Adding quotes around keys
• Replacing dots with underscores

3. Given the following Kafka consumer configuration snippet:

props.put("max.poll.records", "1000");
props.put("fetch.min.bytes", "1048576");

What is the expected effect on consumer throughput and latency?

medium

A. Lower throughput and lower latency

B. Higher throughput and lower latency

C. No change in throughput or latency

D. Higher throughput with potentially increased latency

Solution

1. Step 1: Analyze max.poll.records setting

   Setting it to 1000 means the consumer fetches up to 1000 records per poll, increasing batch size and throughput.

2. Step 2: Analyze fetch.min.bytes setting

   Setting it to 1MB (1048576 bytes) means the consumer waits to fetch at least 1MB of data, which can increase latency waiting for enough data.

3. Final Answer:

   Higher throughput with potentially increased latency -> Option D

4. Quick Check:

   Large batch size = higher throughput, bigger fetch.min.bytes = higher latency [OK]

Hint: Large batch size boosts throughput but may add latency [OK]

Common Mistakes:

• Assuming latency always decreases with throughput
• Ignoring fetch.min.bytes impact
• Confusing max.poll.records with fetch.max.bytes

4. You notice your Kafka consumer is lagging and not processing messages fast enough. Which of the following configuration mistakes could cause this issue?

medium

A. Setting max.poll.records too low, like 1

B. Setting fetch.min.bytes to 1

C. Setting max.poll.interval.ms to a very high value

D. Setting enable.auto.commit to true

Solution

1. Step 1: Understand impact of low max.poll.records

   Setting it to 1 means the consumer fetches only one record per poll, severely limiting throughput and causing lag.

2. Step 2: Evaluate other options

   Setting fetch.min.bytes to 1 reduces wait time, high max.poll.interval.ms delays rebalance but doesn't reduce throughput, and auto commit true is normal behavior.

3. Final Answer:

   Setting max.poll.records too low, like 1 -> Option A

4. Quick Check:

   Too small batch size causes lag [OK]

Hint: Too small max.poll.records limits throughput [OK]

Common Mistakes:

• Confusing fetch.min.bytes effect
• Thinking auto commit affects speed directly
• Ignoring batch size impact

5. You want to optimize a Kafka consumer for high throughput but low latency. Which combination of settings is best?

hard

A. Set both max.poll.records and fetch.min.bytes to high values

B. Set max.poll.records low (e.g., 10) and fetch.min.bytes high (e.g., 1048576)

C. Set max.poll.records high (e.g., 1000) and fetch.min.bytes low (e.g., 1)

D. Set both max.poll.records and fetch.min.bytes to low values

Solution

1. Step 1: Understand throughput and latency tradeoff

   High max.poll.records allows processing many records per poll, increasing throughput.

2. Step 2: Understand fetch.min.bytes effect on latency

   Low fetch.min.bytes means the consumer fetches data quickly without waiting for large batches, reducing latency.

3. Step 3: Combine settings for desired effect

   High batch size with low fetch size balances high throughput and low latency.

4. Final Answer:

   Set max.poll.records high (e.g., 1000) and fetch.min.bytes low (e.g., 1) -> Option C

5. Quick Check:

   High batch + low fetch.min.bytes = throughput + low latency [OK]

Hint: High max.poll.records + low fetch.min.bytes = best balance [OK]

Common Mistakes:

• Setting both values high causes latency spikes
• Setting both low reduces throughput
• Ignoring fetch.min.bytes impact on latency