Consumer throughput optimization in Kafka - Step-by-Step Execution

Process Flow - Consumer throughput optimization

Start Consumer

↓

Fetch batch of messages

↓

Process messages in batch

↓

Commit offsets

↓

Adjust batch size or concurrency

↓

Repeat fetching next batch

↓

Stop Consumer when done

The consumer fetches messages in batches, processes them, commits offsets, and adjusts batch size or concurrency to optimize throughput.

Execution Sample

Kafka

consumer = KafkaConsumer('topic', group_id='group')
for batch in consumer.poll(timeout_ms=1000, max_records=10).values():
    process(batch)
    consumer.commit()

This code fetches messages in batches of up to 10, processes them, then commits offsets to optimize throughput.

Process Table

Step	Action	Batch Size	Messages Fetched	Processing	Offset Commit	Throughput Effect
1	Start Consumer	-	-	-	-	Consumer ready
2	Fetch batch	10	10 messages	No	No	Batch fetch begins
3	Process batch	10	10 messages	Processing 10 messages	No	Processing throughput high
4	Commit offsets	10	10 messages	Processed	Offsets committed	Offsets saved
5	Adjust batch size	10 -> 20	-	-	-	Increase batch size to improve throughput
6	Fetch batch	20	20 messages	No	No	Larger batch fetch
7	Process batch	20	20 messages	Processing 20 messages	No	Higher throughput
8	Commit offsets	20	20 messages	Processed	Offsets committed	Offsets saved
9	Stop Consumer	-	-	-	-	Consumer stopped

💡 Consumer stops after processing all messages or on manual stop.

Status Tracker

Variable	Start	After Step 2	After Step 5	After Step 6	Final
batch_size	10	10	20	20	20
messages_fetched	0	10	10	20	20
offset_committed	False	False	False	True	True

Key Moments - 3 Insights

Why do we increase the batch size after committing offsets?

Increasing batch size after committing offsets (see step 5) helps fetch more messages at once, improving throughput by reducing fetch overhead.

What happens if we commit offsets before processing messages?

Committing offsets before processing risks losing messages if processing fails. The table shows offsets committed only after processing (steps 4 and 8).

Why do we process messages in batches instead of one by one?

Batch processing reduces overhead and improves throughput by handling multiple messages together, as shown in steps 3 and 7.

Visual Quiz - 3 Questions

Look at the execution table, what is the batch size at step 6?

A 10

B 20

C 5

D 15

💡 Hint

Check the 'Batch Size' column at step 6 in the execution table.

At which step are offsets committed for the first time?

A Step 4

B Step 3

C Step 2

D Step 5

💡 Hint

Look at the 'Offset Commit' column to find when offsets are first committed.

If batch size was not increased at step 5, how would throughput be affected?

A Throughput would decrease

B Throughput would increase

C Throughput would stay the same

D Consumer would stop

💡 Hint

Refer to the 'Throughput Effect' column comparing steps before and after batch size change.

Concept Snapshot

Consumer throughput optimization:
- Fetch messages in batches
- Process batch before committing offsets
- Commit offsets after processing
- Adjust batch size or concurrency to improve throughput
- Repeat until done

Full Transcript

This visual trace shows how a Kafka consumer optimizes throughput by fetching messages in batches, processing them, and committing offsets after processing. The consumer starts with a batch size of 10, processes and commits offsets, then increases batch size to 20 to improve throughput. Variables like batch size and messages fetched change accordingly. Key moments include why batch size increases after committing offsets and why committing offsets before processing is risky. The quiz tests understanding of batch size changes, offset commits, and throughput effects.