You want to implement at-least-once delivery in Kafka with batch processing. Which approach is best?
hard🚀 Application Q8 of Q15
Kafka - Message Delivery Semantics
You want to implement at-least-once delivery in Kafka with batch processing. Which approach is best?
AProcess batch, then commit offsets for entire batch after success
BCommit offsets before processing batch to speed up throughput
CCommit offsets after each message in batch individually
DDisable offset commits and rely on auto commit
Step-by-Step Solution
Solution:
Step 1: Understand batch processing with at-least-once
Processing the whole batch first and committing offsets after success ensures no message loss and avoids partial commits.
Step 2: Evaluate options
Process batch, then commit offsets for entire batch after success commits offsets after batch processing, matching at-least-once semantics. Other options risk message loss or duplicates.
Final Answer:
Process batch, then commit offsets for entire batch after success -> Option A
Quick Check:
Commit offsets after batch processing for at-least-once [OK]
Quick Trick:Commit offsets after batch processing to ensure no message loss [OK]
Common Mistakes:
MISTAKES
Committing offsets before processing batch
Committing offsets per message in batch unnecessarily
Relying on auto commit for at-least-once
Master "Message Delivery Semantics" in Kafka
9 interactive learning modes - each teaches the same concept differently