Batch size and compression tuning in Kafka - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

Effect of batch.size on message sending

Consider a Kafka producer configured with batch.size=16384 bytes and linger.ms=0. The producer sends messages of 1000 bytes each. How many messages will be sent in one batch if the producer is continuously sending messages?

A 16384 messages

B 1 message

C 16 messages

D 0 messages

💡 Hint

Batch size limits the total bytes in a batch. Divide batch size by message size.

The batch size is 16384 bytes. Each message is 1000 bytes. So, the producer batches 16 messages (16 * 1000 = 16000 bytes) before sending. Since linger.ms=0, it sends immediately when batch is full.

❓ Predict Output

intermediate

Compression type impact on message size

A Kafka producer sends a batch of 100 messages, each 1KB uncompressed. The producer uses compression.type=gzip. After compression, the batch size is 50KB. What is the approximate compression ratio?

A 2:1

B 0.5:1

C 50:100

D 1:2

💡 Hint

Compression ratio = uncompressed size / compressed size.

Uncompressed size = 100 messages * 1KB = 100KB. Compressed size = 50KB. Ratio = 100KB / 50KB = 2:1.

❓ Predict Output

advanced

Producer batch sending timing with linger.ms

A Kafka producer has batch.size=32768 bytes and linger.ms=100. It sends messages of 5000 bytes each. If the producer sends 3 messages quickly, what happens?

A The producer sends the batch immediately after the first message.

B The producer waits indefinitely until batch is full.

C The producer sends each message immediately without waiting.

D The producer waits 100ms before sending the batch of 3 messages.

💡 Hint

Consider how linger.ms affects batching when batch is not full.

Batch size is 32768 bytes, 3 messages total 15000 bytes, less than batch size. With linger.ms=100, the producer waits up to 100ms to fill batch before sending. So it waits 100ms then sends the batch.

❓ Predict Output

advanced

Effect of compression.type=snappy on CPU usage

Which of the following is true about using compression.type=snappy in Kafka producers compared to gzip?

A Snappy uses less CPU and compresses faster but with lower compression ratio than gzip.

B Snappy uses more CPU and compresses slower but with higher compression ratio than gzip.

C Snappy and gzip have identical CPU usage and compression ratio.

D Snappy does not compress data, only encrypts it.

💡 Hint

Think about speed vs compression trade-offs.

Snappy is designed for fast compression with lower CPU usage and lower compression ratio compared to gzip, which compresses better but slower and uses more CPU.

🧠 Conceptual

expert

Choosing optimal batch size and compression for high throughput

You want to maximize Kafka producer throughput for large messages (~1MB each) on a network with moderate latency. Which combination is best?

A Set batch.size to 1MB and disable compression to reduce CPU usage.

B Set batch.size to 5MB and use compression.type=lz4 for fast compression and decompression.

C Set batch.size to 512KB and use compression.type=gzip for maximum compression ratio.

D Set batch.size to 100KB and use compression.type=snappy for minimal latency.

💡 Hint

Consider message size, network latency, and compression speed.

For large messages (~1MB), a larger batch size (5MB) allows batching multiple messages, improving throughput. LZ4 offers fast compression and decompression, balancing CPU and speed well. Smaller batch sizes or no compression reduce throughput or increase latency.