Why Batch size and compression tuning in Kafka? - Purpose & Use Cases

The Big Idea

What if you could send thousands of messages in the time it takes to send one?

The Scenario

Imagine sending thousands of messages one by one over a network, like mailing each letter separately instead of bundling them in envelopes.

The Problem

This slow, message-by-message sending wastes time and bandwidth, causing delays and higher costs. It's like paying extra postage for every single letter instead of using bulk mailing.

The Solution

Batch size and compression tuning lets you group many messages together and shrink their size before sending. This saves time, reduces network load, and speeds up your data flow.

Before vs After

✗ Before

producer.send(message1)
producer.send(message2)
producer.send(message3)

✓ After

producer.config['batch.size'] = 1000
producer.config['compression.type'] = 'gzip'
producer.send(batch_of_messages)

What It Enables

It enables fast, efficient, and cost-effective data streaming at scale.

Real Life Example

A social media app sending user activity logs can bundle thousands of events and compress them, delivering updates quickly without overloading the network.

Key Takeaways

Sending messages one by one is slow and costly.

Batching groups messages to send together efficiently.

Compression shrinks data size, saving bandwidth and time.