Batch size and compression tuning in Kafka - Commands & Configuration

Introduction

Sending messages efficiently in Kafka helps reduce network use and speeds up processing. Batch size and compression settings control how many messages are sent together and how much data is shrunk before sending.

When you want to reduce the number of network calls by sending many messages at once.

When you need to lower the amount of data sent over the network to save bandwidth.

When your Kafka producer is slow because it sends messages one by one.

When you want to balance between latency and throughput in your Kafka messaging.

When you want to reduce storage space used by Kafka brokers by compressing messages.

Config File - producer.properties

producer.properties

batch.size=32768
compression.type=snappy
linger.ms=5

batch.size: Sets the maximum size (in bytes) of a batch of messages sent to a Kafka partition. Larger batches improve throughput but may increase latency.

compression.type: Defines the compression algorithm used to shrink message data before sending. 'snappy' is fast and balances speed with compression.

linger.ms: Time to wait before sending a batch, allowing more messages to accumulate for better batching.

Commands

Starts a Kafka producer using the configuration file to apply batch size and compression settings for sending messages efficiently.

Terminal

kafka-console-producer --broker-list localhost:9092 --topic example-topic --producer.config producer.properties

Expected Output

> This command opens a prompt to type messages. No output until messages are sent.

→

--producer.config - Specifies the configuration file with batch size and compression settings.

→

--broker-list - Defines the Kafka broker addresses to connect.

→

--topic - Specifies the topic to send messages to.

Updates the Kafka broker configuration to use 'snappy' compression for all messages handled by broker 0, reducing data size on the server side.

Terminal

kafka-configs --bootstrap-server localhost:9092 --entity-type brokers --entity-name 0 --alter --add-config compression.type=snappy

Expected Output

Updated config for broker 0.

→

--alter - Indicates that the configuration is being changed.

→

--add-config - Adds or updates the specified configuration key and value.

Checks the current configuration and status of the topic to verify that messages are being handled with the new batch and compression settings.

Terminal

kafka-topics --bootstrap-server localhost:9092 --describe --topic example-topic

Expected Output

Topic: example-topic	PartitionCount: 1	ReplicationFactor: 1	Configs: compression.type=snappy

Topic: example-topic	Partition: 0	Leader: 0	Replicas: 0	Isr: 0

→

--describe - Shows detailed information about the topic.

Key Concept

If you remember nothing else from this pattern, remember: tuning batch size and compression together helps send messages faster and use less network and storage.

Common Mistakes

Setting batch.size too large without adjusting linger.ms

Messages wait too long to fill the batch, causing high latency.

Set linger.ms to a small value like 5 ms to balance latency and batching.

Using compression.type 'none' or unsupported compression

No data size reduction happens, wasting bandwidth and storage.

Use supported compression types like 'snappy', 'gzip', or 'lz4' for better efficiency.

Changing producer settings but not verifying broker or topic configs

Broker or topic may override or ignore producer compression settings.

Check and align broker and topic compression settings with producer configuration.

Summary

Configure batch.size and compression.type in the producer properties file to control message batching and compression.

Use kafka-console-producer with the config file to send messages efficiently.

Update broker compression settings with kafka-configs to ensure consistent compression.

Verify topic configuration with kafka-topics to confirm settings are applied.

Practice

1. What does increasing the batch.size setting in Kafka producer configuration primarily affect?

easy

A. The number of bytes sent in one request to the broker

B. The number of partitions a topic has

C. The compression algorithm used for messages

D. The timeout for message delivery

Solution

1. Step 1: Understand batch.size meaning

   The batch.size controls how many bytes the producer will try to batch together before sending to the broker.

2. Step 2: Identify what batch.size affects

   Increasing batch.size means more data is sent in one request, improving throughput but possibly increasing latency.

3. Final Answer:

   The number of bytes sent in one request to the broker -> Option A

4. Quick Check:

   batch.size = bytes per request [OK]

Hint: Batch size controls bytes per request, not partitions or compression [OK]

Common Mistakes:

• Confusing batch size with number of partitions
• Thinking batch size sets compression type
• Assuming batch size controls timeout

2. Which of the following is the correct way to enable compression with gzip in Kafka producer configuration?

easy

A. "compression": "gzip"

B. "compress.type": "gzip"

C. "compression.type": "gzip"

D. "compressionType": "gzip"

Solution

1. Step 1: Recall Kafka producer config keys

   The correct key for compression is compression.type.

2. Step 2: Match correct syntax

   The value should be a string like "gzip". So the correct config is "compression.type": "gzip".

3. Final Answer:

   "compression.type": "gzip" -> Option C

4. Quick Check:

   compression.type = gzip [OK]

Hint: Use exact key 'compression.type' to enable compression [OK]

Common Mistakes:

• Using wrong key names like 'compression' or 'compress.type'
• Incorrect camelCase key 'compressionType'
• Missing quotes around 'gzip'

3. Given the following Kafka producer config snippet:

{
  "batch.size": 16384,
  "compression.type": "snappy"
}

What is the expected effect on message sending?

medium

A. Messages are sent in batches up to 16 KB and compressed with Snappy

B. Messages are sent one by one without compression

C. Messages are compressed with gzip and batch size is ignored

D. Batch size is 16 bytes and compression is disabled

Solution

1. Step 1: Interpret batch.size value

   The batch size is set to 16384 bytes, which means messages will be grouped up to 16 KB before sending.

2. Step 2: Interpret compression.type value

   The compression type is set to "snappy", so messages will be compressed using Snappy algorithm.

3. Final Answer:

   Messages are sent in batches up to 16 KB and compressed with Snappy -> Option A

4. Quick Check:

   batch.size=16384 + compression.type=snappy [OK]

Hint: Check batch size bytes and compression.type value carefully [OK]

Common Mistakes:

• Confusing batch size units (bytes vs KB)
• Assuming gzip compression instead of snappy
• Thinking batch size disables compression

4. You have set batch.size to 0 in your Kafka producer config. What issue will this cause?

medium

A. Producer will throw a configuration error and fail to start

B. Producer will send messages immediately without batching

C. Producer will batch messages indefinitely causing delays

D. Producer will compress messages with default gzip

Solution

1. Step 1: Understand batch.size valid values

   Batch size can be set to 0, which disables batching. Messages are sent immediately without waiting to fill a batch.

2. Step 2: Identify Kafka producer behavior on batch.size=0

   The producer will not throw an error; it will work but send each message in its own request, potentially reducing throughput.

3. Final Answer:

   Producer will send messages immediately without batching -> Option B

4. Quick Check:

   batch.size=0 disables batching [OK]

Hint: batch.size=0 disables batching safely, sends immediately [OK]

Common Mistakes:

• Thinking batch.size=0 causes a configuration error
• Believing it causes indefinite batching and delays
• Confusing batch size with compression settings

5. You want to optimize Kafka producer throughput for large messages but keep latency low. Which combination of settings is best?

hard

A. Use default batch.size and compression.type = "none"

B. Decrease batch.size and disable compression

C. Set batch.size to 0 and use compression.type = "gzip"

D. Increase batch.size and use compression.type = "lz4"

Solution

1. Step 1: Consider batch.size impact on throughput and latency

   Larger batch sizes improve throughput by sending more data at once but can increase latency slightly.

2. Step 2: Choose compression type for speed and size

   LZ4 compression is fast and efficient, reducing message size without high CPU cost, helping throughput and latency.

3. Step 3: Evaluate other options

   Decreasing batch size reduces throughput; disabling compression wastes bandwidth; batch.size=0 causes no batching.

4. Final Answer:

   Increase batch.size and use compression.type = "lz4" -> Option D

5. Quick Check:

   Large batch + fast compression = best throughput & low latency [OK]

Hint: Big batch + fast compression (lz4) balances speed and size [OK]

Common Mistakes:

• Choosing gzip which is slower than lz4
• Setting batch.size to 0 reducing throughput
• Disabling compression reduces efficiency