Compression (gzip, snappy, lz4) in Kafka - Commands & Configuration

Introduction

Compression reduces the size of data sent between Kafka producers and brokers. It helps save network bandwidth and storage space, making Kafka more efficient and faster.

When you want to reduce network usage between your Kafka producers and brokers to save costs.

When your Kafka topics handle large messages and you want to store them efficiently on disk.

When you want to improve throughput by sending smaller compressed messages over the network.

When you want to balance CPU usage and compression speed by choosing the right compression type.

When you want to ensure compatibility with Kafka clients and brokers by using supported compression codecs.

Config File - producer.properties

producer.properties

bootstrap.servers=localhost:9092
key.serializer=org.apache.kafka.common.serialization.StringSerializer
value.serializer=org.apache.kafka.common.serialization.StringSerializer
compression.type=snappy

This configuration file sets up a Kafka producer.

bootstrap.servers tells the producer where the Kafka broker is.

key.serializer and value.serializer convert data to bytes.

compression.type sets the compression codec to use (here, snappy).

Commands

Starts a Kafka console producer that sends messages to 'example-topic' using gzip compression to reduce message size.

Terminal

kafka-console-producer --broker-list localhost:9092 --topic example-topic --compression-codec gzip

Expected Output

No output (command runs silently)

→

--broker-list - Specifies the Kafka broker address to connect to.

→

--topic - Specifies the Kafka topic to send messages to.

→

--compression-codec - Sets the compression codec for messages sent by this producer.

Starts a Kafka console consumer to read all messages from 'example-topic' from the beginning, showing that compressed messages are decompressed automatically.

Terminal

kafka-console-consumer --bootstrap-server localhost:9092 --topic example-topic --from-beginning

Expected Output

Hello Kafka

Compressed message example

→

--bootstrap-server - Specifies the Kafka broker address to connect to.

→

--topic - Specifies the Kafka topic to read messages from.

→

--from-beginning - Reads all messages from the start of the topic.

Shows details about 'example-topic' including partition count and replication, confirming the topic exists and is ready for compressed messages.

Terminal

kafka-topics --bootstrap-server localhost:9092 --describe --topic example-topic

Expected Output

Topic: example-topic	PartitionCount: 1	ReplicationFactor: 1	Configs: 

Partition: 0	Leader: 1	Replicas: 1	Isr: 1

→

--bootstrap-server - Specifies the Kafka broker address.

→

--describe - Shows detailed information about the topic.

→

--topic - Specifies the topic to describe.

Key Concept

If you remember nothing else from this pattern, remember: Kafka producers compress messages before sending, and consumers automatically decompress them, saving bandwidth and storage transparently.

Common Mistakes

Setting compression.type on the broker or topic but not on the producer.

Compression is applied by the producer; if the producer does not compress, no compression happens even if the broker or topic is configured.

Always set compression.type in the producer configuration or use the --compression-codec flag with the producer command.

Using an unsupported compression codec like 'lzma' which Kafka does not support.

Kafka only supports gzip, snappy, lz4, and zstd compression codecs; unsupported codecs cause errors or fallback to no compression.

Use only supported codecs: gzip, snappy, lz4, or zstd.

Expecting compression to reduce CPU usage significantly.

Compression saves bandwidth but uses CPU to compress and decompress; some codecs are faster but less compressed, others slower but better compression.

Choose the compression codec based on your CPU and network trade-offs.

Summary

Set compression.type in the Kafka producer configuration to enable message compression.

Use kafka-console-producer with --compression-codec to send compressed messages.

Kafka consumers automatically decompress messages, so no special config is needed on the consumer side.

Check topic details with kafka-topics to confirm topic configuration and status.

Practice

1. Which of the following compression types is NOT supported by Kafka by default?

easy

A. gzip

B. snappy

C. lz4

D. bzip2

Solution

1. Step 1: Recall Kafka's default compression types

   Kafka supports gzip, snappy, and lz4 compression types by default.

2. Step 2: Identify unsupported compression

   bzip2 is not included in Kafka's default compression options.

3. Final Answer:

   bzip2 -> Option D

4. Quick Check:

   Kafka default compression excludes bzip2 [OK]

Hint: Remember Kafka supports gzip, snappy, and lz4 only [OK]

Common Mistakes:

• Confusing bzip2 as supported compression
• Thinking all common compressions are supported
• Mixing compression codecs with serialization formats

2. Which is the correct way to set gzip compression in Kafka producer configuration?

easy

A. compression.type = 'gzip'

B. compressionType = 'gzip'

C. compression.type = gzip

D. compression = 'gzip'

Solution

1. Step 1: Identify the correct property name

   The Kafka producer config uses 'compression.type' to set compression.

2. Step 2: Check the correct value format

   The value should be a string, so 'gzip' must be in quotes.

3. Final Answer:

   compression.type = 'gzip' -> Option A

4. Quick Check:

   Use compression.type with string value [OK]

Hint: Use compression.type with quotes around value [OK]

Common Mistakes:

• Using wrong property name like compressionType
• Omitting quotes around 'gzip'
• Using compression instead of compression.type

3. Given this Kafka producer config snippet:

props.put("compression.type", "snappy");
props.put("acks", "all");

What is the effect of setting compression.type to snappy?

medium

A. Messages will be sent without compression.

B. Messages will be compressed using Snappy before sending.

C. Messages will be compressed using gzip.

D. Producer will throw an error due to invalid compression.

Solution

1. Step 1: Understand the compression.type setting

   Setting compression.type to 'snappy' means messages are compressed with Snappy codec.

2. Step 2: Confirm no errors expected

   Snappy is a valid compression type, so no errors occur and compression is applied.

3. Final Answer:

   Messages will be compressed using Snappy before sending. -> Option B

4. Quick Check:

   compression.type = snappy means Snappy compression [OK]

Hint: compression.type sets the codec used for message compression [OK]

Common Mistakes:

• Assuming no compression happens
• Confusing snappy with gzip
• Expecting an error for valid compression type

4. You wrote this Kafka producer config:

props.put("compression.type", "lz5");

What is the problem with this configuration?

medium

A. lz5 is not a supported compression type in Kafka.

B. The property name should be compressionType, not compression.type.

C. The value should be uppercase: LZ5.

D. There is no problem; this is valid.

Solution

1. Step 1: Check the compression type value

   Kafka supports gzip, snappy, and lz4 only; lz5 is invalid.

2. Step 2: Confirm property name correctness

   The property name compression.type is correct, so no issue there.

3. Final Answer:

   lz5 is not a supported compression type in Kafka. -> Option A

4. Quick Check:

   Only gzip, snappy, lz4 are valid compression types [OK]

Hint: Use only gzip, snappy, or lz4 for compression.type [OK]

Common Mistakes:

• Using unsupported compression types like lz5
• Changing property name incorrectly
• Assuming case sensitivity matters for compression type

5. You want to optimize Kafka message throughput by choosing a compression type that balances speed and compression ratio. Which compression type should you choose?

hard

A. gzip, because it has the best compression ratio but slower speed.

B. No compression, to avoid any overhead.

C. snappy, because it offers fast compression with reasonable compression ratio.

D. lz4, because it is the slowest but compresses best.

Solution

1. Step 1: Understand compression trade-offs

   gzip compresses well but is slower; snappy is faster with decent compression; lz4 is fast and efficient.

2. Step 2: Identify best balance for throughput

   snappy is known for fast compression speed and reasonable compression ratio, ideal for throughput.

3. Final Answer:

   snappy, because it offers fast compression with reasonable compression ratio. -> Option C

4. Quick Check:

   snappy balances speed and compression well [OK]

Hint: Pick snappy for speed and decent compression [OK]

Common Mistakes:

• Choosing gzip for speed instead of compression ratio
• Thinking lz4 is slowest
• Assuming no compression is fastest overall