Compression (gzip, snappy, lz4) in Kafka - Step-by-Step Execution

Process Flow - Compression (gzip, snappy, lz4)

Message Created

↓

Choose Compression Codec

↓

Compress Message

↓

Send Compressed Message

↓

Broker Receives Message

↓

Broker Stores Compressed Message

↓

Consumer Fetches Message

↓

Decompress Message

↓

Consumer Processes Original Message

Messages are compressed before sending using a chosen codec, stored compressed, then decompressed by consumers to get the original message.

Execution Sample

Kafka

Properties props = new Properties();
props.put("compression.type", "gzip");
Producer<String, String> producer = new KafkaProducer<>(props);
producer.send(new ProducerRecord<>("topic", "message"));

This code sets gzip compression for Kafka producer and sends a message compressed with gzip.

Process Table

Step	Action	Compression Codec	Message Size Before	Message Size After	Result
1	Create message	none	100 bytes	100 bytes	Message ready to compress
2	Choose codec	gzip	100 bytes	100 bytes	Compression codec set to gzip
3	Compress message	gzip	100 bytes	40 bytes	Message compressed to smaller size
4	Send message	gzip	40 bytes	40 bytes	Compressed message sent to broker
5	Broker stores message	gzip	40 bytes	40 bytes	Compressed message stored
6	Consumer fetches message	gzip	40 bytes	40 bytes	Compressed message received
7	Decompress message	gzip	40 bytes	100 bytes	Original message restored
8	Process message	none	100 bytes	100 bytes	Consumer processes original message

💡 Message fully sent, stored, fetched, decompressed, and processed.

Status Tracker

Variable	Start	After Step 3	After Step 7	Final
message_size	100 bytes	40 bytes	100 bytes	100 bytes
compression_codec	none	gzip	gzip	none (after decompression)

Key Moments - 3 Insights

Why does the message size decrease after compression but return to original size after decompression?

Because compression reduces the message size for sending and storage (see step 3), but decompression restores it back to original size for processing (see step 7).

What happens if the consumer tries to read the message without decompressing?

The consumer would see compressed data, which is unreadable. Decompression (step 7) is necessary to get the original message.

Can different compression codecs be used in Kafka?

Yes, Kafka supports gzip, snappy, lz4, etc. The codec is chosen before compression (step 2) and must be supported by both producer and consumer.

Visual Quiz - 3 Questions

Look at the execution table, what is the message size after compression at step 3?

A 40 bytes

B 100 bytes

C 60 bytes

D 80 bytes

💡 Hint

Check the 'Message Size After' column at step 3 in the execution_table.

At which step does the consumer restore the original message size?

A Step 4

B Step 5

C Step 7

D Step 2

💡 Hint

Look for the step where 'Message Size After' returns to 100 bytes in the execution_table.

If the compression codec was changed to snappy, which step would be directly affected?

A Step 6 - Consumer fetches message

B Step 2 - Choose codec

C Step 8 - Process message

D Step 1 - Create message

💡 Hint

The codec choice happens at step 2 in the execution_table.

Concept Snapshot

Kafka compression flow:
- Set compression.type (gzip, snappy, lz4)
- Producer compresses message before sending
- Broker stores compressed message
- Consumer decompresses message after fetching
- Compression reduces network and storage size
- Decompression restores original message for processing

Full Transcript

In Kafka, messages can be compressed using codecs like gzip, snappy, or lz4. The producer sets the compression type before sending messages. When a message is created, it starts at its original size. The producer compresses it, reducing its size, then sends it to the broker. The broker stores the compressed message as is. When a consumer fetches the message, it receives the compressed data and must decompress it to restore the original message size before processing. This flow helps reduce network usage and storage space while ensuring consumers get the full original message. The execution table shows each step with message sizes and codec used. Key moments include understanding why size changes and the importance of decompression. The visual quiz tests understanding of these steps and codec choice.

Practice

1. Which of the following compression types is NOT supported by Kafka by default?

easy

A. gzip

B. snappy

C. lz4

D. bzip2

Solution

1. Step 1: Recall Kafka's default compression types

   Kafka supports gzip, snappy, and lz4 compression types by default.

2. Step 2: Identify unsupported compression

   bzip2 is not included in Kafka's default compression options.

3. Final Answer:

   bzip2 -> Option D

4. Quick Check:

   Kafka default compression excludes bzip2 [OK]

Hint: Remember Kafka supports gzip, snappy, and lz4 only [OK]

Common Mistakes:

• Confusing bzip2 as supported compression
• Thinking all common compressions are supported
• Mixing compression codecs with serialization formats

2. Which is the correct way to set gzip compression in Kafka producer configuration?

easy

A. compression.type = 'gzip'

B. compressionType = 'gzip'

C. compression.type = gzip

D. compression = 'gzip'

Solution

1. Step 1: Identify the correct property name

   The Kafka producer config uses 'compression.type' to set compression.

2. Step 2: Check the correct value format

   The value should be a string, so 'gzip' must be in quotes.

3. Final Answer:

   compression.type = 'gzip' -> Option A

4. Quick Check:

   Use compression.type with string value [OK]

Hint: Use compression.type with quotes around value [OK]

Common Mistakes:

• Using wrong property name like compressionType
• Omitting quotes around 'gzip'
• Using compression instead of compression.type

3. Given this Kafka producer config snippet:

props.put("compression.type", "snappy");
props.put("acks", "all");

What is the effect of setting compression.type to snappy?

medium

A. Messages will be sent without compression.

B. Messages will be compressed using Snappy before sending.

C. Messages will be compressed using gzip.

D. Producer will throw an error due to invalid compression.

Solution

1. Step 1: Understand the compression.type setting

   Setting compression.type to 'snappy' means messages are compressed with Snappy codec.

2. Step 2: Confirm no errors expected

   Snappy is a valid compression type, so no errors occur and compression is applied.

3. Final Answer:

   Messages will be compressed using Snappy before sending. -> Option B

4. Quick Check:

   compression.type = snappy means Snappy compression [OK]

Hint: compression.type sets the codec used for message compression [OK]

Common Mistakes:

• Assuming no compression happens
• Confusing snappy with gzip
• Expecting an error for valid compression type

4. You wrote this Kafka producer config:

props.put("compression.type", "lz5");

What is the problem with this configuration?

medium

A. lz5 is not a supported compression type in Kafka.

B. The property name should be compressionType, not compression.type.

C. The value should be uppercase: LZ5.

D. There is no problem; this is valid.

Solution

1. Step 1: Check the compression type value

   Kafka supports gzip, snappy, and lz4 only; lz5 is invalid.

2. Step 2: Confirm property name correctness

   The property name compression.type is correct, so no issue there.

3. Final Answer:

   lz5 is not a supported compression type in Kafka. -> Option A

4. Quick Check:

   Only gzip, snappy, lz4 are valid compression types [OK]

Hint: Use only gzip, snappy, or lz4 for compression.type [OK]

Common Mistakes:

• Using unsupported compression types like lz5
• Changing property name incorrectly
• Assuming case sensitivity matters for compression type

5. You want to optimize Kafka message throughput by choosing a compression type that balances speed and compression ratio. Which compression type should you choose?

hard

A. gzip, because it has the best compression ratio but slower speed.

B. No compression, to avoid any overhead.

C. snappy, because it offers fast compression with reasonable compression ratio.

D. lz4, because it is the slowest but compresses best.

Solution

1. Step 1: Understand compression trade-offs

   gzip compresses well but is slower; snappy is faster with decent compression; lz4 is fast and efficient.

2. Step 2: Identify best balance for throughput

   snappy is known for fast compression speed and reasonable compression ratio, ideal for throughput.

3. Final Answer:

   snappy, because it offers fast compression with reasonable compression ratio. -> Option C

4. Quick Check:

   snappy balances speed and compression well [OK]

Hint: Pick snappy for speed and decent compression [OK]

Common Mistakes:

• Choosing gzip for speed instead of compression ratio
• Thinking lz4 is slowest
• Assuming no compression is fastest overall