Compression (gzip, snappy, lz4) in Kafka - Step-by-Step Execution

Process Flow - Compression (gzip, snappy, lz4)

Message Created

↓

Choose Compression Codec

↓

Compress Message

↓

Send Compressed Message

↓

Broker Receives Message

↓

Broker Stores Compressed Message

↓

Consumer Fetches Message

↓

Decompress Message

↓

Consumer Processes Original Message

Messages are compressed before sending using a chosen codec, stored compressed, then decompressed by consumers to get the original message.

Execution Sample

Kafka

Properties props = new Properties();
props.put("compression.type", "gzip");
Producer<String, String> producer = new KafkaProducer<>(props);
producer.send(new ProducerRecord<>("topic", "message"));

This code sets gzip compression for Kafka producer and sends a message compressed with gzip.

Process Table

Step	Action	Compression Codec	Message Size Before	Message Size After	Result
1	Create message	none	100 bytes	100 bytes	Message ready to compress
2	Choose codec	gzip	100 bytes	100 bytes	Compression codec set to gzip
3	Compress message	gzip	100 bytes	40 bytes	Message compressed to smaller size
4	Send message	gzip	40 bytes	40 bytes	Compressed message sent to broker
5	Broker stores message	gzip	40 bytes	40 bytes	Compressed message stored
6	Consumer fetches message	gzip	40 bytes	40 bytes	Compressed message received
7	Decompress message	gzip	40 bytes	100 bytes	Original message restored
8	Process message	none	100 bytes	100 bytes	Consumer processes original message

💡 Message fully sent, stored, fetched, decompressed, and processed.

Status Tracker

Variable	Start	After Step 3	After Step 7	Final
message_size	100 bytes	40 bytes	100 bytes	100 bytes
compression_codec	none	gzip	gzip	none (after decompression)

Key Moments - 3 Insights

Why does the message size decrease after compression but return to original size after decompression?

Because compression reduces the message size for sending and storage (see step 3), but decompression restores it back to original size for processing (see step 7).

What happens if the consumer tries to read the message without decompressing?

The consumer would see compressed data, which is unreadable. Decompression (step 7) is necessary to get the original message.

Can different compression codecs be used in Kafka?

Yes, Kafka supports gzip, snappy, lz4, etc. The codec is chosen before compression (step 2) and must be supported by both producer and consumer.

Visual Quiz - 3 Questions

Look at the execution table, what is the message size after compression at step 3?

A 40 bytes

B 100 bytes

C 60 bytes

D 80 bytes

💡 Hint

Check the 'Message Size After' column at step 3 in the execution_table.

At which step does the consumer restore the original message size?

A Step 4

B Step 5

C Step 7

D Step 2

💡 Hint

Look for the step where 'Message Size After' returns to 100 bytes in the execution_table.

If the compression codec was changed to snappy, which step would be directly affected?

A Step 6 - Consumer fetches message

B Step 2 - Choose codec

C Step 8 - Process message

D Step 1 - Create message

💡 Hint

The codec choice happens at step 2 in the execution_table.

Concept Snapshot

Kafka compression flow:
- Set compression.type (gzip, snappy, lz4)
- Producer compresses message before sending
- Broker stores compressed message
- Consumer decompresses message after fetching
- Compression reduces network and storage size
- Decompression restores original message for processing

Full Transcript

In Kafka, messages can be compressed using codecs like gzip, snappy, or lz4. The producer sets the compression type before sending messages. When a message is created, it starts at its original size. The producer compresses it, reducing its size, then sends it to the broker. The broker stores the compressed message as is. When a consumer fetches the message, it receives the compressed data and must decompress it to restore the original message size before processing. This flow helps reduce network usage and storage space while ensuring consumers get the full original message. The execution table shows each step with message sizes and codec used. Key moments include understanding why size changes and the importance of decompression. The visual quiz tests understanding of these steps and codec choice.