Compression (gzip, snappy, lz4) in Kafka - Mini Project: Build & Apply

Kafka Message Compression with gzip, snappy, and lz4

📖 Scenario: You are working on a Kafka producer application that sends messages to a Kafka topic. To save bandwidth and storage, you want to compress the messages before sending them.This project will guide you through setting up Kafka producer properties to use different compression algorithms: gzip, snappy, and lz4.

🎯 Goal: Build a Kafka producer configuration that sets the compression type to gzip, snappy, or lz4, and send a simple message to a Kafka topic.

📋 What You'll Learn

Create a Kafka producer properties map with bootstrap servers and key/value serializers

Add a configuration property called compression.type with the value gzip, snappy, or lz4

Create a KafkaProducer instance using the properties

Send a message with key testKey and value Compressed message to topic test-topic

Print Message sent with compression: [compression type] after sending

💡 Why This Matters

🌍 Real World

Compressing Kafka messages reduces network bandwidth and storage costs in real-time data pipelines.

💼 Career

Kafka producers with compression are common in data engineering and backend development roles to optimize performance.

1

Set up Kafka producer properties

Create a Properties object called props. Add these entries: bootstrap.servers set to "localhost:9092", key.serializer set to "org.apache.kafka.common.serialization.StringSerializer", and value.serializer set to "org.apache.kafka.common.serialization.StringSerializer".

Kafka

Properties props = new Properties();
// Your code here

Hint

Use props.put(key, value) to add each property.

2

Configure compression type

Add a property to props with key compression.type and value gzip.

Kafka

import java.util.Properties;

Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
// Your code here

Hint

Use props.put("compression.type", "gzip") to set gzip compression.

3

Create KafkaProducer and send message

Create a KafkaProducer<String, String> called producer using props. Send a message with key "testKey" and value "Compressed message" to topic "test-topic" using producer.send(). Close the producer after sending.

Kafka

import java.util.Properties;
import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.ProducerRecord;

Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("compression.type", "gzip");
// Your code here

Hint

Create the producer with new KafkaProducer<>(props). Use producer.send(new ProducerRecord<>(topic, key, value)) to send.

4

Print confirmation message

Print the message "Message sent with compression: gzip" using System.out.println().

Kafka

import java.util.Properties;
import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.ProducerRecord;

Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("compression.type", "gzip");

KafkaProducer producer = new KafkaProducer(props);
producer.send(new ProducerRecord("test-topic", "testKey", "Compressed message"));
producer.close();
// Your code here

Hint

Use System.out.println("Message sent with compression: gzip") to print.

Practice

1. Which of the following compression types is NOT supported by Kafka by default?

easy

A. gzip

B. snappy

C. lz4

D. bzip2

Solution

1. Step 1: Recall Kafka's default compression types

   Kafka supports gzip, snappy, and lz4 compression types by default.

2. Step 2: Identify unsupported compression

   bzip2 is not included in Kafka's default compression options.

3. Final Answer:

   bzip2 -> Option D

4. Quick Check:

   Kafka default compression excludes bzip2 [OK]

Hint: Remember Kafka supports gzip, snappy, and lz4 only [OK]

Common Mistakes:

• Confusing bzip2 as supported compression
• Thinking all common compressions are supported
• Mixing compression codecs with serialization formats

2. Which is the correct way to set gzip compression in Kafka producer configuration?

easy

A. compression.type = 'gzip'

B. compressionType = 'gzip'

C. compression.type = gzip

D. compression = 'gzip'

Solution

1. Step 1: Identify the correct property name

   The Kafka producer config uses 'compression.type' to set compression.

2. Step 2: Check the correct value format

   The value should be a string, so 'gzip' must be in quotes.

3. Final Answer:

   compression.type = 'gzip' -> Option A

4. Quick Check:

   Use compression.type with string value [OK]

Hint: Use compression.type with quotes around value [OK]

Common Mistakes:

• Using wrong property name like compressionType
• Omitting quotes around 'gzip'
• Using compression instead of compression.type

3. Given this Kafka producer config snippet:

props.put("compression.type", "snappy");
props.put("acks", "all");

What is the effect of setting compression.type to snappy?

medium

A. Messages will be sent without compression.

B. Messages will be compressed using Snappy before sending.

C. Messages will be compressed using gzip.

D. Producer will throw an error due to invalid compression.

Solution

1. Step 1: Understand the compression.type setting

   Setting compression.type to 'snappy' means messages are compressed with Snappy codec.

2. Step 2: Confirm no errors expected

   Snappy is a valid compression type, so no errors occur and compression is applied.

3. Final Answer:

   Messages will be compressed using Snappy before sending. -> Option B

4. Quick Check:

   compression.type = snappy means Snappy compression [OK]

Hint: compression.type sets the codec used for message compression [OK]

Common Mistakes:

• Assuming no compression happens
• Confusing snappy with gzip
• Expecting an error for valid compression type

4. You wrote this Kafka producer config:

props.put("compression.type", "lz5");

What is the problem with this configuration?

medium

A. lz5 is not a supported compression type in Kafka.

B. The property name should be compressionType, not compression.type.

C. The value should be uppercase: LZ5.

D. There is no problem; this is valid.

Solution

1. Step 1: Check the compression type value

   Kafka supports gzip, snappy, and lz4 only; lz5 is invalid.

2. Step 2: Confirm property name correctness

   The property name compression.type is correct, so no issue there.

3. Final Answer:

   lz5 is not a supported compression type in Kafka. -> Option A

4. Quick Check:

   Only gzip, snappy, lz4 are valid compression types [OK]

Hint: Use only gzip, snappy, or lz4 for compression.type [OK]

Common Mistakes:

• Using unsupported compression types like lz5
• Changing property name incorrectly
• Assuming case sensitivity matters for compression type

5. You want to optimize Kafka message throughput by choosing a compression type that balances speed and compression ratio. Which compression type should you choose?

hard

A. gzip, because it has the best compression ratio but slower speed.

B. No compression, to avoid any overhead.

C. snappy, because it offers fast compression with reasonable compression ratio.

D. lz4, because it is the slowest but compresses best.

Solution

1. Step 1: Understand compression trade-offs

   gzip compresses well but is slower; snappy is faster with decent compression; lz4 is fast and efficient.

2. Step 2: Identify best balance for throughput

   snappy is known for fast compression speed and reasonable compression ratio, ideal for throughput.

3. Final Answer:

   snappy, because it offers fast compression with reasonable compression ratio. -> Option C

4. Quick Check:

   snappy balances speed and compression well [OK]

Hint: Pick snappy for speed and decent compression [OK]

Common Mistakes:

• Choosing gzip for speed instead of compression ratio
• Thinking lz4 is slowest
• Assuming no compression is fastest overall