Compression (gzip, snappy, lz4) in Kafka - Mini Project: Build & Apply

Kafka Message Compression with gzip, snappy, and lz4

📖 Scenario: You are working on a Kafka producer application that sends messages to a Kafka topic. To save bandwidth and storage, you want to compress the messages before sending them.This project will guide you through setting up Kafka producer properties to use different compression algorithms: gzip, snappy, and lz4.

🎯 Goal: Build a Kafka producer configuration that sets the compression type to gzip, snappy, or lz4, and send a simple message to a Kafka topic.

📋 What You'll Learn

Create a Kafka producer properties map with bootstrap servers and key/value serializers

Add a configuration property called compression.type with the value gzip, snappy, or lz4

Create a KafkaProducer instance using the properties

Send a message with key testKey and value Compressed message to topic test-topic

Print Message sent with compression: [compression type] after sending

💡 Why This Matters

🌍 Real World

Compressing Kafka messages reduces network bandwidth and storage costs in real-time data pipelines.

💼 Career

Kafka producers with compression are common in data engineering and backend development roles to optimize performance.

1

Set up Kafka producer properties

Create a Properties object called props. Add these entries: bootstrap.servers set to "localhost:9092", key.serializer set to "org.apache.kafka.common.serialization.StringSerializer", and value.serializer set to "org.apache.kafka.common.serialization.StringSerializer".

Kafka

Properties props = new Properties();
// Your code here

Need a hint?

Use props.put(key, value) to add each property.

2

Configure compression type

Add a property to props with key compression.type and value gzip.

Kafka

import java.util.Properties;

Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
// Your code here

Need a hint?

Use props.put("compression.type", "gzip") to set gzip compression.

3

Create KafkaProducer and send message

Create a KafkaProducer<String, String> called producer using props. Send a message with key "testKey" and value "Compressed message" to topic "test-topic" using producer.send(). Close the producer after sending.

Kafka

import java.util.Properties;
import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.ProducerRecord;

Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("compression.type", "gzip");
// Your code here

Need a hint?

Create the producer with new KafkaProducer<>(props). Use producer.send(new ProducerRecord<>(topic, key, value)) to send.

4

Print confirmation message

Print the message "Message sent with compression: gzip" using System.out.println().

Kafka

import java.util.Properties;
import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.ProducerRecord;

Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("compression.type", "gzip");

KafkaProducer producer = new KafkaProducer(props);
producer.send(new ProducerRecord("test-topic", "testKey", "Compressed message"));
producer.close();
// Your code here

Need a hint?

Use System.out.println("Message sent with compression: gzip") to print.