What is Protobuf with Kafka: Simple Explanation and Example

Using Protobuf with Kafka means encoding Kafka messages in a compact, fast, and structured binary format defined by Protobuf schemas. This helps Kafka efficiently send and receive data with clear structure and smaller message sizes compared to plain text formats.

How It Works

Imagine you want to send a letter, but instead of writing it in long sentences, you use a special code that both you and the receiver understand perfectly. Protobuf (Protocol Buffers) is like that code for data. It defines a clear structure for your data and turns it into a small, fast, and easy-to-send package.

Kafka is like a post office that delivers these packages (messages) between different parts of a system. When you use Protobuf with Kafka, you first define your message format in a Protobuf schema. Then, Kafka sends these compact Protobuf messages instead of plain text, making communication faster and saving space.

This method also helps avoid confusion because both sender and receiver know exactly what data to expect, just like having a shared language.

Example

This example shows how to define a Protobuf message, serialize it, and send it to Kafka, then receive and deserialize it back.

java

syntax = "proto3";

message User {
  string id = 1;
  string name = 2;
  int32 age = 3;
}

// Producer code snippet (Java)
import com.example.UserProtos.User;
import org.apache.kafka.clients.producer.*;
import org.apache.kafka.common.serialization.ByteArraySerializer;
import java.util.Properties;

Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer", ByteArraySerializer.class.getName());

Producer<String, byte[]> producer = new KafkaProducer<>(props);

User user = User.newBuilder()
    .setId("123")
    .setName("Alice")
    .setAge(30)
    .build();

byte[] userBytes = user.toByteArray();

ProducerRecord<String, byte[]> record = new ProducerRecord<>("users", "user1", userBytes);
producer.send(record);
producer.close();

// Consumer code snippet (Java)
import com.example.UserProtos.User;
import org.apache.kafka.clients.consumer.*;
import org.apache.kafka.common.serialization.ByteArrayDeserializer;
import java.time.Duration;
import java.util.Collections;

Properties consumerProps = new Properties();
consumerProps.put("bootstrap.servers", "localhost:9092");
consumerProps.put("group.id", "group1");
consumerProps.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
consumerProps.put("value.deserializer", ByteArrayDeserializer.class.getName());

Consumer<String, byte[]> consumer = new KafkaConsumer<>(consumerProps);
consumer.subscribe(Collections.singletonList("users"));

ConsumerRecords<String, byte[]> records = consumer.poll(Duration.ofSeconds(1));
for (ConsumerRecord<String, byte[]> rec : records) {
    User receivedUser = User.parseFrom(rec.value());
    System.out.println("Received user: " + receivedUser.getName() + ", age " + receivedUser.getAge());
}
consumer.close();

Output

Received user: Alice, age 30

When to Use

Use Protobuf with Kafka when you need to send structured data efficiently between services. It is great for systems where performance and bandwidth matter, like real-time data pipelines, microservices communication, or event-driven architectures.

Protobuf helps keep messages small and fast to process, which is important when you have many messages or large data volumes. It also ensures data consistency by enforcing a schema, so both sender and receiver agree on the data format.

For example, a company tracking user activity might use Kafka with Protobuf to send user events quickly and reliably to analytics systems.

Key Points

• Protobuf is a compact binary format for structured data.
• Kafka is a messaging system that transports data between apps.
• Using Protobuf with Kafka makes messages smaller and faster to send.
• Protobuf schemas ensure both sender and receiver understand the data.
• This combo is ideal for high-performance, real-time data pipelines.

Key Takeaways

Protobuf encodes Kafka messages in a compact, structured binary format for efficiency.

Kafka transports these Protobuf messages between systems quickly and reliably.

Use this combination to improve performance and data consistency in messaging.

Protobuf schemas define message structure, preventing data misunderstandings.

Ideal for real-time pipelines, microservices, and high-volume data flows.