How to Use Avro Serializer with Kafka: Simple Guide

To use AvroSerializer in Kafka, configure your producer with the Avro serializer class and provide the Schema Registry URL. This setup allows Kafka to serialize messages using Avro format, ensuring schema compatibility and efficient data exchange.

Syntax

To use Avro serialization in Kafka, set the following properties in your producer configuration:

• key.serializer: Serializer class for message keys, usually io.confluent.kafka.serializers.KafkaAvroSerializer.
• value.serializer: Serializer class for message values, also io.confluent.kafka.serializers.KafkaAvroSerializer.
• schema.registry.url: URL of the Schema Registry service that manages Avro schemas.

This configuration enables Kafka to serialize messages in Avro format and register schemas centrally.

java

Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("key.serializer", "io.confluent.kafka.serializers.KafkaAvroSerializer");
props.put("value.serializer", "io.confluent.kafka.serializers.KafkaAvroSerializer");
props.put("schema.registry.url", "http://localhost:8081");

Example

This example shows a Kafka producer sending Avro serialized messages using a generated Avro class User. It connects to Kafka and Schema Registry, serializes the User object, and sends it to a topic.

java

import io.confluent.kafka.serializers.KafkaAvroSerializer;
import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.ProducerRecord;
import org.apache.kafka.clients.producer.RecordMetadata;
import java.util.Properties;

public class AvroProducerExample {
    public static void main(String[] args) throws Exception {
        Properties props = new Properties();
        props.put("bootstrap.servers", "localhost:9092");
        props.put("key.serializer", KafkaAvroSerializer.class.getName());
        props.put("value.serializer", KafkaAvroSerializer.class.getName());
        props.put("schema.registry.url", "http://localhost:8081");

        KafkaProducer<String, User> producer = new KafkaProducer<>(props);

        User user = User.newBuilder()
                .setName("Alice")
                .setFavoriteNumber(42)
                .setFavoriteColor("blue")
                .build();

        ProducerRecord<String, User> record = new ProducerRecord<>("users", "user1", user);

        RecordMetadata metadata = producer.send(record).get();
        System.out.println("Sent record to topic " + metadata.topic() + " partition " + metadata.partition() + " offset " + metadata.offset());

        producer.close();
    }
}

Output

Sent record to topic users partition 0 offset 15

Common Pitfalls

Common mistakes when using Avro serializer with Kafka include:

• Not setting schema.registry.url, causing serialization errors.
• Using incompatible Avro schemas between producer and consumer.
• Forgetting to include Avro generated classes or schema files in the project.
• Using the wrong serializer class (e.g., StringSerializer instead of KafkaAvroSerializer).

Always ensure the Schema Registry is running and accessible, and schemas are compatible.

java

/* Wrong: Missing schema.registry.url */
Properties propsWrong = new Properties();
propsWrong.put("bootstrap.servers", "localhost:9092");
propsWrong.put("key.serializer", "io.confluent.kafka.serializers.KafkaAvroSerializer");
propsWrong.put("value.serializer", "io.confluent.kafka.serializers.KafkaAvroSerializer");

/* Right: Include schema.registry.url */
Properties propsRight = new Properties();
propsRight.put("bootstrap.servers", "localhost:9092");
propsRight.put("key.serializer", "io.confluent.kafka.serializers.KafkaAvroSerializer");
propsRight.put("value.serializer", "io.confluent.kafka.serializers.KafkaAvroSerializer");
propsRight.put("schema.registry.url", "http://localhost:8081");

Quick Reference

Remember these key points when using Avro serializer with Kafka:

• Set key.serializer and value.serializer to KafkaAvroSerializer.
• Always provide the schema.registry.url.
• Use generated Avro classes for type safety.
• Ensure Schema Registry service is running and reachable.

Key Takeaways

Configure Kafka producer with KafkaAvroSerializer and schema.registry.url to enable Avro serialization.

Use generated Avro classes to create messages that match the schema.

Ensure Schema Registry is running and schemas are compatible between producer and consumer.

Avoid missing schema.registry.url to prevent serialization failures.

Test sending and receiving messages to verify correct Avro serialization setup.