Serialization (String, JSON, Avro) in Kafka - Commands & Configuration

Introduction

Serialization converts data into a format that Kafka can send and receive. It solves the problem of sharing data between different systems by making data easy to transport and understand.

When you want to send simple text messages between services using Kafka.

When you need to send structured data like user info in JSON format through Kafka topics.

When you want to ensure data is compact and schema-validated using Avro serialization.

When different applications need to read the same Kafka messages with a shared schema.

When you want to reduce errors by enforcing data format rules before sending messages.

Config File - producer.properties

producer.properties

bootstrap.servers=localhost:9092
key.serializer=org.apache.kafka.common.serialization.StringSerializer
value.serializer=io.confluent.kafka.serializers.KafkaAvroSerializer
schema.registry.url=http://localhost:8081

This configuration file sets up a Kafka producer to connect to the Kafka server at localhost:9092.

The key.serializer is set to StringSerializer to send keys as plain text.

The value.serializer is set to KafkaAvroSerializer to send values in Avro format.

The schema.registry.url points to the schema registry service that manages Avro schemas.

Commands

Starts a Kafka console producer to send messages to 'example-topic'. It allows typing messages with keys separated by ':'.

Terminal

kafka-console-producer --broker-list localhost:9092 --topic example-topic --property parse.key=true --property key.separator=:

Expected Output

No output (command runs silently)

→

--broker-list - Specifies Kafka server address

→

--topic - Specifies the topic to send messages to

→

--property parse.key=true - Enables key parsing from input

→

--property key.separator=: - Defines ':' as the separator between key and value

Sends a JSON string message to 'json-topic' using StringSerializer for the value.

Terminal

echo '{"name":"Alice","age":30}' | kafka-console-producer --broker-list localhost:9092 --topic json-topic --property value.serializer=org.apache.kafka.common.serialization.StringSerializer

Expected Output

No output (command runs silently)

→

--broker-list - Specifies Kafka server address

→

--topic - Specifies the topic to send messages to

→

--property value.serializer=org.apache.kafka.common.serialization.StringSerializer - Sends message value as plain string

Starts an Avro console producer to send messages with a defined Avro schema to 'avro-topic'.

Terminal

kafka-avro-console-producer --broker-list localhost:9092 --topic avro-topic --property schema.registry.url=http://localhost:8081 --property value.schema='{"type":"record","name":"User","fields":[{"name":"name","type":"string"},{"name":"age","type":"int"}]}'

Expected Output

No output (command runs silently)

→

--broker-list - Specifies Kafka server address

→

--topic - Specifies the topic to send messages to

→

--property schema.registry.url - Points to the schema registry service

→

--property value.schema - Defines the Avro schema for the message value

Starts a Kafka console consumer to read messages from 'example-topic' from the beginning, showing keys and values separated by ':'.

Terminal

kafka-console-consumer --bootstrap-server localhost:9092 --topic example-topic --from-beginning --property print.key=true --property key.separator=:

Expected Output

myKey:myValue

user1:hello

→

--bootstrap-server - Specifies Kafka server address

→

--topic - Specifies the topic to read messages from

→

--from-beginning - Reads all messages from the start

→

--property print.key=true - Shows message keys in output

→

--property key.separator=: - Separates key and value in output

Key Concept

If you remember nothing else from this pattern, remember: serialization turns data into a format Kafka can send and receive, and choosing the right serializer ensures your data is understood correctly.

Common Mistakes

Using StringSerializer for complex structured data without converting it to JSON or Avro.

Kafka will treat the data as plain text, losing structure and causing errors in consumers expecting structured data.

Use JSON or Avro serializers for structured data and ensure consumers use matching deserializers.

Not setting the schema.registry.url when using Avro serialization.

The producer cannot register or retrieve schemas, causing message send failures.

Always configure schema.registry.url to point to your running schema registry service.

Sending messages without keys when keys are expected for partitioning or message ordering.

Kafka may distribute messages unevenly or lose ordering guarantees.

Include keys in messages and configure the producer to parse and send keys properly.

Summary

Configure Kafka producers with appropriate serializers for your data format: String, JSON, or Avro.

Use console producer commands to send messages with keys and values to Kafka topics.

Use console consumer commands to read messages, showing keys and values for verification.

Avro serialization requires a schema registry to manage and validate data schemas.

Practice

1. What is the main purpose of serialization in Kafka?

easy

A. To convert data into a format suitable for transmission or storage

B. To encrypt data for security

C. To compress data to save space

D. To validate data against a schema

Solution

1. Step 1: Understand serialization concept

   Serialization means converting data into a format that can be sent or saved easily.

2. Step 2: Identify Kafka's use of serialization

   Kafka uses serialization to prepare data for sending between producers and consumers.

3. Final Answer:

   To convert data into a format suitable for transmission or storage -> Option A

4. Quick Check:

   Serialization = data format conversion [OK]

Hint: Serialization means data format conversion for sending [OK]

Common Mistakes:

• Confusing serialization with encryption
• Thinking serialization compresses data
• Assuming serialization validates data

2. Which of the following is the correct way to configure a Kafka producer to use JSON serialization in Java?

easy

A. props.put("value.serializer", "org.apache.kafka.connect.json.JsonSerializer");

B. props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");

C. props.put("value.serializer", "org.apache.kafka.common.serialization.ByteArraySerializer");

D. props.put("value.serializer", "io.confluent.kafka.serializers.KafkaAvroSerializer");

Solution

1. Step 1: Identify JSON serializer class

   The JSON serializer class in Kafka Connect is "org.apache.kafka.connect.json.JsonSerializer".

2. Step 2: Match configuration property

   The producer property for value serializer is "value.serializer" and should be set to the JSON serializer class.

3. Final Answer:

   props.put("value.serializer", "org.apache.kafka.connect.json.JsonSerializer"); -> Option A

4. Quick Check:

   JSON serializer = org.apache.kafka.connect.json.JsonSerializer [OK]

Hint: JSON serializer class is in kafka.connect.json package [OK]

Common Mistakes:

• Using StringSerializer for JSON data
• Confusing Avro serializer with JSON serializer
• Using ByteArraySerializer for JSON

3. Given the following Kafka producer code snippet using Avro serialization, what will be the output if the schema registry URL is incorrect?

props.put("value.serializer", "io.confluent.kafka.serializers.KafkaAvroSerializer");
props.put("schema.registry.url", "http://wrong-url:8081");
producer.send(new ProducerRecord<>("topic", avroRecord));

medium

A. NullPointerException during serialization

B. Message sent successfully without errors

C. SerializationException due to schema registry connection failure

D. Producer silently drops the message

Solution

1. Step 1: Understand Avro serialization dependency

   Avro serialization requires connecting to the schema registry to fetch schemas.

2. Step 2: Analyze effect of wrong schema registry URL

   If the URL is wrong, the serializer cannot fetch the schema and throws a SerializationException.

3. Final Answer:

   SerializationException due to schema registry connection failure -> Option C

4. Quick Check:

   Wrong schema URL = SerializationException [OK]

Hint: Avro needs schema registry; wrong URL causes SerializationException [OK]

Common Mistakes:

• Expecting message to send without error
• Thinking NullPointerException occurs
• Assuming message is dropped silently

4. Identify the error in this Kafka producer configuration for String serialization:

Properties props = new Properties();
props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer", "org.apache.kafka.common.serialization.StringDeserializer");
props.put("bootstrap.servers", "localhost:9092");
Producer producer = new KafkaProducer<>(props);

medium

A. Incorrect generic types in KafkaProducer declaration

B. Using StringDeserializer instead of StringSerializer for value.serializer

C. Missing bootstrap.servers configuration

D. No error, configuration is correct

Solution

1. Step 1: Check serializer and deserializer usage

   Producer requires serializers for key and value, not deserializers.

2. Step 2: Identify wrong class usage

   Value serializer is incorrectly set to StringDeserializer instead of StringSerializer.

3. Final Answer:

   Using StringDeserializer instead of StringSerializer for value.serializer -> Option B

4. Quick Check:

   Producer needs serializers, not deserializers [OK]

Hint: Producer uses serializers, not deserializers [OK]

Common Mistakes:

• Mixing serializer and deserializer classes
• Forgetting bootstrap.servers (not asked here)
• Incorrect generic types (not the main error)

5. You want to send complex nested data with schema evolution support in Kafka. Which serialization method should you choose and why?

hard

A. ByteArray serialization, because it handles any data type

B. JSON serialization, because it supports nested data and is human-readable

C. String serialization, because it is simple and fast

D. Avro serialization, because it supports schema evolution and compact binary format

Solution

1. Step 1: Understand requirements for nested data and schema evolution

   Complex nested data needs a format that supports schemas and changes over time.

2. Step 2: Compare serialization options

   String is simple but no schema support; JSON supports nested but no schema evolution; Avro supports both schema evolution and compact binary format.

3. Final Answer:

   Avro serialization, because it supports schema evolution and compact binary format -> Option D

4. Quick Check:

   Schema evolution + nested data = Avro [OK]

Hint: Avro supports schema evolution and nested data best [OK]

Common Mistakes:

• Choosing String for complex data
• Assuming JSON supports schema evolution
• Using ByteArray without schema control