Serialization (String, JSON, Avro) in Kafka - Step-by-Step Execution

Process Flow - Serialization (String, JSON, Avro)

Start: Data in memory

↓

Choose serialization format

↓

Serialize data to bytes

↓

Send bytes to Kafka topic

↓

Consumer reads bytes

↓

Deserialize bytes back to data

↓

Use data in application

↓

End

Data is converted from memory format to bytes using a chosen serialization method, sent through Kafka, then converted back to usable data by the consumer.

Execution Sample

Kafka

producer.send(topic, value=message.encode('utf-8'))  # String
producer.send(topic, value=json.dumps(message_dict).encode('utf-8'))  # JSON
producer.send(topic, value=avro_serializer(message_dict))  # Avro

Shows how to serialize data as String, JSON, and Avro before sending to Kafka.

Process Table

Step	Data Type	Action	Serialized Output (bytes)	Kafka Topic Send
1	String	Encode string to UTF-8 bytes	b'Hello Kafka'	Sent to topic
2	JSON	Convert dict to JSON string, then UTF-8 bytes	b'{"key": "value"}'	Sent to topic
3	Avro	Serialize dict using Avro schema to bytes	b'\x06value'	Sent to topic
4	Consumer	Read bytes from topic	b'\x06value'	Received
5	Avro	Deserialize bytes back to dict	{"key": "value"}	Used in app
6	End	All data processed	-	-

💡 All data serialized, sent, received, and deserialized successfully.

Status Tracker

Variable	Start	After String Serialization	After JSON Serialization	After Avro Serialization	After Deserialization
message	"Hello Kafka"	b'Hello Kafka'	-	-	-
message_dict	{"key": "value"}	-	b'{"key": "value"}'	b'\x06value'	{"key": "value"}

Key Moments - 3 Insights

Why do we encode strings to bytes before sending to Kafka?

Kafka sends data as bytes, so strings must be encoded (see execution_table step 1) to convert text into bytes.

How is JSON different from a plain string in serialization?

JSON converts structured data (like dict) to a string format before encoding to bytes (execution_table step 2), unlike plain strings which are encoded directly.

What makes Avro serialization special compared to String and JSON?

Avro uses a schema to serialize data into a compact binary format (step 3), which is more efficient and supports schema evolution.

Visual Quiz - 3 Questions

Look at the execution_table, what is the serialized output for JSON at step 2?

A b'{"key": "value"}'

B b'Hello Kafka'

C b'\x06value'

D {"key": "value"}

💡 Hint

Check the 'Serialized Output (bytes)' column at step 2 in execution_table.

At which step does the consumer deserialize Avro bytes back to a dictionary?

A Step 3

B Step 5

C Step 1

D Step 2

💡 Hint

Look for 'Deserialize bytes back to dict' in the 'Action' column in execution_table.

If we skip encoding strings to bytes, what will happen when sending to Kafka?

A Data sends successfully as string

B Data converts automatically to JSON

C Kafka throws an error because it expects bytes

D Avro serialization happens instead

💡 Hint

Recall why encoding to bytes is needed from key_moments and execution_table step 1.

Concept Snapshot

Serialization converts data into bytes for Kafka.
String: encode text to UTF-8 bytes.
JSON: convert dict to JSON string, then encode.
Avro: use schema to serialize compact binary.
Consumer deserializes bytes back to data.
Serialization ensures data travels safely and efficiently.

Full Transcript

Serialization is the process of converting data into bytes so Kafka can send it. We can serialize simple strings by encoding them to UTF-8 bytes. For structured data like dictionaries, JSON serialization converts the dict to a JSON string, then encodes it to bytes. Avro serialization uses a schema to convert data into a compact binary format, which is efficient and supports schema changes. The producer sends these bytes to a Kafka topic. The consumer reads the bytes and deserializes them back to the original data format to use in the application. This process ensures data is safely and efficiently transmitted through Kafka.

Practice

1. What is the main purpose of serialization in Kafka?

easy

A. To convert data into a format suitable for transmission or storage

B. To encrypt data for security

C. To compress data to save space

D. To validate data against a schema

Solution

1. Step 1: Understand serialization concept

   Serialization means converting data into a format that can be sent or saved easily.

2. Step 2: Identify Kafka's use of serialization

   Kafka uses serialization to prepare data for sending between producers and consumers.

3. Final Answer:

   To convert data into a format suitable for transmission or storage -> Option A

4. Quick Check:

   Serialization = data format conversion [OK]

Hint: Serialization means data format conversion for sending [OK]

Common Mistakes:

• Confusing serialization with encryption
• Thinking serialization compresses data
• Assuming serialization validates data

2. Which of the following is the correct way to configure a Kafka producer to use JSON serialization in Java?

easy

A. props.put("value.serializer", "org.apache.kafka.connect.json.JsonSerializer");

B. props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");

C. props.put("value.serializer", "org.apache.kafka.common.serialization.ByteArraySerializer");

D. props.put("value.serializer", "io.confluent.kafka.serializers.KafkaAvroSerializer");

Solution

1. Step 1: Identify JSON serializer class

   The JSON serializer class in Kafka Connect is "org.apache.kafka.connect.json.JsonSerializer".

2. Step 2: Match configuration property

   The producer property for value serializer is "value.serializer" and should be set to the JSON serializer class.

3. Final Answer:

   props.put("value.serializer", "org.apache.kafka.connect.json.JsonSerializer"); -> Option A

4. Quick Check:

   JSON serializer = org.apache.kafka.connect.json.JsonSerializer [OK]

Hint: JSON serializer class is in kafka.connect.json package [OK]

Common Mistakes:

• Using StringSerializer for JSON data
• Confusing Avro serializer with JSON serializer
• Using ByteArraySerializer for JSON

3. Given the following Kafka producer code snippet using Avro serialization, what will be the output if the schema registry URL is incorrect?

props.put("value.serializer", "io.confluent.kafka.serializers.KafkaAvroSerializer");
props.put("schema.registry.url", "http://wrong-url:8081");
producer.send(new ProducerRecord<>("topic", avroRecord));

medium

A. NullPointerException during serialization

B. Message sent successfully without errors

C. SerializationException due to schema registry connection failure

D. Producer silently drops the message

Solution

1. Step 1: Understand Avro serialization dependency

   Avro serialization requires connecting to the schema registry to fetch schemas.

2. Step 2: Analyze effect of wrong schema registry URL

   If the URL is wrong, the serializer cannot fetch the schema and throws a SerializationException.

3. Final Answer:

   SerializationException due to schema registry connection failure -> Option C

4. Quick Check:

   Wrong schema URL = SerializationException [OK]

Hint: Avro needs schema registry; wrong URL causes SerializationException [OK]

Common Mistakes:

• Expecting message to send without error
• Thinking NullPointerException occurs
• Assuming message is dropped silently

4. Identify the error in this Kafka producer configuration for String serialization:

Properties props = new Properties();
props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer", "org.apache.kafka.common.serialization.StringDeserializer");
props.put("bootstrap.servers", "localhost:9092");
Producer producer = new KafkaProducer<>(props);

medium

A. Incorrect generic types in KafkaProducer declaration

B. Using StringDeserializer instead of StringSerializer for value.serializer

C. Missing bootstrap.servers configuration

D. No error, configuration is correct

Solution

1. Step 1: Check serializer and deserializer usage

   Producer requires serializers for key and value, not deserializers.

2. Step 2: Identify wrong class usage

   Value serializer is incorrectly set to StringDeserializer instead of StringSerializer.

3. Final Answer:

   Using StringDeserializer instead of StringSerializer for value.serializer -> Option B

4. Quick Check:

   Producer needs serializers, not deserializers [OK]

Hint: Producer uses serializers, not deserializers [OK]

Common Mistakes:

• Mixing serializer and deserializer classes
• Forgetting bootstrap.servers (not asked here)
• Incorrect generic types (not the main error)

5. You want to send complex nested data with schema evolution support in Kafka. Which serialization method should you choose and why?

hard

A. ByteArray serialization, because it handles any data type

B. JSON serialization, because it supports nested data and is human-readable

C. String serialization, because it is simple and fast

D. Avro serialization, because it supports schema evolution and compact binary format

Solution

1. Step 1: Understand requirements for nested data and schema evolution

   Complex nested data needs a format that supports schemas and changes over time.

2. Step 2: Compare serialization options

   String is simple but no schema support; JSON supports nested but no schema evolution; Avro supports both schema evolution and compact binary format.

3. Final Answer:

   Avro serialization, because it supports schema evolution and compact binary format -> Option D

4. Quick Check:

   Schema evolution + nested data = Avro [OK]

Hint: Avro supports schema evolution and nested data best [OK]

Common Mistakes:

• Choosing String for complex data
• Assuming JSON supports schema evolution
• Using ByteArray without schema control