Serialization (String, JSON, Avro) in Kafka - Step-by-Step Execution

Process Flow - Serialization (String, JSON, Avro)

Start: Data in memory

↓

Choose serialization format

↓

Serialize data to bytes

↓

Send bytes to Kafka topic

↓

Consumer reads bytes

↓

Deserialize bytes back to data

↓

Use data in application

↓

End

Data is converted from memory format to bytes using a chosen serialization method, sent through Kafka, then converted back to usable data by the consumer.

Execution Sample

Kafka

producer.send(topic, value=message.encode('utf-8'))  # String
producer.send(topic, value=json.dumps(message_dict).encode('utf-8'))  # JSON
producer.send(topic, value=avro_serializer(message_dict))  # Avro

Shows how to serialize data as String, JSON, and Avro before sending to Kafka.

Process Table

Step	Data Type	Action	Serialized Output (bytes)	Kafka Topic Send
1	String	Encode string to UTF-8 bytes	b'Hello Kafka'	Sent to topic
2	JSON	Convert dict to JSON string, then UTF-8 bytes	b'{"key": "value"}'	Sent to topic
3	Avro	Serialize dict using Avro schema to bytes	b'\x06value'	Sent to topic
4	Consumer	Read bytes from topic	b'\x06value'	Received
5	Avro	Deserialize bytes back to dict	{"key": "value"}	Used in app
6	End	All data processed	-	-

💡 All data serialized, sent, received, and deserialized successfully.

Status Tracker

Variable	Start	After String Serialization	After JSON Serialization	After Avro Serialization	After Deserialization
message	"Hello Kafka"	b'Hello Kafka'	-	-	-
message_dict	{"key": "value"}	-	b'{"key": "value"}'	b'\x06value'	{"key": "value"}

Key Moments - 3 Insights

Why do we encode strings to bytes before sending to Kafka?

Kafka sends data as bytes, so strings must be encoded (see execution_table step 1) to convert text into bytes.

How is JSON different from a plain string in serialization?

JSON converts structured data (like dict) to a string format before encoding to bytes (execution_table step 2), unlike plain strings which are encoded directly.

What makes Avro serialization special compared to String and JSON?

Avro uses a schema to serialize data into a compact binary format (step 3), which is more efficient and supports schema evolution.

Visual Quiz - 3 Questions

Look at the execution_table, what is the serialized output for JSON at step 2?

A b'{"key": "value"}'

B b'Hello Kafka'

C b'\x06value'

D {"key": "value"}

💡 Hint

Check the 'Serialized Output (bytes)' column at step 2 in execution_table.

At which step does the consumer deserialize Avro bytes back to a dictionary?

A Step 3

B Step 5

C Step 1

D Step 2

💡 Hint

Look for 'Deserialize bytes back to dict' in the 'Action' column in execution_table.

If we skip encoding strings to bytes, what will happen when sending to Kafka?

A Data sends successfully as string

B Data converts automatically to JSON

C Kafka throws an error because it expects bytes

D Avro serialization happens instead

💡 Hint

Recall why encoding to bytes is needed from key_moments and execution_table step 1.

Concept Snapshot

Serialization converts data into bytes for Kafka.
String: encode text to UTF-8 bytes.
JSON: convert dict to JSON string, then encode.
Avro: use schema to serialize compact binary.
Consumer deserializes bytes back to data.
Serialization ensures data travels safely and efficiently.

Full Transcript

Serialization is the process of converting data into bytes so Kafka can send it. We can serialize simple strings by encoding them to UTF-8 bytes. For structured data like dictionaries, JSON serialization converts the dict to a JSON string, then encodes it to bytes. Avro serialization uses a schema to convert data into a compact binary format, which is efficient and supports schema changes. The producer sends these bytes to a Kafka topic. The consumer reads the bytes and deserializes them back to the original data format to use in the application. This process ensures data is safely and efficiently transmitted through Kafka.