Transform and converter chains in Kafka - Step-by-Step Execution

Process Flow - Transform and converter chains

Input Data

↓

Converter 1: Deserialize

↓

Transform 1: Modify Data

↓

Converter 2: Serialize

↓

Output Data

Data flows through a chain of converters and transforms, converting formats and modifying content step-by-step.

Execution Sample

Kafka

key.converter=org.apache.kafka.connect.storage.StringConverter
value.converter=org.apache.kafka.connect.json.JsonConverter
value.converter.schemas.enable=false
transforms=InsertField
transforms.InsertField.type=org.apache.kafka.connect.transforms.InsertField$Value
transforms.InsertField.static.field=source
transforms.InsertField.static.value=app1

This config deserializes key as string, value as JSON without schema, then adds a static field 'source' with value 'app1' to the value.

Process Table

Step	Component	Input	Action	Output
1	Key Converter	Raw bytes	Deserialize bytes to string	"myKey"
2	Value Converter	Raw bytes	Deserialize bytes to JSON object	{"name":"Alice"}
3	Transform InsertField	{"name":"Alice"}	Add field 'source':'app1'	{"name":"Alice", "source":"app1"}
4	Value Converter	{"name":"Alice", "source":"app1"}	Serialize JSON object to bytes	Serialized bytes
5	Output	Serialized bytes	Send to Kafka topic	Message with key 'myKey' and value with added field

💡 All converters and transforms applied, message ready for output.

Status Tracker

Variable	Start	After Step 1	After Step 2	After Step 3	After Step 4	Final
key	Raw bytes	"myKey"	"myKey"	"myKey"	"myKey"	"myKey"
value	Raw bytes	Raw bytes	{"name":"Alice"}	{"name":"Alice", "source":"app1"}	Serialized bytes	Serialized bytes

Key Moments - 2 Insights

Why do we deserialize the value before applying the transform?

Because transforms work on structured data, not raw bytes. See execution_table step 2 and 3 where value is deserialized before adding the field.

Why is the value serialized again after the transform?

Kafka expects messages as bytes. After modifying the data, we serialize it back to bytes for sending. See execution_table step 4.

Visual Quiz - 3 Questions

Look at the execution table, what is the output of the Key Converter at step 1?

A Serialized bytes

B Raw bytes

C "myKey"

D {"name":"Alice"}

💡 Hint

Check the 'Output' column at step 1 in execution_table.

At which step is the field 'source' added to the value?

A Step 2

B Step 3

C Step 4

D Step 5

💡 Hint

Look at the 'Action' column describing adding the field in execution_table.

If we skip the final serialization (step 4), what would happen?

A Message would be sent as JSON object, causing errors

B Message would be sent as raw bytes

C Message key would be lost

D Transform would not apply

💡 Hint

Refer to the importance of serialization in step 4 and exit_note.

Concept Snapshot

Transform and converter chains process Kafka messages step-by-step.
Converters deserialize and serialize data formats.
Transforms modify the data between converters.
Order matters: deserialize -> transform -> serialize.
This ensures data is correctly formatted and enriched before sending.

Full Transcript

In Kafka, messages pass through converters and transforms in a chain. First, converters deserialize raw bytes into usable data formats like strings or JSON objects. Then, transforms modify this data, for example by adding fields. Finally, converters serialize the data back into bytes for Kafka to send. This flow ensures data is correctly handled and enriched. The example shows a key converted to string, a JSON value deserialized, a field added, then the value serialized again before output.