Avro schema definition in Kafka - Step-by-Step Execution

Process Flow - Avro schema definition

Start: Define schema fields

↓

Specify field names and types

↓

Set record name and namespace

↓

Add optional default values

↓

Validate schema format

↓

Use schema for serialization/deserialization

↓

End

This flow shows how to create an Avro schema by defining fields, setting metadata, validating, and then using it for data encoding.

Execution Sample

Kafka

 {
  "type": "record",
  "name": "User",
  "fields": [
    {"name": "id", "type": "int"},
    {"name": "name", "type": "string"}
  ]
}

Defines a simple Avro schema for a User record with id and name fields.

Process Table

Step	Action	Field/Property	Value	Result
1	Define schema type	"type"	"record"	Schema type set to record
2	Set record name	"name"	"User"	Record named User
3	Add field	"fields[0]"	{"name": "id", "type": "int"}	Field id of type int added
4	Add field	"fields[1]"	{"name": "name", "type": "string"}	Field name of type string added
5	Validate schema	Full schema	Valid JSON with required keys	Schema is valid
6	Use schema	Serialization	Encode User data	Data serialized using schema
7	Use schema	Deserialization	Decode User data	Data deserialized back to User object

💡 All fields defined and schema validated; ready for use in Kafka serialization.

Status Tracker

Variable	Start	After Step 3	After Step 4	Final
schema	{}	{"type": "record", "name": "User", "fields": [{"name": "id", "type": "int"}]}	{"type": "record", "name": "User", "fields": [{"name": "id", "type": "int"}, {"name": "name", "type": "string"}]}	Complete schema object

Key Moments - 3 Insights

Why do we need to specify the "type" as "record" at the start?

The "type": "record" tells Avro this schema defines a structured object with named fields, not just a simple type. See execution_table step 1.

Can fields be added without a name or type?

No, each field must have a "name" and a "type" to be valid. Missing these causes schema validation to fail. See execution_table steps 3 and 4.

What happens if the schema is not valid JSON?

The schema cannot be used for serialization or deserialization until fixed. Validation step (5) ensures the schema is correct.

Visual Quiz - 3 Questions

Look at the execution_table, what is the value of the "name" property after step 2?

A "record"

B "User"

C "id"

D "fields"

💡 Hint

Check the 'Field/Property' and 'Value' columns at step 2 in execution_table.

At which step is the second field "name" added to the schema?

A Step 1

B Step 3

C Step 4

D Step 5

💡 Hint

Look for the action 'Add field' and field details in execution_table rows.

If the "type" property was missing from a field, what would happen during validation?

A Schema validation would fail

B Field would default to string type

C Schema would still be valid

D Kafka would ignore the field

💡 Hint

Refer to key_moments about required field properties and validation step.

Concept Snapshot

Avro schema defines data structure for Kafka.
Use "type": "record" for objects.
List fields with "name" and "type".
Validate schema before use.
Schema enables data serialization/deserialization.

Full Transcript

This visual execution shows how to define an Avro schema for Kafka. First, set the schema type to 'record' to indicate a structured object. Then, assign a name to the record, like 'User'. Next, add fields one by one, each with a name and a type, such as 'id' as int and 'name' as string. After defining all fields, validate the schema to ensure it is correct JSON and has all required parts. Once validated, the schema can be used to serialize data into bytes for Kafka and deserialize bytes back into objects. Key points include always specifying 'type' and 'name' for fields and validating the schema before use. The execution table tracks each step, showing how the schema object grows and is checked. The variable tracker shows the schema's state after each field addition. This step-by-step helps beginners see how Avro schemas are built and used in Kafka.

Practice

1. What is the main purpose of an Avro schema in Kafka?

easy

A. To route messages to different topics

B. To define the structure and data types of messages

C. To compress messages for faster transmission

D. To encrypt messages for security

Solution

1. Step 1: Understand Avro schema role

   An Avro schema describes how data is organized and typed in messages.

2. Step 2: Compare options with schema purpose

   Only defining structure and data types matches the schema's role.

3. Final Answer:

   To define the structure and data types of messages -> Option B

4. Quick Check:

   Avro schema = data structure definition [OK]

Hint: Avro schema describes message format, not security or routing [OK]

Common Mistakes:

• Confusing schema with encryption or compression
• Thinking schema controls message routing
• Assuming schema is for message storage

2. Which of the following is the correct way to start an Avro record schema definition?

easy

A. {"type": "enum", "symbols": ["A", "B"]}

B. {"type": "array", "items": "string"}

C. {"type": "map", "values": "int"}

D. {"type": "record", "name": "User", "fields": []}

Solution

1. Step 1: Identify record schema syntax

   An Avro record schema starts with type "record", a name, and a fields array.

2. Step 2: Match options to record schema

   Only {"type": "record", "name": "User", "fields": []} matches the record schema format correctly.

3. Final Answer:

   {"type": "record", "name": "User", "fields": []} -> Option D

4. Quick Check:

   Record schema = type "record" with name and fields [OK]

Hint: Record schemas always start with type "record" and have fields [OK]

Common Mistakes:

• Using array or map types instead of record
• Missing the name or fields keys
• Confusing enum with record

3. Given this Avro schema snippet:

{"type": "record", "name": "Person", "fields": [{"name": "age", "type": "int"}]}

What is the expected type of the age field in messages?

medium

A. int

B. boolean

C. string

D. float

Solution

1. Step 1: Read the field definition

   The field "age" has type "int" in the schema.

2. Step 2: Determine expected data type

   Messages must have an integer value for "age" as per schema.

3. Final Answer:

   int -> Option A

4. Quick Check:

   Field type "int" means integer value expected [OK]

Hint: Field type in schema directly shows expected data type [OK]

Common Mistakes:

• Assuming string type by default
• Confusing int with float or boolean
• Ignoring the schema field type

4. Identify the error in this Avro schema snippet:

{"type": "record", "name": "Product", "fields": [{"name": "price", "type": "double"}, {"name": "available"}]}

medium

A. Missing type for field "available"

B. Invalid record name

C. Fields array should be empty

D. Type "double" is not allowed

Solution

1. Step 1: Check each field for required keys

   Each field must have a "name" and a "type" key.

2. Step 2: Find missing keys

   The field "available" lacks a "type" key, causing an error.

3. Final Answer:

   Missing type for field "available" -> Option A

4. Quick Check:

   Every field needs a type [OK]

Hint: Every field must have a type key defined [OK]

Common Mistakes:

• Forgetting to add type for all fields
• Thinking record name can be empty
• Assuming double type is invalid

5. You want to add an optional field email of type string with a default empty value to an Avro record schema. Which field definition is correct?

hard

A. {"name": "email", "type": "string"}

B. {"name": "email", "type": ["null", "string"], "default": ""}

C. {"name": "email", "type": ["string", "null"], "default": ""}

D. {"name": "email", "type": ["null", "string"], "default": null}

Solution

1. Step 1: Understand optional field in Avro

   Optional fields use a union type with "null" and the actual type, with a default value matching the first type in the union.

2. Step 2: Check default value and type order

   For default empty string "", the first type must be "string" followed by "null"; {"name": "email", "type": ["string", "null"], "default": ""} matches.

3. Final Answer:

   {"name": "email", "type": ["string", "null"], "default": ""} -> Option C

4. Quick Check:

   Optional with default "" = ["string", "null"] union [OK]

Hint: Union first type must match default; include null for optional [OK]

Common Mistakes:

• Not using union with null for optional fields
• Using null first with string default (invalid)
• Setting default to null when empty string needed