Avro schema definition in Kafka - Commands & Configuration

Introduction

When sending data between systems, you need a clear format so both sides understand it. Avro schema defines the structure of your data in a simple way, making sure everyone reads it the same way.

When you want to send messages in Kafka with a fixed structure so consumers can read them correctly.

When you need to evolve your data format over time without breaking old consumers.

When you want to validate data before sending it to avoid errors downstream.

When you want to compress data efficiently while keeping the schema.

When you want to share data formats between different programming languages easily.

Config File - user.avsc

user.avsc

{
  "type": "record",
  "name": "User",
  "namespace": "com.example",
  "fields": [
    {"name": "id", "type": "int"},
    {"name": "name", "type": "string"},
    {"name": "email", "type": ["null", "string"], "default": null}
  ]
}

This file defines an Avro schema for a User record.

type: The data type, here a record (like an object).

name: The name of the record.

namespace: A way to group schemas to avoid name clashes.

fields: The list of data fields with their names and types.

The email field can be null or a string, with null as default.

Commands

This command compiles the Avro schema file into Java classes in the current directory, so you can use the schema in your code.

Terminal

avro-tools compile schema user.avsc .

Expected Output

No output (command runs silently)

This command starts a Kafka producer that uses the Avro schema to send messages to the 'users' topic, ensuring messages follow the schema.

Terminal

kafka-avro-console-producer --broker-list localhost:9092 --topic users --property value.schema.file=user.avsc

Expected Output

>

→

--broker-list - Specifies Kafka broker addresses

→

--topic - Specifies the Kafka topic to send messages to

→

--property value.schema.file - Points to the Avro schema file for message validation

This command starts a Kafka consumer that reads messages from the 'users' topic from the beginning, printing keys and values decoded using the Avro schema.

Terminal

kafka-avro-console-consumer --bootstrap-server localhost:9092 --topic users --from-beginning --property print.key=true --property print.value=true

Expected Output

null	{"id":1,"name":"Alice","email":"[e-mail]"}

null	{"id":2,"name":"Bob","email":null}

→

--bootstrap-server - Specifies Kafka broker addresses

→

--topic - Specifies the Kafka topic to read messages from

→

--from-beginning - Reads all messages from the start

→

--property print.key=true - Prints the message key

→

--property print.value=true - Prints the message value

Key Concept

If you remember nothing else from this pattern, remember: Avro schema defines a clear, shared structure for your data so producers and consumers agree on the format.

Common Mistakes

Not specifying the schema file when producing messages

Messages won't be validated or serialized correctly, causing errors or unreadable data.

Always use the --property value.schema.file flag with the correct schema file when producing Avro messages.

Changing the schema fields without updating consumers

Consumers may fail to read messages or get wrong data if the schema changes unexpectedly.

Use schema evolution rules and update all consumers to handle new schema versions.

Using incompatible types in the schema fields

Data won't serialize or deserialize properly, causing runtime errors.

Use valid Avro types and unions for optional fields, like ["null", "string"] for nullable strings.

Summary

Define your data structure clearly in an Avro schema file with fields and types.

Use avro-tools or Kafka Avro console producer to compile and send messages with schema validation.

Use Kafka Avro console consumer to read and decode messages using the same schema.

Practice

1. What is the main purpose of an Avro schema in Kafka?

easy

A. To route messages to different topics

B. To define the structure and data types of messages

C. To compress messages for faster transmission

D. To encrypt messages for security

Solution

1. Step 1: Understand Avro schema role

   An Avro schema describes how data is organized and typed in messages.

2. Step 2: Compare options with schema purpose

   Only defining structure and data types matches the schema's role.

3. Final Answer:

   To define the structure and data types of messages -> Option B

4. Quick Check:

   Avro schema = data structure definition [OK]

Hint: Avro schema describes message format, not security or routing [OK]

Common Mistakes:

• Confusing schema with encryption or compression
• Thinking schema controls message routing
• Assuming schema is for message storage

2. Which of the following is the correct way to start an Avro record schema definition?

easy

A. {"type": "enum", "symbols": ["A", "B"]}

B. {"type": "array", "items": "string"}

C. {"type": "map", "values": "int"}

D. {"type": "record", "name": "User", "fields": []}

Solution

1. Step 1: Identify record schema syntax

   An Avro record schema starts with type "record", a name, and a fields array.

2. Step 2: Match options to record schema

   Only {"type": "record", "name": "User", "fields": []} matches the record schema format correctly.

3. Final Answer:

   {"type": "record", "name": "User", "fields": []} -> Option D

4. Quick Check:

   Record schema = type "record" with name and fields [OK]

Hint: Record schemas always start with type "record" and have fields [OK]

Common Mistakes:

• Using array or map types instead of record
• Missing the name or fields keys
• Confusing enum with record

3. Given this Avro schema snippet:

{"type": "record", "name": "Person", "fields": [{"name": "age", "type": "int"}]}

What is the expected type of the age field in messages?

medium

A. int

B. boolean

C. string

D. float

Solution

1. Step 1: Read the field definition

   The field "age" has type "int" in the schema.

2. Step 2: Determine expected data type

   Messages must have an integer value for "age" as per schema.

3. Final Answer:

   int -> Option A

4. Quick Check:

   Field type "int" means integer value expected [OK]

Hint: Field type in schema directly shows expected data type [OK]

Common Mistakes:

• Assuming string type by default
• Confusing int with float or boolean
• Ignoring the schema field type

4. Identify the error in this Avro schema snippet:

{"type": "record", "name": "Product", "fields": [{"name": "price", "type": "double"}, {"name": "available"}]}

medium

A. Missing type for field "available"

B. Invalid record name

C. Fields array should be empty

D. Type "double" is not allowed

Solution

1. Step 1: Check each field for required keys

   Each field must have a "name" and a "type" key.

2. Step 2: Find missing keys

   The field "available" lacks a "type" key, causing an error.

3. Final Answer:

   Missing type for field "available" -> Option A

4. Quick Check:

   Every field needs a type [OK]

Hint: Every field must have a type key defined [OK]

Common Mistakes:

• Forgetting to add type for all fields
• Thinking record name can be empty
• Assuming double type is invalid

5. You want to add an optional field email of type string with a default empty value to an Avro record schema. Which field definition is correct?

hard

A. {"name": "email", "type": "string"}

B. {"name": "email", "type": ["null", "string"], "default": ""}

C. {"name": "email", "type": ["string", "null"], "default": ""}

D. {"name": "email", "type": ["null", "string"], "default": null}

Solution

1. Step 1: Understand optional field in Avro

   Optional fields use a union type with "null" and the actual type, with a default value matching the first type in the union.

2. Step 2: Check default value and type order

   For default empty string "", the first type must be "string" followed by "null"; {"name": "email", "type": ["string", "null"], "default": ""} matches.

3. Final Answer:

   {"name": "email", "type": ["string", "null"], "default": ""} -> Option C

4. Quick Check:

   Optional with default "" = ["string", "null"] union [OK]

Hint: Union first type must match default; include null for optional [OK]

Common Mistakes:

• Not using union with null for optional fields
• Using null first with string default (invalid)
• Setting default to null when empty string needed