Avro schema definition in Kafka - Cheat Sheet & Quick Revision

Recall & Review

beginner

What is an Avro schema?

An Avro schema is a JSON-based format that defines the structure of data, including fields and their types, used to serialize and deserialize data in Kafka.

beginner

What are the main components of an Avro schema?

The main components are:
1. type: The data type (e.g., record, string, int).
2. name: The name of the record.
3. fields: A list of fields with their names and types.

beginner

How do you define a record type in Avro schema?

You define a record by specifying "type": "record", giving it a "name", and listing its "fields" with their names and types.

intermediate

What is the purpose of the namespace in an Avro schema?

The namespace groups related schemas together, like a folder name, to avoid name conflicts between schemas with the same record name.

intermediate

How do you specify optional fields in an Avro schema?

You use a union type with "null" and the actual type, for example: ["null", "string"]. This means the field can be a string or null (optional).

Which data type is used to define a complex structure in Avro schema?

A boolean

B record

C int

D string

The 'record' type defines a complex structure with multiple fields.

How do you make a field optional in Avro schema?

A Leave the field out

B Set 'optional': true

C Use 'nullable': true

D Use a union with 'null' and the field type

Avro uses a union type with 'null' to indicate optional fields.

What is the purpose of the 'namespace' in an Avro schema?

A To specify the data type

B To list the fields

C To group related schemas and avoid name conflicts

D To define default values

Namespace helps organize schemas and prevent name clashes.

Which format is used to write Avro schemas?

A JSON

B XML

C YAML

D CSV

Avro schemas are written in JSON format.

What does the 'fields' array in an Avro schema contain?

A List of field names and their types

B List of namespaces

C List of default values

D List of schema versions

The 'fields' array defines each field's name and type.

Explain how to define a simple Avro schema for a user record with name (string) and age (int).

Think about the 'type', 'name', and 'fields' keys.

Describe how to make a field optional in an Avro schema and why it is useful.

Consider how to allow a field to be missing or null.

Practice

1. What is the main purpose of an Avro schema in Kafka?

easy

A. To route messages to different topics

B. To define the structure and data types of messages

C. To compress messages for faster transmission

D. To encrypt messages for security

Solution

1. Step 1: Understand Avro schema role

   An Avro schema describes how data is organized and typed in messages.

2. Step 2: Compare options with schema purpose

   Only defining structure and data types matches the schema's role.

3. Final Answer:

   To define the structure and data types of messages -> Option B

4. Quick Check:

   Avro schema = data structure definition [OK]

Hint: Avro schema describes message format, not security or routing [OK]

Common Mistakes:

• Confusing schema with encryption or compression
• Thinking schema controls message routing
• Assuming schema is for message storage

2. Which of the following is the correct way to start an Avro record schema definition?

easy

A. {"type": "enum", "symbols": ["A", "B"]}

B. {"type": "array", "items": "string"}

C. {"type": "map", "values": "int"}

D. {"type": "record", "name": "User", "fields": []}

Solution

1. Step 1: Identify record schema syntax

   An Avro record schema starts with type "record", a name, and a fields array.

2. Step 2: Match options to record schema

   Only {"type": "record", "name": "User", "fields": []} matches the record schema format correctly.

3. Final Answer:

   {"type": "record", "name": "User", "fields": []} -> Option D

4. Quick Check:

   Record schema = type "record" with name and fields [OK]

Hint: Record schemas always start with type "record" and have fields [OK]

Common Mistakes:

• Using array or map types instead of record
• Missing the name or fields keys
• Confusing enum with record

3. Given this Avro schema snippet:

{"type": "record", "name": "Person", "fields": [{"name": "age", "type": "int"}]}

What is the expected type of the age field in messages?

medium

A. int

B. boolean

C. string

D. float

Solution

1. Step 1: Read the field definition

   The field "age" has type "int" in the schema.

2. Step 2: Determine expected data type

   Messages must have an integer value for "age" as per schema.

3. Final Answer:

   int -> Option A

4. Quick Check:

   Field type "int" means integer value expected [OK]

Hint: Field type in schema directly shows expected data type [OK]

Common Mistakes:

• Assuming string type by default
• Confusing int with float or boolean
• Ignoring the schema field type

4. Identify the error in this Avro schema snippet:

{"type": "record", "name": "Product", "fields": [{"name": "price", "type": "double"}, {"name": "available"}]}

medium

A. Missing type for field "available"

B. Invalid record name

C. Fields array should be empty

D. Type "double" is not allowed

Solution

1. Step 1: Check each field for required keys

   Each field must have a "name" and a "type" key.

2. Step 2: Find missing keys

   The field "available" lacks a "type" key, causing an error.

3. Final Answer:

   Missing type for field "available" -> Option A

4. Quick Check:

   Every field needs a type [OK]

Hint: Every field must have a type key defined [OK]

Common Mistakes:

• Forgetting to add type for all fields
• Thinking record name can be empty
• Assuming double type is invalid

5. You want to add an optional field email of type string with a default empty value to an Avro record schema. Which field definition is correct?

hard

A. {"name": "email", "type": "string"}

B. {"name": "email", "type": ["null", "string"], "default": ""}

C. {"name": "email", "type": ["string", "null"], "default": ""}

D. {"name": "email", "type": ["null", "string"], "default": null}

Solution

1. Step 1: Understand optional field in Avro

   Optional fields use a union type with "null" and the actual type, with a default value matching the first type in the union.

2. Step 2: Check default value and type order

   For default empty string "", the first type must be "string" followed by "null"; {"name": "email", "type": ["string", "null"], "default": ""} matches.

3. Final Answer:

   {"name": "email", "type": ["string", "null"], "default": ""} -> Option C

4. Quick Check:

   Optional with default "" = ["string", "null"] union [OK]

Hint: Union first type must match default; include null for optional [OK]

Common Mistakes:

• Not using union with null for optional fields
• Using null first with string default (invalid)
• Setting default to null when empty string needed