JSON Schema and Protobuf support in Kafka - Commands & Configuration

Introduction

Kafka can use JSON Schema and Protobuf to define the structure of messages. This helps ensure that data sent between services is consistent and understood by all parts of the system.

When you want to make sure all messages follow a specific format before they are sent or received.

When multiple teams or services share data and need a common agreement on message structure.

When you want to catch errors early by validating messages against a schema.

When you want to evolve message formats safely without breaking existing consumers.

When you want to use efficient binary formats like Protobuf for faster data transfer.

Config File - schema-registry-config.properties

schema-registry-config.properties

schema.registry.url=http://localhost:8081

# Enable support for JSON Schema and Protobuf
schema.registry.enable.json.schema=true
schema.registry.enable.protobuf=true

This configuration file sets up the Kafka Schema Registry to support JSON Schema and Protobuf formats. The schema.registry.url points to the running Schema Registry service. The flags schema.registry.enable.json.schema and schema.registry.enable.protobuf enable support for these schema types so Kafka can validate and serialize messages accordingly.

Commands

This command registers a JSON Schema for a Kafka topic named 'user'. It tells the Schema Registry what the message structure should be for the 'user-value' subject.

Terminal

curl -X POST -H "Content-Type: application/vnd.schemaregistry.v1+json" --data '{"schema": "{\"type\":\"record\",\"name\":\"User\",\"fields\":[{\"name\":\"name\",\"type\":\"string\"}]}"}' http://localhost:8081/subjects/user-value/versions

Expected Output

{"id":1}

→

-X POST - Send data to register the schema

→

-H "Content-Type: application/vnd.schemaregistry.v1+json" - Specify the content type for the Schema Registry API

This command registers a Protobuf schema for the Kafka topic 'user-protobuf'. It defines the message format using Protobuf syntax.

Terminal

curl -X POST -H "Content-Type: application/vnd.schemaregistry.v1+json" --data '{"schema": "syntax = \"proto3\"; message User { string name = 1; }"}' http://localhost:8081/subjects/user-protobuf-value/versions

Expected Output

{"id":2}

→

-X POST - Send data to register the schema

→

-H "Content-Type: application/vnd.schemaregistry.v1+json" - Specify the content type for the Schema Registry API

This command starts a Kafka producer that sends messages to the 'user' topic using the registered JSON Schema with ID 1. It ensures messages follow the schema.

Terminal

kafka-console-producer --broker-list localhost:9092 --topic user --property value.schema.id=1 --property value.schema.type=JSON

Expected Output

No output (command runs silently)

→

--property value.schema.id=1 - Use the JSON Schema with ID 1 for message validation

→

--property value.schema.type=JSON - Specify that the message format is JSON

This command starts a Kafka consumer that reads messages from the 'user' topic and validates them against the JSON Schema with ID 1.

Terminal

kafka-console-consumer --bootstrap-server localhost:9092 --topic user --from-beginning --property print.key=true --property value.schema.id=1 --property value.schema.type=JSON

Expected Output

(shows messages in JSON format as they arrive)

→

--property value.schema.id=1 - Use the JSON Schema with ID 1 to decode messages

→

--property value.schema.type=JSON - Specify that the message format is JSON

Key Concept

If you remember nothing else from this pattern, remember: JSON Schema and Protobuf let Kafka check and enforce message formats so all parts of your system agree on data structure.

Common Mistakes

Not enabling JSON Schema or Protobuf support in the Schema Registry configuration.

Kafka will reject or fail to validate messages using these schema types.

Set schema.registry.enable.json.schema=true and schema.registry.enable.protobuf=true in the Schema Registry config.

Registering schemas with incorrect JSON escaping or invalid Protobuf syntax.

Schema Registry will reject the schema registration request.

Ensure JSON strings are properly escaped and Protobuf syntax is valid before registering.

Producing or consuming messages without specifying the correct schema ID and type properties.

Kafka clients won't know how to serialize or deserialize messages properly, causing errors.

Always specify --property value.schema.id and --property value.schema.type when using schema-based messages.

Summary

Configure Schema Registry to enable JSON Schema and Protobuf support.

Register your message schemas using the Schema Registry REST API.

Use Kafka console producer and consumer with schema ID and type properties to send and receive validated messages.

Practice

1. What is the main purpose of using JSON Schema or Protobuf with Kafka messages?

easy

A. To define the structure and format of messages so producers and consumers agree

B. To encrypt Kafka messages for security

C. To speed up Kafka message delivery by skipping validation

D. To store Kafka messages permanently in a database

Solution

1. Step 1: Understand message format agreement

   JSON Schema and Protobuf define how messages look so both sender and receiver understand the data.

2. Step 2: Identify the main purpose

   This agreement prevents errors and confusion when reading messages.

3. Final Answer:

   To define the structure and format of messages so producers and consumers agree -> Option A

4. Quick Check:

   Message format agreement = C [OK]

Hint: They define message format so systems agree on data [OK]

Common Mistakes:

• Thinking JSON Schema encrypts messages
• Believing Protobuf speeds delivery by skipping validation
• Confusing schema with storage

2. Which of the following is the correct way to specify a Protobuf schema for a Kafka message?

easy

A. { 'name': 'string', 'id': 'int' }

B. schema { field name: string; field id: int; }

C. stringint

D. message Person { string name = 1; int32 id = 2; }

Solution

1. Step 1: Recognize Protobuf syntax

   Protobuf uses the 'message' keyword with fields and their types and numbers.

2. Step 2: Match the correct syntax

   message Person { string name = 1; int32 id = 2; } shows correct Protobuf message definition syntax.

3. Final Answer:

   message Person { string name = 1; int32 id = 2; } -> Option D

4. Quick Check:

   Protobuf syntax = B [OK]

Hint: Protobuf uses 'message' keyword and numbered fields [OK]

Common Mistakes:

• Confusing JSON or XML syntax with Protobuf
• Missing field numbers in Protobuf
• Using invalid keywords like 'schema'

3. Given this JSON Schema snippet for a Kafka message:

{
  "type": "object",
  "properties": {
    "userId": {"type": "string"},
    "age": {"type": "integer"}
  },
  "required": ["userId"]
}

Which message is valid according to this schema?

medium

A. {"userId": "abc123", "age": 30}

B. {"age": 30}

C. {"userId": 123, "age": 30}

D. {"userId": "abc123", "age": "30"}

Solution

1. Step 1: Check required fields and types

   The schema requires 'userId' as a string and 'age' as an integer (optional).

2. Step 2: Validate each option

   {"userId": "abc123", "age": 30} has 'userId' as string and 'age' as integer, so it is valid.

3. Final Answer:

   {"userId": "abc123", "age": 30} -> Option A

4. Quick Check:

   Required string userId and integer age = A [OK]

Hint: Required fields must be present with correct types [OK]

Common Mistakes:

• Missing required 'userId' field
• Using wrong data types for fields
• Confusing string and integer types

4. You have this Protobuf schema:

message Event {
  string id = 1;
  int32 count = 2;
}

Which of the following is CORRECT about using it with Kafka?

medium

A. Using 'int32' instead of 'integer' causes error

B. Field numbers must be unique and start from 1

C. Missing a root 'schema' keyword causes failure

D. Field names cannot be strings

Solution

1. Step 1: Check Protobuf field numbering rules

   Field numbers must be unique and start from 1, which is true here.

2. Step 2: Identify the correct statement

   Field numbers must be unique and start from 1 correctly describes the field numbering rule. Options B, C, D are incorrect statements.

3. Final Answer:

   Field numbers must be unique and start from 1 -> Option B

4. Quick Check:

   Protobuf field numbering rule = A [OK]

Hint: Field numbers must be unique and start at 1 [OK]

Common Mistakes:

• Thinking 'int32' is invalid
• Expecting a 'schema' keyword in Protobuf
• Confusing field names with types

5. You want to optimize Kafka message size and speed. Which schema choice is best and why?

hard

A. Use JSON Schema because it is human-readable and easy to debug

B. Use XML schema because it supports complex data types

C. Use Protobuf because it is compact and faster to serialize/deserialize

D. Use plain text messages without schema for maximum speed

Solution

1. Step 1: Compare JSON Schema and Protobuf

   JSON Schema is readable but larger; Protobuf is compact and faster.

2. Step 2: Choose based on optimization goal

   For smaller size and speed, Protobuf is better.

3. Final Answer:

   Use Protobuf because it is compact and faster to serialize/deserialize -> Option C

4. Quick Check:

   Compact and fast = D [OK]

Hint: Protobuf is compact and faster than JSON Schema [OK]

Common Mistakes:

• Choosing JSON Schema for speed optimization
• Ignoring schema benefits and using plain text
• Confusing XML with Protobuf advantages