Schema validation in producers in Kafka - Step-by-Step Execution

Process Flow - Schema validation in producers

Start Producer

↓

Prepare Message

↓

Validate Message Against Schema

↓

Send

↓

Message Sent

↓

End Producer

The producer prepares a message, checks it against the schema, sends it if valid, or rejects it if invalid.

Execution Sample

Kafka

producer.send({"name": "Alice", "age": 30})
# Validate message schema
# If valid, send
# Else, raise error

This code sends a message after checking it matches the schema.

Process Table

Step	Action	Message Content	Schema Valid?	Result
1	Prepare message	{"name": "Alice", "age": 30}	N/A	Message ready
2	Validate message	{"name": "Alice", "age": 30}	Yes	Validation passed
3	Send message	{"name": "Alice", "age": 30}	Yes	Message sent
4	Prepare message	{"name": "Bob", "age": "thirty"}	N/A	Message ready
5	Validate message	{"name": "Bob", "age": "thirty"}	No	Validation failed - reject message
6	Stop sending	{"name": "Bob", "age": "thirty"}	No	Error raised, message not sent

💡 Producer stops sending when message fails schema validation.

Status Tracker

Variable	Start	After Step 1	After Step 2	After Step 3	After Step 4	After Step 5	After Step 6
message	null	{"name": "Alice", "age": 30}	{"name": "Alice", "age": 30}	sent	{"name": "Bob", "age": "thirty"}	{"name": "Bob", "age": "thirty"}	error
validation_status	null	pending	valid	sent	pending	invalid	error

Key Moments - 3 Insights

Why does the producer reject the message with age as a string?

Because the schema expects 'age' to be a number, not a string. This is shown in execution_table rows 4 and 5 where validation fails.

What happens if the message passes schema validation?

The message is sent to Kafka as shown in execution_table row 3, where validation_status is 'valid' and result is 'Message sent'.

Does the producer send any message after validation fails?

No, as shown in execution_table row 6, the producer raises an error and stops sending the invalid message.

Visual Quiz - 3 Questions

Look at the execution table, what is the validation status after step 2?

A invalid

B pending

C valid

D error

💡 Hint

Check the 'validation_status' variable after step 2 in variable_tracker.

At which step does the producer reject the message due to schema mismatch?

A Step 5

B Step 3

C Step 2

D Step 6

💡 Hint

Look at execution_table rows where 'Schema Valid?' is 'No'.

If the message had age as 25 (number), how would the validation result change at step 5?

A No change, still invalid

B Validation would pass

C Validation would fail

D Producer would raise error

💡 Hint

Refer to execution_table rows 2 and 5 comparing valid and invalid age types.

Concept Snapshot

Schema validation in producers:
- Producer prepares message
- Checks message against schema
- If valid, sends message
- If invalid, rejects and raises error
- Prevents bad data from entering Kafka

Full Transcript

Schema validation in Kafka producers means the producer checks each message against a predefined schema before sending it. If the message matches the schema, it is sent to Kafka. If not, the producer rejects the message and raises an error. This prevents invalid data from entering the system. For example, if the schema expects 'age' as a number, sending a string will cause validation to fail and the message will not be sent. This process ensures data quality and consistency in Kafka topics.