Sink connectors in Kafka - Step-by-Step Execution

Process Flow - Sink connectors

Kafka Topic receives messages

↓

Sink Connector polls messages

↓

Transforms/Processes messages

↓

Writes messages to external system

↓

Acknowledges success or retries on failure

Sink connectors read messages from Kafka topics and write them to external systems, handling processing and delivery.

Execution Sample

Kafka

name=jdbc-sink
connector.class=io.confluent.connect.jdbc.JdbcSinkConnector
tasks.max=1
topics=my_topic
connection.url=jdbc:postgresql://localhost:5432/mydb
insert.mode=insert

This config sets up a JDBC sink connector to write Kafka topic 'my_topic' data into a PostgreSQL database.

Process Table

Step	Action	Input	Output	Notes
1	Kafka topic receives message	{"id":1,"name":"Alice"}	Message stored in topic	Message produced by producer
2	Sink connector polls topic	Message available in topic	Message fetched by connector	Connector reads new message
3	Process message	{"id":1,"name":"Alice"}	Transformed record	May apply schema or conversion
4	Write to external system	Transformed record	Record inserted in DB	Writes to PostgreSQL table
5	Acknowledge success	Record inserted	Commit offset	Connector commits offset to avoid reprocessing
6	Poll next message	Waiting for new messages	Idle or next fetch	Connector waits or fetches next batch
7	No more messages	Topic empty	Stop polling	Connector pauses until new data arrives

💡 No more messages in topic, connector waits for new data

Status Tracker

Variable	Start	After Step 2	After Step 3	After Step 4	After Step 5	Final
message	null	{"id":1,"name":"Alice"}	Transformed record	Written to DB	Committed offset	null (ready for next)
offset	0	0	0	0	1	1

Key Moments - 3 Insights

Why does the connector commit the offset after writing to the external system?

Committing the offset after writing ensures messages are not lost or duplicated. See execution_table step 5 where offset commit happens only after successful write.

What happens if the external system is down when the connector tries to write?

The connector retries writing and does not commit the offset until successful, preventing data loss. This is implied between steps 4 and 5 in the execution_table.

Does the sink connector modify the original Kafka message?

The connector may transform the message format but does not change the original Kafka message in the topic. See step 3 where processing happens separately.

Visual Quiz - 3 Questions

Look at the execution_table, what is the connector doing at step 3?

A Polling the Kafka topic for new messages

B Processing or transforming the fetched message

C Writing the message to the external system

D Committing the offset after successful write

💡 Hint

Refer to execution_table row 3 under 'Action' and 'Output'

At which step does the connector commit the offset to avoid reprocessing?

A Step 5

B Step 4

C Step 2

D Step 6

💡 Hint

Check execution_table step 5 notes about offset commit

If the external system is unavailable, what will the connector do according to the flow?

A Skip the message and commit offset

B Delete the message from Kafka topic

C Retry writing and not commit offset

D Stop polling new messages immediately

💡 Hint

See key_moments explanation about retries and offset commit

Concept Snapshot

Sink connectors read messages from Kafka topics.
They transform and write data to external systems.
Offset commits happen only after successful writes.
Retries occur if external systems fail.
This ensures no data loss or duplication.

Full Transcript

Sink connectors in Kafka take messages from topics and send them to external systems like databases. The connector polls the topic, processes messages, writes them out, and commits offsets only after successful delivery. This prevents losing or duplicating data. If the external system is down, the connector retries writing and waits before committing. This flow ensures reliable data transfer from Kafka to other systems.