Source connectors in Kafka - Step-by-Step Execution

Process Flow - Source connectors

Start Connector

↓

Connect to Source System

↓

Read Data from Source

↓

Convert Data to Kafka Format

↓

Send Data to Kafka Topic

↓

Repeat Reading Continuously

↓

Stop Connector

The source connector starts, connects to the source system, reads data, converts it, sends it to Kafka, and repeats until stopped.

Execution Sample

Kafka

name=source-connector
connector.class=FileStreamSourceConnector
tasks.max=1
file=/tmp/input.txt
topic=source-topic

This config sets up a source connector that reads from a file and sends data to a Kafka topic.

Process Table

Step	Action	Source System State	Connector State	Kafka Topic State
1	Start connector	File /tmp/input.txt exists	Connector initialized	Topic empty
2	Connect to source	File accessible	Connected to file	Topic empty
3	Read line 1	File has lines	Read first line	Topic empty
4	Convert data	Line 1 read	Data converted	Topic empty
5	Send to topic	Line 1 read	Data sent	Topic has 1 message
6	Read line 2	File has more lines	Read second line	Topic has 1 message
7	Convert data	Line 2 read	Data converted	Topic has 1 message
8	Send to topic	Line 2 read	Data sent	Topic has 2 messages
9	No more lines	End of file	Waiting for new data	Topic has 2 messages
10	Stop connector	File unchanged	Connector stopped	Topic has 2 messages

💡 Connector stops when manually stopped or on error.

Status Tracker

Variable	Start	After Step 3	After Step 5	After Step 8	Final
Connector State	Not started	Reading line 1	Sent line 1	Sent line 2	Stopped
Kafka Topic Messages	0	0	1	2	2
Source File Lines Read	0	1	1	2	2

Key Moments - 2 Insights

Why does the connector keep running after reading all current lines?

Because source connectors continuously poll the source for new data, as shown in step 9 where it waits for new data after reaching end of file.

What happens if the source file is not accessible at start?

The connector fails to connect and does not proceed, as indicated in step 2 where connection to source is required before reading.

Visual Quiz - 3 Questions

Look at the execution table, how many messages are in the Kafka topic after step 5?

A 1

B 0

C 2

D None

💡 Hint

Check the 'Kafka Topic State' column at step 5 in the execution table.

At which step does the connector read the second line from the source file?

A Step 3

B Step 6

C Step 8

D Step 9

💡 Hint

Look at the 'Action' column for reading lines in the execution table.

If the source file had no lines, what would the connector state be after step 3?

A Reading line 1

B Connector stopped

C Waiting for new data

D Data sent

💡 Hint

Refer to the connector behavior when no data is available in the execution table and variable tracker.

Concept Snapshot

Source connectors read data from external systems
Convert data to Kafka format
Send data continuously to Kafka topics
Keep running to capture new data
Configured via connector properties

Full Transcript

Source connectors in Kafka start by connecting to an external source system like a file or database. They read data from that source, convert it into Kafka's message format, and send it to a Kafka topic. This process repeats continuously to capture new data as it arrives. The connector runs until it is stopped manually or encounters an error. The example configuration shows a file source connector reading lines from a file and sending them to a Kafka topic. The execution table traces each step: starting, connecting, reading lines, converting, sending, waiting for new data, and stopping. Variables like connector state, messages in the topic, and lines read from the source change step by step. Key points include understanding that connectors keep running to capture new data and that they must connect successfully to the source before reading. The visual quiz tests understanding of message counts, reading steps, and connector states when no data is present.