Kafka Connect architecture - Step-by-Step Execution

Process Flow - Kafka Connect architecture

Start Kafka Connect

↓

Workers run Connectors

↓

Connectors manage Tasks

↓

Tasks move data between Kafka and External Systems

↓

Data flows into/out of Kafka topics

↓

Workers monitor and manage tasks

↓

Stop or Restart Connectors/Tasks

Kafka Connect runs workers that manage connectors, which split work into tasks moving data between Kafka and external systems.

Execution Sample

Kafka

Start Kafka Connect Worker
Load Connector Config
Create Tasks
Tasks move data
Monitor and manage tasks

This sequence shows how Kafka Connect starts, loads connectors, creates tasks, moves data, and manages tasks.

Process Table

Step	Action	Component	Result
1	Start Kafka Connect Worker	Worker	Worker process starts and is ready
2	Load Connector Config	Connector	Connector configuration loaded
3	Create Tasks	Tasks	Tasks created based on connector config
4	Tasks move data	Tasks	Data flows between Kafka and external system
5	Monitor and manage tasks	Worker	Worker monitors task health and status
6	Stop or Restart Connectors/Tasks	Worker	Worker stops or restarts connectors/tasks as needed

💡 Kafka Connect stops when worker process is terminated or connectors/tasks are stopped

Status Tracker

Component	Initial State	After Step 2	After Step 3	After Step 4	Final State
Worker	Not running	Running	Running	Running	Running or stopped
Connector	Not loaded	Loaded	Loaded	Loaded	Stopped or running
Tasks	None	None	Created	Running and moving data	Stopped or running

Key Moments - 3 Insights

Why does Kafka Connect split work into multiple tasks?

Splitting work into tasks allows parallel data movement and better scalability, as shown in execution_table step 3 where tasks are created from connector config.

What role does the worker play in managing connectors and tasks?

The worker runs connectors and tasks, monitors their health, and can restart or stop them, as seen in steps 1, 5, and 6 of the execution_table.

How does data flow between Kafka and external systems?

Tasks handle the actual data movement between Kafka topics and external systems, shown in step 4 where tasks move data.

Visual Quiz - 3 Questions

Look at the execution_table, at which step are tasks created?

A Step 3

B Step 4

C Step 2

D Step 5

💡 Hint

Check the 'Action' column in execution_table row with Step 3

According to variable_tracker, what is the state of tasks after Step 4?

A Not created

B Created but idle

C Running and moving data

D Stopped

💡 Hint

Look at the 'Tasks' row under 'After Step 4' in variable_tracker

If the worker stops, what happens to connectors and tasks according to the flow?

A Connectors stop but tasks continue

B Both connectors and tasks stop

C Connectors and tasks continue running

D Only tasks stop

💡 Hint

Refer to exit_note and final states in variable_tracker

Concept Snapshot

Kafka Connect architecture:
- Worker runs connectors and tasks
- Connectors define data source/sink configs
- Tasks do actual data movement
- Worker monitors and manages tasks
- Data flows between Kafka topics and external systems

Full Transcript

Kafka Connect architecture involves a worker process that runs connectors. Connectors load configuration and create tasks. Tasks move data between Kafka topics and external systems. The worker monitors tasks and can stop or restart them. This flow ensures scalable and reliable data integration.