Kafka Connect architecture - Step-by-Step Execution

Process Flow - Kafka Connect architecture

Start Kafka Connect

↓

Workers run Connectors

↓

Connectors manage Tasks

↓

Tasks move data between Kafka and External Systems

↓

Data flows into/out of Kafka topics

↓

Workers monitor and manage tasks

↓

Stop or Restart Connectors/Tasks

Kafka Connect runs workers that manage connectors, which split work into tasks moving data between Kafka and external systems.

Execution Sample

Kafka

Start Kafka Connect Worker
Load Connector Config
Create Tasks
Tasks move data
Monitor and manage tasks

This sequence shows how Kafka Connect starts, loads connectors, creates tasks, moves data, and manages tasks.

Process Table

Step	Action	Component	Result
1	Start Kafka Connect Worker	Worker	Worker process starts and is ready
2	Load Connector Config	Connector	Connector configuration loaded
3	Create Tasks	Tasks	Tasks created based on connector config
4	Tasks move data	Tasks	Data flows between Kafka and external system
5	Monitor and manage tasks	Worker	Worker monitors task health and status
6	Stop or Restart Connectors/Tasks	Worker	Worker stops or restarts connectors/tasks as needed

💡 Kafka Connect stops when worker process is terminated or connectors/tasks are stopped

Status Tracker

Component	Initial State	After Step 2	After Step 3	After Step 4	Final State
Worker	Not running	Running	Running	Running	Running or stopped
Connector	Not loaded	Loaded	Loaded	Loaded	Stopped or running
Tasks	None	None	Created	Running and moving data	Stopped or running

Key Moments - 3 Insights

Why does Kafka Connect split work into multiple tasks?

Splitting work into tasks allows parallel data movement and better scalability, as shown in execution_table step 3 where tasks are created from connector config.

What role does the worker play in managing connectors and tasks?

The worker runs connectors and tasks, monitors their health, and can restart or stop them, as seen in steps 1, 5, and 6 of the execution_table.

How does data flow between Kafka and external systems?

Tasks handle the actual data movement between Kafka topics and external systems, shown in step 4 where tasks move data.

Visual Quiz - 3 Questions

Look at the execution_table, at which step are tasks created?

A Step 3

B Step 4

C Step 2

D Step 5

💡 Hint

Check the 'Action' column in execution_table row with Step 3

According to variable_tracker, what is the state of tasks after Step 4?

A Not created

B Created but idle

C Running and moving data

D Stopped

💡 Hint

Look at the 'Tasks' row under 'After Step 4' in variable_tracker

If the worker stops, what happens to connectors and tasks according to the flow?

A Connectors stop but tasks continue

B Both connectors and tasks stop

C Connectors and tasks continue running

D Only tasks stop

💡 Hint

Refer to exit_note and final states in variable_tracker

Concept Snapshot

Kafka Connect architecture:
- Worker runs connectors and tasks
- Connectors define data source/sink configs
- Tasks do actual data movement
- Worker monitors and manages tasks
- Data flows between Kafka topics and external systems

Full Transcript

Kafka Connect architecture involves a worker process that runs connectors. Connectors load configuration and create tasks. Tasks move data between Kafka topics and external systems. The worker monitors tasks and can stop or restart them. This flow ensures scalable and reliable data integration.

Practice

1. What is the main role of a Kafka Connect worker in the Kafka Connect architecture?

easy

A. To run connectors and tasks that move data between Kafka and external systems

B. To store Kafka topic data permanently

C. To produce messages to Kafka topics

D. To manage Kafka broker configurations

Solution

1. Step 1: Understand Kafka Connect components

   Kafka Connect architecture includes workers, connectors, and tasks. Workers run connectors and tasks.

2. Step 2: Identify worker's role

   Workers execute connectors and tasks to move data between Kafka and external systems.

3. Final Answer:

   To run connectors and tasks that move data between Kafka and external systems -> Option A

4. Quick Check:

   Worker = runs connectors/tasks [OK]

Hint: Workers run connectors and tasks to move data [OK]

Common Mistakes:

• Confusing workers with Kafka brokers
• Thinking workers store data permanently
• Mixing up producers and workers

2. Which of the following is the correct way to start Kafka Connect in standalone mode using the command line?

easy

A. connect-distributed.sh worker.properties connector.properties

B. connect-standalone.sh worker.properties connector.properties

C. kafka-server-start.sh connect-standalone.properties

D. kafka-connect-start.sh standalone.properties

Solution

1. Step 1: Recall Kafka Connect startup commands

   Standalone mode uses connect-standalone.sh with worker and connector config files.

2. Step 2: Match correct command syntax

   connect-standalone.sh worker.properties connector.properties matches the correct command format: connect-standalone.sh worker.properties connector.properties

3. Final Answer:

   connect-standalone.sh worker.properties connector.properties -> Option B

4. Quick Check:

   Standalone mode uses connect-standalone.sh [OK]

Hint: Standalone mode uses connect-standalone.sh command [OK]

Common Mistakes:

• Using connect-distributed.sh for standalone mode
• Confusing Kafka broker start scripts with Connect scripts
• Wrong file names or missing config files

3. Given the following Kafka Connect distributed mode configuration snippet:

{
  "name": "my-source-connector",
  "config": {
    "connector.class": "FileStreamSource",
    "tasks.max": "2",
    "file": "/tmp/input.txt",
    "topic": "test-topic"
  }
}

What does the tasks.max setting control?

medium

A. The maximum number of tasks to run for this connector

B. The maximum number of Kafka brokers to connect

C. The maximum size of each message sent

D. The maximum number of topics to read from

Solution

1. Step 1: Understand the tasks.max parameter

   In Kafka Connect, tasks.max sets how many tasks a connector can run in parallel.

2. Step 2: Apply to given config

   Here, tasks.max is 2, so the connector can run up to 2 tasks to increase throughput.

3. Final Answer:

   The maximum number of tasks to run for this connector -> Option A

4. Quick Check:

   tasks.max = max tasks per connector [OK]

Hint: tasks.max limits parallel tasks for connector [OK]

Common Mistakes:

• Confusing tasks.max with broker count
• Thinking it limits message size
• Assuming it controls topic count

4. You have a distributed Kafka Connect cluster but notice that connectors are not starting tasks as expected. Which of the following is the most likely cause?

medium

A. The connector configuration file is missing the tasks.max property

B. Kafka brokers are offline

C. The worker.properties file is empty

D. The connector class name is misspelled in the configuration

Solution

1. Step 1: Analyze why tasks don't start

   If connectors don't start tasks, a common cause is an invalid connector class name.

2. Step 2: Check other options

   Missing tasks.max defaults to 1 task, so not likely cause. Brokers offline cause other errors. Empty worker.properties would prevent worker start. Misspelled connector class prevents task creation.

3. Final Answer:

   The connector class name is misspelled in the configuration -> Option D

4. Quick Check:

   Misspelled connector class blocks tasks [OK]

Hint: Check connector class spelling if tasks don't start [OK]

Common Mistakes:

• Assuming missing tasks.max stops tasks
• Ignoring connector class name correctness
• Not verifying worker config completeness

5. You want to build a fault-tolerant data pipeline using Kafka Connect that can scale horizontally and recover from worker failures. Which Kafka Connect mode should you choose and why?

hard

A. Standalone mode, because it is simpler and easier to configure

B. Distributed mode, because it requires no configuration files

C. Distributed mode, because it supports multiple workers and fault tolerance

D. Standalone mode, because it automatically balances tasks across workers

Solution

1. Step 1: Understand Kafka Connect modes

   Standalone mode runs a single worker, no fault tolerance or scaling. Distributed mode runs multiple workers, supports scaling and fault tolerance.

2. Step 2: Match requirements to mode

   For fault tolerance and horizontal scaling, distributed mode is required.

3. Final Answer:

   Distributed mode, because it supports multiple workers and fault tolerance -> Option C

4. Quick Check:

   Fault tolerance needs distributed mode [OK]

Hint: Use distributed mode for scaling and fault tolerance [OK]

Common Mistakes:

• Choosing standalone for fault tolerance
• Thinking standalone balances tasks
• Believing distributed mode needs no config