Kafka Connect architecture - Deep Dive

Overview - Kafka Connect architecture

What is it?

Kafka Connect is a tool that helps move data between Apache Kafka and other systems automatically. It uses connectors to read data from sources or write data to destinations without writing code. This makes it easier to integrate Kafka with databases, files, or other services. Kafka Connect runs as a separate service that manages these data flows reliably.

Why it matters

Without Kafka Connect, moving data in and out of Kafka would require custom code for each system, which is slow and error-prone. Kafka Connect solves this by providing reusable connectors and managing data transfer automatically. This saves time, reduces mistakes, and helps keep data pipelines running smoothly. It makes Kafka practical for real-world data integration tasks.

Where it fits

Before learning Kafka Connect, you should understand basic Kafka concepts like topics, producers, and consumers. After Kafka Connect, you can explore Kafka Streams for processing data or Kafka's schema registry for managing data formats. Kafka Connect fits in the data integration layer between Kafka and external systems.

Mental Model

Core Idea

Kafka Connect is a bridge that automatically moves data between Kafka and other systems using reusable connectors.

Think of it like...

Imagine a postal service that picks up letters from your home and delivers them to friends, and also collects letters from friends to bring back to you, all without you having to travel or sort mail yourself.

┌───────────────┐      ┌───────────────┐      ┌───────────────┐
│   Source      │─────▶│ Kafka Connect │─────▶│   Sink        │
│  System(s)    │      │   Service     │      │  System(s)    │
└───────────────┘      └───────────────┘      └───────────────┘
        ▲                      ▲                      ▲
        │                      │                      │
   Source Connectors      Connectors Manager     Sink Connectors
        │                      │                      │
        ▼                      ▼                      ▼
   Data ingestion        Connector tasks       Data delivery

Build-Up - 7 Steps

1

Foundation

Understanding Kafka Connect basics

Concept: Kafka Connect is a framework to move data between Kafka and other systems using connectors.

Kafka Connect runs as a separate process that manages connectors. Connectors are plugins that know how to read from or write to external systems. There are two types: source connectors (to bring data into Kafka) and sink connectors (to send data out). This setup avoids writing custom code for each integration.

Result

You get a running Kafka Connect service that can load connectors to move data automatically.

Understanding Kafka Connect as a separate service with connectors helps see how it simplifies data integration without coding.

2

Foundation

Connectors and tasks explained

Concept: Connectors define what data to move, and tasks do the actual work in parallel.

A connector configures the source or sink system and how to connect to Kafka topics. Kafka Connect splits the work into tasks, which run in parallel to increase throughput. For example, a source connector reading from a database might have multiple tasks reading different tables or partitions.

Result

Data moves efficiently and reliably between Kafka and external systems using multiple tasks.

Knowing the division between connectors (configuration) and tasks (work units) explains how Kafka Connect scales data movement.

3

Intermediate

Distributed vs standalone modes

🤔Before reading on: do you think Kafka Connect can run only on one machine or can it run on many machines together? Commit to your answer.

Concept: Kafka Connect can run in standalone mode for simple setups or distributed mode for scalable, fault-tolerant clusters.

Standalone mode runs Kafka Connect on a single machine, suitable for testing or small jobs. Distributed mode runs Kafka Connect on multiple machines forming a cluster. The cluster shares connector configurations and tasks, balancing work and handling failures automatically.

Result

Distributed mode provides high availability and scalability, while standalone is simpler but limited.

Understanding the two modes helps choose the right setup for your data integration needs and scale.

4

Intermediate

Connector configuration and management

🤔Before reading on: do you think connector configurations are stored locally or centrally in Kafka Connect? Commit to your answer.

Concept: Kafka Connect stores connector configurations centrally in Kafka topics for distributed coordination and easy management.

In distributed mode, connector configs are stored in Kafka’s internal topics. This allows all workers to share the same configs and coordinate tasks. You can add, update, or remove connectors via REST API, and the cluster applies changes automatically.

Result

Connector configurations are consistent and manageable across the cluster without manual syncing.

Knowing that configs live in Kafka itself explains how Kafka Connect achieves fault tolerance and easy updates.

5

Intermediate

Data serialization and converters

Concept: Kafka Connect uses converters to translate data between Kafka’s format and external systems’ formats.

Data in Kafka is stored as bytes. Connectors use converters to serialize and deserialize data. Common converters include JSON, Avro, and String. Using a schema registry with Avro helps manage data formats and compatibility.

Result

Data flows correctly between Kafka and external systems with proper format handling.

Understanding converters clarifies how Kafka Connect handles diverse data formats seamlessly.

6

Advanced

Fault tolerance and offset management

🤔Before reading on: do you think Kafka Connect tracks what data it has processed internally or externally? Commit to your answer.

Concept: Kafka Connect tracks offsets to know what data has been processed, enabling fault tolerance and exactly-once delivery.

For source connectors, Kafka Connect stores offsets (positions in source data) in Kafka topics. If a task fails or restarts, it resumes from the last offset. This prevents data loss or duplication. Sink connectors commit offsets after writing data to the target system.

Result

Data pipelines are reliable and can recover from failures without losing or repeating data.

Knowing how offsets are managed explains Kafka Connect’s strong reliability guarantees.

7

Expert

Internal architecture and worker coordination

🤔Before reading on: do you think Kafka Connect workers communicate directly or only via Kafka? Commit to your answer.

Concept: Kafka Connect workers coordinate via Kafka topics and REST APIs to manage connectors and tasks in a distributed cluster.

Workers use Kafka internal topics to share connector configs, task assignments, and status. They also expose REST APIs for management. When a worker joins or leaves, the cluster rebalances tasks automatically. This design avoids a single point of failure and enables dynamic scaling.

Result

Kafka Connect clusters are resilient, scalable, and self-managing without manual intervention.

Understanding worker coordination reveals how Kafka Connect achieves distributed fault tolerance and elasticity.

Under the Hood

Kafka Connect runs as one or more worker processes that manage connectors and tasks. Connectors define configurations for data sources or sinks. Workers coordinate through Kafka internal topics to share configs and task assignments. Tasks perform the actual data transfer, reading or writing records. Offsets track progress to ensure no data is lost or duplicated. The REST API allows dynamic management. This architecture enables scaling, fault tolerance, and easy integration without custom code.

Why designed this way?

Kafka Connect was designed to solve the problem of integrating Kafka with many external systems reliably and at scale. Using Kafka itself for coordination avoids extra infrastructure and leverages Kafka’s durability. Separating connectors and tasks allows parallelism and flexibility. The REST API enables easy automation and management. Alternatives like custom scripts or standalone tools lacked scalability and fault tolerance, so Kafka Connect’s architecture balances simplicity and robustness.

┌───────────────┐      ┌───────────────┐      ┌───────────────┐
│ External      │      │ Kafka Connect │      │ External      │
│ Systems       │◀────▶│ Workers       │◀────▶│ Systems       │
└───────────────┘      └───────────────┘      └───────────────┘
        ▲                      ▲                      ▲
        │                      │                      │
  Source Connectors       Connector Manager      Sink Connectors
        │                      │                      │
        ▼                      ▼                      ▼
   Tasks read/write     Kafka Internal Topics    Tasks read/write
        │                      │                      │
        ▼                      ▼                      ▼
   Offsets stored     REST API for management   Offsets committed

Myth Busters - 4 Common Misconceptions

Quick: Do you think Kafka Connect can only move data in one direction, either into or out of Kafka? Commit to yes or no.

Common Belief: Kafka Connect only moves data into Kafka from external systems.

Reality: Kafka Connect supports both source connectors (into Kafka) and sink connectors (out of Kafka).

Why it matters: Believing it only moves data in one direction limits understanding and prevents using sink connectors to export data.

Quick: Do you think Kafka Connect requires writing custom code for each integration? Commit to yes or no.

Common Belief: You must write custom code to connect Kafka with each external system.

Reality: Kafka Connect uses reusable connectors that require configuration, not coding, for most integrations.

Why it matters: Thinking custom code is needed wastes time and misses Kafka Connect’s main benefit of automation.

Quick: Do you think Kafka Connect workers communicate only through direct network calls? Commit to yes or no.

Common Belief: Workers coordinate by direct network communication only.

Reality: Workers coordinate primarily through Kafka internal topics and use REST APIs for management.

Why it matters: Misunderstanding coordination can lead to wrong assumptions about scaling and fault tolerance.

Quick: Do you think Kafka Connect guarantees exactly-once delivery by default for all connectors? Commit to yes or no.

Common Belief: Kafka Connect always guarantees exactly-once delivery for all connectors.

Reality: Exactly-once delivery depends on connector implementation and external system capabilities; Kafka Connect provides at-least-once guarantees by default.

Why it matters: Assuming exactly-once always can cause data duplication or loss if connector or system does not support it.

Expert Zone

1

Kafka Connect’s use of Kafka internal topics for config and offset storage means it inherits Kafka’s durability and replication guarantees, making the system highly reliable.

2

Task rebalancing in distributed mode is automatic and can cause brief pauses; understanding this helps optimize connector configurations to minimize disruption.

3

Converters and transformations can be chained in Kafka Connect to handle complex data format changes without external processing.

When NOT to use

Kafka Connect is not ideal for very low-latency or complex data transformations; in such cases, Kafka Streams or custom processing pipelines are better. Also, if the external system lacks a connector or has unusual APIs, custom integration code might be necessary.

Production Patterns

In production, teams use distributed mode with multiple workers for high availability. They monitor connector health via REST API and logs, use schema registry for data compatibility, and apply Single Message Transforms (SMTs) for lightweight data changes. Connectors are often deployed with infrastructure as code and integrated into CI/CD pipelines.

Connections

Message Queues

Kafka Connect builds on message queue concepts by automating data movement between queues and external systems.

Understanding message queues helps grasp how Kafka Connect manages data flow reliably and asynchronously.

ETL (Extract, Transform, Load)

Kafka Connect automates the Extract and Load parts of ETL pipelines, often integrating with transformation tools.

Knowing ETL workflows clarifies Kafka Connect’s role in data pipelines and how it fits with processing stages.

Supply Chain Logistics

Kafka Connect’s role in moving data between systems is like logistics moving goods between warehouses and stores efficiently.

Seeing Kafka Connect as logistics highlights the importance of coordination, reliability, and scaling in data integration.

Common Pitfalls

#1 Running Kafka Connect in standalone mode for production workloads needing high availability.

Wrong approach:

bin/connect-standalone.sh config/connect-standalone.properties config/my-connector.properties

Correct approach:

bin/connect-distributed.sh config/connect-distributed.properties

Root cause:

Misunderstanding standalone mode as suitable for production leads to lack of fault tolerance and scalability.

#2 Not configuring converters properly, causing data format errors.

Wrong approach:

"key.converter": "org.apache.kafka.connect.storage.StringConverter",
"value.converter": "org.apache.kafka.connect.storage.StringConverter"

Correct approach:

"key.converter": "org.apache.kafka.connect.json.JsonConverter",
"value.converter": "org.apache.kafka.connect.json.JsonConverter",
"key.converter.schemas.enable": false,
"value.converter.schemas.enable": false

Root cause:

Using default converters without matching data formats causes serialization failures.

#3 Ignoring offset management leading to data duplication after restarts.

Wrong approach:

Not setting or committing offsets in source connectors, or deleting offset topics.

Correct approach:

Allow Kafka Connect to manage offsets automatically and avoid deleting internal topics.

Root cause:

Lack of understanding offset tracking causes data loss or duplication on failures.

Key Takeaways

Kafka Connect is a powerful tool that automates moving data between Kafka and external systems using reusable connectors.

It runs as a service with workers that manage connectors and tasks, enabling scalable and fault-tolerant data pipelines.

Connector configurations and offsets are stored in Kafka topics, allowing distributed coordination and recovery.

Kafka Connect supports both source and sink connectors, handling data serialization with converters for smooth integration.

Choosing the right mode and understanding internal coordination are key to building reliable production data pipelines.

Practice

1. What is the main role of a Kafka Connect worker in the Kafka Connect architecture?

easy

A. To run connectors and tasks that move data between Kafka and external systems

B. To store Kafka topic data permanently

C. To produce messages to Kafka topics

D. To manage Kafka broker configurations

Solution

1. Step 1: Understand Kafka Connect components

   Kafka Connect architecture includes workers, connectors, and tasks. Workers run connectors and tasks.

2. Step 2: Identify worker's role

   Workers execute connectors and tasks to move data between Kafka and external systems.

3. Final Answer:

   To run connectors and tasks that move data between Kafka and external systems -> Option A

4. Quick Check:

   Worker = runs connectors/tasks [OK]

Hint: Workers run connectors and tasks to move data [OK]

Common Mistakes:

• Confusing workers with Kafka brokers
• Thinking workers store data permanently
• Mixing up producers and workers

2. Which of the following is the correct way to start Kafka Connect in standalone mode using the command line?

easy

A. connect-distributed.sh worker.properties connector.properties

B. connect-standalone.sh worker.properties connector.properties

C. kafka-server-start.sh connect-standalone.properties

D. kafka-connect-start.sh standalone.properties

Solution

1. Step 1: Recall Kafka Connect startup commands

   Standalone mode uses connect-standalone.sh with worker and connector config files.

2. Step 2: Match correct command syntax

   connect-standalone.sh worker.properties connector.properties matches the correct command format: connect-standalone.sh worker.properties connector.properties

3. Final Answer:

   connect-standalone.sh worker.properties connector.properties -> Option B

4. Quick Check:

   Standalone mode uses connect-standalone.sh [OK]

Hint: Standalone mode uses connect-standalone.sh command [OK]

Common Mistakes:

• Using connect-distributed.sh for standalone mode
• Confusing Kafka broker start scripts with Connect scripts
• Wrong file names or missing config files

3. Given the following Kafka Connect distributed mode configuration snippet:

{
  "name": "my-source-connector",
  "config": {
    "connector.class": "FileStreamSource",
    "tasks.max": "2",
    "file": "/tmp/input.txt",
    "topic": "test-topic"
  }
}

What does the tasks.max setting control?

medium

A. The maximum number of tasks to run for this connector

B. The maximum number of Kafka brokers to connect

C. The maximum size of each message sent

D. The maximum number of topics to read from

Solution

1. Step 1: Understand the tasks.max parameter

   In Kafka Connect, tasks.max sets how many tasks a connector can run in parallel.

2. Step 2: Apply to given config

   Here, tasks.max is 2, so the connector can run up to 2 tasks to increase throughput.

3. Final Answer:

   The maximum number of tasks to run for this connector -> Option A

4. Quick Check:

   tasks.max = max tasks per connector [OK]

Hint: tasks.max limits parallel tasks for connector [OK]

Common Mistakes:

• Confusing tasks.max with broker count
• Thinking it limits message size
• Assuming it controls topic count

4. You have a distributed Kafka Connect cluster but notice that connectors are not starting tasks as expected. Which of the following is the most likely cause?

medium

A. The connector configuration file is missing the tasks.max property

B. Kafka brokers are offline

C. The worker.properties file is empty

D. The connector class name is misspelled in the configuration

Solution

1. Step 1: Analyze why tasks don't start

   If connectors don't start tasks, a common cause is an invalid connector class name.

2. Step 2: Check other options

   Missing tasks.max defaults to 1 task, so not likely cause. Brokers offline cause other errors. Empty worker.properties would prevent worker start. Misspelled connector class prevents task creation.

3. Final Answer:

   The connector class name is misspelled in the configuration -> Option D

4. Quick Check:

   Misspelled connector class blocks tasks [OK]

Hint: Check connector class spelling if tasks don't start [OK]

Common Mistakes:

• Assuming missing tasks.max stops tasks
• Ignoring connector class name correctness
• Not verifying worker config completeness

5. You want to build a fault-tolerant data pipeline using Kafka Connect that can scale horizontally and recover from worker failures. Which Kafka Connect mode should you choose and why?

hard

A. Standalone mode, because it is simpler and easier to configure

B. Distributed mode, because it requires no configuration files

C. Distributed mode, because it supports multiple workers and fault tolerance

D. Standalone mode, because it automatically balances tasks across workers

Solution

1. Step 1: Understand Kafka Connect modes

   Standalone mode runs a single worker, no fault tolerance or scaling. Distributed mode runs multiple workers, supports scaling and fault tolerance.

2. Step 2: Match requirements to mode

   For fault tolerance and horizontal scaling, distributed mode is required.

3. Final Answer:

   Distributed mode, because it supports multiple workers and fault tolerance -> Option C

4. Quick Check:

   Fault tolerance needs distributed mode [OK]

Hint: Use distributed mode for scaling and fault tolerance [OK]

Common Mistakes:

• Choosing standalone for fault tolerance
• Thinking standalone balances tasks
• Believing distributed mode needs no config