Kafka Connect architecture - Deep Dive

Overview - Kafka Connect architecture

What is it?

Kafka Connect is a tool that helps move data between Apache Kafka and other systems automatically. It uses connectors to read data from sources or write data to destinations without writing code. This makes it easier to integrate Kafka with databases, files, or other services. Kafka Connect runs as a separate service that manages these data flows reliably.

Why it matters

Without Kafka Connect, moving data in and out of Kafka would require custom code for each system, which is slow and error-prone. Kafka Connect solves this by providing reusable connectors and managing data transfer automatically. This saves time, reduces mistakes, and helps keep data pipelines running smoothly. It makes Kafka practical for real-world data integration tasks.

Where it fits

Before learning Kafka Connect, you should understand basic Kafka concepts like topics, producers, and consumers. After Kafka Connect, you can explore Kafka Streams for processing data or Kafka's schema registry for managing data formats. Kafka Connect fits in the data integration layer between Kafka and external systems.

Mental Model

Core Idea

Kafka Connect is a bridge that automatically moves data between Kafka and other systems using reusable connectors.

Think of it like...

Imagine a postal service that picks up letters from your home and delivers them to friends, and also collects letters from friends to bring back to you, all without you having to travel or sort mail yourself.

┌───────────────┐      ┌───────────────┐      ┌───────────────┐
│   Source      │─────▶│ Kafka Connect │─────▶│   Sink        │
│  System(s)    │      │   Service     │      │  System(s)    │
└───────────────┘      └───────────────┘      └───────────────┘
        ▲                      ▲                      ▲
        │                      │                      │
   Source Connectors      Connectors Manager     Sink Connectors
        │                      │                      │
        ▼                      ▼                      ▼
   Data ingestion        Connector tasks       Data delivery

Build-Up - 7 Steps

1

Foundation

Understanding Kafka Connect basics

Concept: Kafka Connect is a framework to move data between Kafka and other systems using connectors.

Kafka Connect runs as a separate process that manages connectors. Connectors are plugins that know how to read from or write to external systems. There are two types: source connectors (to bring data into Kafka) and sink connectors (to send data out). This setup avoids writing custom code for each integration.

Result

You get a running Kafka Connect service that can load connectors to move data automatically.

Understanding Kafka Connect as a separate service with connectors helps see how it simplifies data integration without coding.

2

Foundation

Connectors and tasks explained

Concept: Connectors define what data to move, and tasks do the actual work in parallel.

A connector configures the source or sink system and how to connect to Kafka topics. Kafka Connect splits the work into tasks, which run in parallel to increase throughput. For example, a source connector reading from a database might have multiple tasks reading different tables or partitions.

Result

Data moves efficiently and reliably between Kafka and external systems using multiple tasks.

Knowing the division between connectors (configuration) and tasks (work units) explains how Kafka Connect scales data movement.

3

Intermediate

Distributed vs standalone modes

🤔Before reading on: do you think Kafka Connect can run only on one machine or can it run on many machines together? Commit to your answer.

Concept: Kafka Connect can run in standalone mode for simple setups or distributed mode for scalable, fault-tolerant clusters.

Standalone mode runs Kafka Connect on a single machine, suitable for testing or small jobs. Distributed mode runs Kafka Connect on multiple machines forming a cluster. The cluster shares connector configurations and tasks, balancing work and handling failures automatically.

Result

Distributed mode provides high availability and scalability, while standalone is simpler but limited.

Understanding the two modes helps choose the right setup for your data integration needs and scale.

4

Intermediate

Connector configuration and management

🤔Before reading on: do you think connector configurations are stored locally or centrally in Kafka Connect? Commit to your answer.

Concept: Kafka Connect stores connector configurations centrally in Kafka topics for distributed coordination and easy management.

In distributed mode, connector configs are stored in Kafka’s internal topics. This allows all workers to share the same configs and coordinate tasks. You can add, update, or remove connectors via REST API, and the cluster applies changes automatically.

Result

Connector configurations are consistent and manageable across the cluster without manual syncing.

Knowing that configs live in Kafka itself explains how Kafka Connect achieves fault tolerance and easy updates.

5

Intermediate

Data serialization and converters

Concept: Kafka Connect uses converters to translate data between Kafka’s format and external systems’ formats.

Data in Kafka is stored as bytes. Connectors use converters to serialize and deserialize data. Common converters include JSON, Avro, and String. Using a schema registry with Avro helps manage data formats and compatibility.

Result

Data flows correctly between Kafka and external systems with proper format handling.

Understanding converters clarifies how Kafka Connect handles diverse data formats seamlessly.

6

Advanced

Fault tolerance and offset management

🤔Before reading on: do you think Kafka Connect tracks what data it has processed internally or externally? Commit to your answer.

Concept: Kafka Connect tracks offsets to know what data has been processed, enabling fault tolerance and exactly-once delivery.

For source connectors, Kafka Connect stores offsets (positions in source data) in Kafka topics. If a task fails or restarts, it resumes from the last offset. This prevents data loss or duplication. Sink connectors commit offsets after writing data to the target system.

Result

Data pipelines are reliable and can recover from failures without losing or repeating data.

Knowing how offsets are managed explains Kafka Connect’s strong reliability guarantees.

7

Expert

Internal architecture and worker coordination

🤔Before reading on: do you think Kafka Connect workers communicate directly or only via Kafka? Commit to your answer.

Concept: Kafka Connect workers coordinate via Kafka topics and REST APIs to manage connectors and tasks in a distributed cluster.

Workers use Kafka internal topics to share connector configs, task assignments, and status. They also expose REST APIs for management. When a worker joins or leaves, the cluster rebalances tasks automatically. This design avoids a single point of failure and enables dynamic scaling.

Result

Kafka Connect clusters are resilient, scalable, and self-managing without manual intervention.

Understanding worker coordination reveals how Kafka Connect achieves distributed fault tolerance and elasticity.

Under the Hood

Kafka Connect runs as one or more worker processes that manage connectors and tasks. Connectors define configurations for data sources or sinks. Workers coordinate through Kafka internal topics to share configs and task assignments. Tasks perform the actual data transfer, reading or writing records. Offsets track progress to ensure no data is lost or duplicated. The REST API allows dynamic management. This architecture enables scaling, fault tolerance, and easy integration without custom code.

Why designed this way?

Kafka Connect was designed to solve the problem of integrating Kafka with many external systems reliably and at scale. Using Kafka itself for coordination avoids extra infrastructure and leverages Kafka’s durability. Separating connectors and tasks allows parallelism and flexibility. The REST API enables easy automation and management. Alternatives like custom scripts or standalone tools lacked scalability and fault tolerance, so Kafka Connect’s architecture balances simplicity and robustness.

┌───────────────┐      ┌───────────────┐      ┌───────────────┐
│ External      │      │ Kafka Connect │      │ External      │
│ Systems       │◀────▶│ Workers       │◀────▶│ Systems       │
└───────────────┘      └───────────────┘      └───────────────┘
        ▲                      ▲                      ▲
        │                      │                      │
  Source Connectors       Connector Manager      Sink Connectors
        │                      │                      │
        ▼                      ▼                      ▼
   Tasks read/write     Kafka Internal Topics    Tasks read/write
        │                      │                      │
        ▼                      ▼                      ▼
   Offsets stored     REST API for management   Offsets committed

Myth Busters - 4 Common Misconceptions

Quick: Do you think Kafka Connect can only move data in one direction, either into or out of Kafka? Commit to yes or no.

Common Belief: Kafka Connect only moves data into Kafka from external systems.

Reality: Kafka Connect supports both source connectors (into Kafka) and sink connectors (out of Kafka).

Why it matters: Believing it only moves data in one direction limits understanding and prevents using sink connectors to export data.

Quick: Do you think Kafka Connect requires writing custom code for each integration? Commit to yes or no.

Common Belief: You must write custom code to connect Kafka with each external system.

Reality: Kafka Connect uses reusable connectors that require configuration, not coding, for most integrations.

Why it matters: Thinking custom code is needed wastes time and misses Kafka Connect’s main benefit of automation.

Quick: Do you think Kafka Connect workers communicate only through direct network calls? Commit to yes or no.

Common Belief: Workers coordinate by direct network communication only.

Reality: Workers coordinate primarily through Kafka internal topics and use REST APIs for management.

Why it matters: Misunderstanding coordination can lead to wrong assumptions about scaling and fault tolerance.

Quick: Do you think Kafka Connect guarantees exactly-once delivery by default for all connectors? Commit to yes or no.

Common Belief: Kafka Connect always guarantees exactly-once delivery for all connectors.

Reality: Exactly-once delivery depends on connector implementation and external system capabilities; Kafka Connect provides at-least-once guarantees by default.

Why it matters: Assuming exactly-once always can cause data duplication or loss if connector or system does not support it.

Expert Zone

1

Kafka Connect’s use of Kafka internal topics for config and offset storage means it inherits Kafka’s durability and replication guarantees, making the system highly reliable.

2

Task rebalancing in distributed mode is automatic and can cause brief pauses; understanding this helps optimize connector configurations to minimize disruption.

3

Converters and transformations can be chained in Kafka Connect to handle complex data format changes without external processing.

When NOT to use

Kafka Connect is not ideal for very low-latency or complex data transformations; in such cases, Kafka Streams or custom processing pipelines are better. Also, if the external system lacks a connector or has unusual APIs, custom integration code might be necessary.

Production Patterns

In production, teams use distributed mode with multiple workers for high availability. They monitor connector health via REST API and logs, use schema registry for data compatibility, and apply Single Message Transforms (SMTs) for lightweight data changes. Connectors are often deployed with infrastructure as code and integrated into CI/CD pipelines.

Connections

Message Queues

Kafka Connect builds on message queue concepts by automating data movement between queues and external systems.

Understanding message queues helps grasp how Kafka Connect manages data flow reliably and asynchronously.

ETL (Extract, Transform, Load)

Kafka Connect automates the Extract and Load parts of ETL pipelines, often integrating with transformation tools.

Knowing ETL workflows clarifies Kafka Connect’s role in data pipelines and how it fits with processing stages.

Supply Chain Logistics

Kafka Connect’s role in moving data between systems is like logistics moving goods between warehouses and stores efficiently.

Seeing Kafka Connect as logistics highlights the importance of coordination, reliability, and scaling in data integration.

Common Pitfalls

#1 Running Kafka Connect in standalone mode for production workloads needing high availability.

Wrong approach:

bin/connect-standalone.sh config/connect-standalone.properties config/my-connector.properties

Correct approach:

bin/connect-distributed.sh config/connect-distributed.properties

Root cause:

Misunderstanding standalone mode as suitable for production leads to lack of fault tolerance and scalability.

#2 Not configuring converters properly, causing data format errors.

Wrong approach:

"key.converter": "org.apache.kafka.connect.storage.StringConverter",
"value.converter": "org.apache.kafka.connect.storage.StringConverter"

Correct approach:

"key.converter": "org.apache.kafka.connect.json.JsonConverter",
"value.converter": "org.apache.kafka.connect.json.JsonConverter",
"key.converter.schemas.enable": false,
"value.converter.schemas.enable": false

Root cause:

Using default converters without matching data formats causes serialization failures.

#3 Ignoring offset management leading to data duplication after restarts.

Wrong approach:

Not setting or committing offsets in source connectors, or deleting offset topics.

Correct approach:

Allow Kafka Connect to manage offsets automatically and avoid deleting internal topics.

Root cause:

Lack of understanding offset tracking causes data loss or duplication on failures.

Key Takeaways

Kafka Connect is a powerful tool that automates moving data between Kafka and external systems using reusable connectors.

It runs as a service with workers that manage connectors and tasks, enabling scalable and fault-tolerant data pipelines.

Connector configurations and offsets are stored in Kafka topics, allowing distributed coordination and recovery.

Kafka Connect supports both source and sink connectors, handling data serialization with converters for smooth integration.

Choosing the right mode and understanding internal coordination are key to building reliable production data pipelines.