Why connectors integrate external systems in Kafka - Why It Works

Introduction

Connectors help move data between Kafka and other systems automatically. They solve the problem of manually writing code to connect different software, making data flow easier and faster.

When you want to send data from a database to Kafka without writing custom code

When you need to export Kafka messages to a search engine like Elasticsearch

When you want to keep data synchronized between Kafka and cloud storage like Amazon S3

When you want to integrate Kafka with monitoring tools to track data flow

When you want to connect Kafka with legacy systems that do not support Kafka natively

Commands

Starts a Kafka Connect worker in standalone mode using the main config and a connector config to read data from a file and send it to Kafka.

Terminal

kafka-connect-standalone /usr/local/etc/kafka/connect-standalone.properties /usr/local/etc/kafka/connect-file-source.properties

Expected Output

[2024-06-01 12:00:00,000] INFO Kafka Connect started (org.apache.kafka.connect.cli.ConnectStandalone)

[2024-06-01 12:00:01,000] INFO FileStreamSourceConnector started (org.apache.kafka.connect.file.FileStreamSourceConnector)

[2024-06-01 12:00:02,000] INFO Task started (org.apache.kafka.connect.runtime.WorkerTask)

Checks the list of connectors currently running on the Kafka Connect REST API.

Terminal

curl -X GET http://localhost:8083/connectors

Expected Output

["file-source-connector"]

Creates a new JDBC Sink connector that writes data from Kafka topic 'my-topic' into a PostgreSQL database automatically.

Terminal

curl -X POST -H "Content-Type: application/json" --data '{"name": "jdbc-sink-connector", "config": {"connector.class": "io.confluent.connect.jdbc.JdbcSinkConnector", "tasks.max": "1", "topics": "my-topic", "connection.url": "jdbc:postgresql://localhost:5432/mydb", "connection.user": "user", "connection.password": "password", "auto.create": "true"}}' http://localhost:8083/connectors

Expected Output

{"name":"jdbc-sink-connector","config":{...}}

Checks the status of the JDBC Sink connector to ensure it is running properly.

Terminal

curl -X GET http://localhost:8083/connectors/jdbc-sink-connector/status

Expected Output

{"name":"jdbc-sink-connector","connector":{"state":"RUNNING","worker_id":"worker1"},"tasks":[{"id":0,"state":"RUNNING","worker_id":"worker1"}]}

Key Concept

Connectors automate data movement between Kafka and external systems without manual coding.

Common Mistakes

Trying to connect external systems to Kafka without using connectors

This requires writing custom code and is error-prone and slow to maintain

Use Kafka Connect connectors to handle integration automatically

Not verifying connector status after creation

The connector might fail to start or have configuration errors unnoticed

Always check connector status via the REST API to confirm it is running

Using incorrect connector configuration properties

Connectors will fail to connect or behave unexpectedly if configs are wrong

Use official connector documentation and examples to set correct properties

Summary

Connectors simplify integration by automating data flow between Kafka and other systems.

You start connectors using Kafka Connect with configuration files or REST API calls.

Always verify connector status to ensure data is moving correctly.