How to Use Kafka Connect: Simple Guide and Example

Use Kafka Connect by creating connector configurations in JSON or properties files and running them with the connect-standalone or connect-distributed command. Connectors move data between Kafka topics and external systems without coding, using source connectors to import data and sink connectors to export data.

Syntax

The basic syntax to run Kafka Connect is:

• connect-standalone <worker-config> <connector-config> for single-node mode.
• connect-distributed <worker-config> for cluster mode.

Here, worker-config sets up the Kafka Connect worker properties like Kafka brokers and serialization, and connector-config defines the connector details such as connector class, topics, and connection info.

bash

connect-standalone worker.properties connector.properties

Example

This example shows how to run a simple file source connector that reads lines from a file and writes them to a Kafka topic.

properties

# worker.properties
bootstrap.servers=localhost:9092
key.converter=org.apache.kafka.connect.storage.StringConverter
value.converter=org.apache.kafka.connect.storage.StringConverter

# connector.properties
name=local-file-source
connector.class=FileStreamSourceConnector
tasks.max=1
file=/tmp/input.txt
topic=connect-test

# Run command
connect-standalone worker.properties connector.properties

Output

INFO Kafka Connect started
INFO Source task started reading /tmp/input.txt
INFO Records sent to topic connect-test

Common Pitfalls

Common mistakes include:

• Not setting the correct bootstrap.servers in worker config, causing connection failures.
• Misconfiguring the connector class name or missing required properties.
• Running connect-standalone without proper file paths or permissions.
• For distributed mode, forgetting to start multiple workers or not configuring the internal Kafka topics.

Always check logs for errors and validate your JSON or properties syntax.

properties

## Wrong connector class example
name=bad-connector
connector.class=NonExistentConnector

## Correct connector class example
name=file-source
connector.class=FileStreamSourceConnector

Quick Reference

Command/Property	Description
connect-standalone	Run Kafka Connect in single-node mode
connect-distributed	Run Kafka Connect in cluster mode
bootstrap.servers	Kafka broker addresses
connector.class	Connector plugin class name
tasks.max	Maximum parallel tasks for connector
file	File path for FileStreamSource connector
topic	Kafka topic to read from or write to

Key Takeaways

Kafka Connect runs connectors using worker and connector configuration files.

Use source connectors to import data into Kafka and sink connectors to export data.

Start with connect-standalone for simple setups and connect-distributed for production clusters.

Check connector class names and required properties carefully to avoid errors.

Monitor logs to troubleshoot connection and configuration issues.