What Is Sink Connector in Kafka: Explanation and Example

A sink connector in Kafka is a component that reads data from Kafka topics and sends it to an external system like a database or storage. It acts like a bridge that moves data out of Kafka to where it is needed for further use or analysis.

How It Works

Think of Kafka as a post office that holds messages (data) in boxes called topics. A sink connector is like a delivery person who picks up these messages from the post office and delivers them to a specific destination, such as a database or a file system.

The sink connector continuously listens to one or more Kafka topics. When new messages arrive, it reads them and writes them to the external system in the format that system expects. This process happens automatically and reliably, so data flows smoothly from Kafka to other tools without manual work.

Example

This example shows a simple Kafka sink connector configuration that sends data from a Kafka topic named my-topic to a file on the local system.

json

{
  "name": "file-sink-connector",
  "config": {
    "connector.class": "FileStreamSinkConnector",
    "tasks.max": "1",
    "file": "/tmp/output.txt",
    "topics": "my-topic"
  }
}

Output

When this connector runs, it writes all messages from the 'my-topic' Kafka topic into the file /tmp/output.txt line by line.

When to Use

Use a sink connector when you want to move data from Kafka to another system automatically. For example:

• Saving Kafka messages into a database for reporting or analysis.
• Exporting data to a search engine like Elasticsearch for fast searching.
• Writing logs or events to files for backup or auditing.
• Integrating Kafka with cloud storage services like Amazon S3.

Sink connectors help automate data pipelines, reduce manual coding, and ensure data consistency between Kafka and other systems.

Key Points

• A sink connector moves data from Kafka topics to external systems.
• It runs continuously to keep data flowing out of Kafka.
• Common destinations include databases, file systems, and cloud storage.
• Sink connectors simplify building data pipelines without custom code.

Key Takeaways

A sink connector exports data from Kafka topics to external systems automatically.

It helps build reliable data pipelines without manual intervention.

Common uses include saving data to databases, files, or cloud storage.

Sink connectors run continuously to keep data synchronized.

They reduce the need for custom code to move Kafka data.