Sink connectors in Kafka - Commands & Configuration

Introduction

Sink connectors move data from Kafka topics to external systems like databases or storage. They help automate data export without writing custom code.

When you want to save Kafka topic data into a database automatically.

When you need to export streaming data to a file system for backups.

When integrating Kafka with analytics tools that require data in their own storage.

When you want to keep external systems updated in real-time from Kafka.

When you want to avoid manual data transfer from Kafka to other systems.

Config File - sink-connector.json

sink-connector.json

{
  "name": "example-sink-connector",
  "config": {
    "connector.class": "FileStreamSinkConnector",
    "tasks.max": "1",
    "topics": "example-topic",
    "file": "/tmp/output.txt",
    "key.converter": "org.apache.kafka.connect.storage.StringConverter",
    "value.converter": "org.apache.kafka.connect.storage.StringConverter"
  }
}

This JSON configures a sink connector named example-sink-connector.

connector.class specifies the type of sink connector; here it writes to a file.

tasks.max sets how many parallel tasks run.

topics lists Kafka topics to read from.

file is the output file path where data will be saved.

key.converter and value.converter define how Kafka data is converted to strings.

Commands

This command creates the sink connector by sending the JSON config to Kafka Connect's REST API.

Terminal

curl -X POST -H "Content-Type: application/json" --data @sink-connector.json http://localhost:8083/connectors

Expected Output

{"name":"example-sink-connector","config":{...},"tasks":[],"type":"sink"}

→

-X POST - Specifies the HTTP method to create the connector.

→

-H "Content-Type: application/json" - Sets the content type to JSON for the request.

→

--data @sink-connector.json - Sends the connector configuration file as the request body.

This command checks the status of the sink connector to verify it is running.

Terminal

curl http://localhost:8083/connectors/example-sink-connector/status

Expected Output

{"name":"example-sink-connector","connector":{"state":"RUNNING","worker_id":"worker-1"},"tasks":[{"id":0,"state":"RUNNING","worker_id":"worker-1"}]}

This command shows the live output file where the sink connector writes data from the Kafka topic.

Terminal

tail -f /tmp/output.txt

Expected Output

example message 1

example message 2

example message 3

→

-f - Follows the file to show new data as it arrives.

Key Concept

Sink connectors automatically export Kafka topic data to external systems without manual coding.

Common Mistakes

Not specifying the correct topic name in the connector config.

The connector won't receive any data if the topic name is wrong or missing.

Double-check the topic name matches exactly the Kafka topic you want to export.

Using incompatible converters for key or value data.

Data may not be properly serialized or deserialized, causing errors or empty output.

Use converters that match your Kafka data format, like StringConverter for plain text.

Not checking connector status after creation.

You might think the connector is running when it has failed or is paused.

Always check the connector status via the REST API to confirm it is RUNNING.

Summary

Create a sink connector by posting a JSON config to Kafka Connect REST API.

Verify the connector is running by checking its status endpoint.

View the exported data in the external system, like a file, to confirm data flow.

Practice

1.

What is the main purpose of a Kafka sink connector?

easy

A. To create new Kafka topics dynamically

B. To move data from external systems into Kafka topics

C. To monitor Kafka cluster health

D. To move data from Kafka topics to external systems automatically

Solution

1. Step 1: Understand the role of sink connectors

   Sink connectors are designed to export data from Kafka topics to other systems.

2. Step 2: Differentiate from source connectors

   Source connectors import data into Kafka, while sink connectors export data out.

3. Final Answer:

   To move data from Kafka topics to external systems automatically -> Option D

4. Quick Check:

   Sink connector = export data from Kafka [OK]

Hint: Sink connectors export data out of Kafka topics [OK]

Common Mistakes:

• Confusing sink with source connectors
• Thinking sink connectors create topics
• Assuming sink connectors monitor Kafka health

2.

Which of the following is a correct JSON field to specify the Kafka topic for a sink connector configuration?

{
  "name": "my-sink-connector",
  "config": {
    "connector.class": "io.confluent.connect.jdbc.JdbcSinkConnector",
    "topics": "my_topic",
    "connection.url": "jdbc:mysql://localhost:3306/mydb"
  }
}

easy

A. "topics"

B. "topic"

C. "topic_name"

D. "source.topics"

Solution

1. Step 1: Check the standard Kafka Connect config field

   The field to specify Kafka topics in sink connector config is "topics" (plural).

2. Step 2: Verify the example JSON

   The example uses "topics": "my_topic", which is correct syntax.

3. Final Answer:

   "topics" -> Option A

4. Quick Check:

   Kafka sink config uses "topics" field [OK]

Hint: Use "topics" (plural) to list Kafka topics in config [OK]

Common Mistakes:

• Using "topic" (singular) instead of "topics"
• Using non-standard fields like "topic_name"
• Confusing source connector fields with sink

3.

Given this sink connector config snippet, what will be the destination system?

{
  "name": "file-sink-connector",
  "config": {
    "connector.class": "FileStreamSink",
    "topics": "logs",
    "file": "/tmp/output.log"
  }
}

medium

A. A Kafka topic named 'logs'

B. A file located at /tmp/output.log

C. A database table named 'logs'

D. A remote HTTP endpoint

Solution

1. Step 1: Identify the connector class

   The connector class is "FileStreamSink", which writes data to a file.

2. Step 2: Check the "file" config field

   The "file" field specifies the destination file path "/tmp/output.log".

3. Final Answer:

   A file located at /tmp/output.log -> Option B

4. Quick Check:

   FileStreamSink writes to file path [OK]

Hint: FileStreamSink writes Kafka data to a file path [OK]

Common Mistakes:

• Assuming data goes back to Kafka topic
• Confusing sink with source connectors
• Thinking it writes to a database or HTTP

4.

Identify the error in this sink connector config JSON:

{
  "name": "jdbc-sink",
  "config": {
    "connector.class": "io.confluent.connect.jdbc.JdbcSinkConnector",
    "topic": "orders",
    "connection.url": "jdbc:postgresql://localhost:5432/shop"
  }
}

medium

A. Invalid JDBC URL format

B. Connector class name is incorrect

C. Missing 'topics' field; uses 'topic' instead

D. Missing 'name' field

Solution

1. Step 1: Check the topic field name

   The config uses "topic" (singular), but Kafka sink connectors require "topics" (plural).

2. Step 2: Verify other fields

   The JDBC URL and connector class are correctly formatted, and "name" is present.

3. Final Answer:

   Missing 'topics' field; uses 'topic' instead -> Option C

4. Quick Check:

   Use "topics" not "topic" in sink config [OK]

Hint: Use plural "topics" field in sink connector config [OK]

Common Mistakes:

• Using singular "topic" instead of "topics"
• Assuming JDBC URL is wrong without checking
• Thinking connector class name is invalid

5.

You want to export data from multiple Kafka topics to a single database table using a JDBC sink connector. Which configuration approach is correct?

{
  "name": "multi-topic-jdbc-sink",
  "config": {
    "connector.class": "io.confluent.connect.jdbc.JdbcSinkConnector",
    "topics": "topic1,topic2,topic3",
    "connection.url": "jdbc:mysql://localhost:3306/mydb",
    "table.name.format": "combined_table"
  }
}

hard

A. This config exports all topics to one table named 'combined_table'

B. Each topic will create a separate table ignoring 'table.name.format'

C. The connector will fail because multiple topics are not allowed

D. Data will be merged but only from 'topic1'

Solution

1. Step 1: Understand 'topics' field with multiple topics

   The "topics" field lists multiple topics separated by commas, which is valid.

2. Step 2: Check 'table.name.format' usage

   Setting "table.name.format" to a fixed table name causes all data to go into that single table.

3. Step 3: Confirm connector behavior

   The JDBC sink connector supports writing multiple topics to one table if configured this way.

4. Final Answer:

   This config exports all topics to one table named 'combined_table' -> Option A

5. Quick Check:

   Multiple topics + fixed table name = single table export [OK]

Hint: Use comma-separated topics and fixed table name for single table export [OK]

Common Mistakes:

• Assuming multiple topics create separate tables automatically
• Thinking multiple topics are not allowed in sink config
• Ignoring 'table.name.format' effect