Why connectors integrate external systems in Kafka - Why It Works

Introduction

Connectors help move data between Kafka and other systems automatically. They solve the problem of manually writing code to connect different software, making data flow easier and faster.

When you want to send data from a database to Kafka without writing custom code

When you need to export Kafka messages to a search engine like Elasticsearch

When you want to keep data synchronized between Kafka and cloud storage like Amazon S3

When you want to integrate Kafka with monitoring tools to track data flow

When you want to connect Kafka with legacy systems that do not support Kafka natively

Commands

Starts a Kafka Connect worker in standalone mode using the main config and a connector config to read data from a file and send it to Kafka.

Terminal

kafka-connect-standalone /usr/local/etc/kafka/connect-standalone.properties /usr/local/etc/kafka/connect-file-source.properties

Expected Output

[2024-06-01 12:00:00,000] INFO Kafka Connect started (org.apache.kafka.connect.cli.ConnectStandalone)

[2024-06-01 12:00:01,000] INFO FileStreamSourceConnector started (org.apache.kafka.connect.file.FileStreamSourceConnector)

[2024-06-01 12:00:02,000] INFO Task started (org.apache.kafka.connect.runtime.WorkerTask)

Checks the list of connectors currently running on the Kafka Connect REST API.

Terminal

curl -X GET http://localhost:8083/connectors

Expected Output

["file-source-connector"]

Creates a new JDBC Sink connector that writes data from Kafka topic 'my-topic' into a PostgreSQL database automatically.

Terminal

curl -X POST -H "Content-Type: application/json" --data '{"name": "jdbc-sink-connector", "config": {"connector.class": "io.confluent.connect.jdbc.JdbcSinkConnector", "tasks.max": "1", "topics": "my-topic", "connection.url": "jdbc:postgresql://localhost:5432/mydb", "connection.user": "user", "connection.password": "password", "auto.create": "true"}}' http://localhost:8083/connectors

Expected Output

{"name":"jdbc-sink-connector","config":{...}}

Checks the status of the JDBC Sink connector to ensure it is running properly.

Terminal

curl -X GET http://localhost:8083/connectors/jdbc-sink-connector/status

Expected Output

{"name":"jdbc-sink-connector","connector":{"state":"RUNNING","worker_id":"worker1"},"tasks":[{"id":0,"state":"RUNNING","worker_id":"worker1"}]}

Key Concept

Connectors automate data movement between Kafka and external systems without manual coding.

Common Mistakes

Trying to connect external systems to Kafka without using connectors

This requires writing custom code and is error-prone and slow to maintain

Use Kafka Connect connectors to handle integration automatically

Not verifying connector status after creation

The connector might fail to start or have configuration errors unnoticed

Always check connector status via the REST API to confirm it is running

Using incorrect connector configuration properties

Connectors will fail to connect or behave unexpectedly if configs are wrong

Use official connector documentation and examples to set correct properties

Summary

Connectors simplify integration by automating data flow between Kafka and other systems.

You start connectors using Kafka Connect with configuration files or REST API calls.

Always verify connector status to ensure data is moving correctly.

Practice

1. Why do Kafka connectors integrate external systems?

easy

A. To automatically move data between Kafka and other systems

B. To create new Kafka topics manually

C. To write custom Kafka broker code

D. To monitor Kafka cluster health

Solution

1. Step 1: Understand the role of connectors

   Connectors are designed to link Kafka with external systems to automate data movement.

2. Step 2: Identify the main purpose

   The main purpose is to move data automatically, not to create topics or monitor health.

3. Final Answer:

   To automatically move data between Kafka and other systems -> Option A

4. Quick Check:

   Connectors move data automatically = A [OK]

Hint: Connectors move data automatically between systems [OK]

Common Mistakes:

• Thinking connectors create Kafka topics
• Confusing connectors with monitoring tools
• Assuming connectors require manual data transfer

2. Which of the following is the correct way to specify a connector configuration file in Kafka Connect?

easy

A. connect-standalone.sh config/connect-standalone.properties config/my-connector.properties

B. kafka-topics.sh --create --config my-connector.properties

C. kafka-console-producer.sh --broker-list localhost:9092 --topic my-connector.properties

D. connect-distributed.sh --config my-connector.properties

Solution

1. Step 1: Identify the correct command for standalone mode

   The command connect-standalone.sh is used with two config files: worker and connector.

2. Step 2: Check the options

   connect-standalone.sh config/connect-standalone.properties config/my-connector.properties correctly shows the worker config and connector config files together.

3. Final Answer:

   connect-standalone.sh config/connect-standalone.properties config/my-connector.properties -> Option A

4. Quick Check:

   Standalone mode uses two config files = A [OK]

Hint: Standalone mode needs worker and connector config files [OK]

Common Mistakes:

• Using topic commands instead of connector commands
• Passing only one config file
• Confusing distributed and standalone commands

3. Given this connector configuration snippet:

{
  "name": "my-source-connector",
  "config": {
    "connector.class": "FileStreamSource",
    "tasks.max": "1",
    "file": "/tmp/input.txt",
    "topic": "test-topic"
  }
}

What will this connector do when started?

medium

A. Create a new Kafka topic named '/tmp/input.txt'

B. Read data from /tmp/input.txt and send it to Kafka topic 'test-topic'

C. Write data from Kafka topic 'test-topic' to /tmp/input.txt

D. Monitor the file /tmp/input.txt for changes but do not send data

Solution

1. Step 1: Identify connector type and direction

   The connector class is FileStreamSource, which reads from a file and sends data to Kafka.

2. Step 2: Understand configuration parameters

   The file parameter points to the input file, and topic specifies the Kafka topic to send data to.

3. Final Answer:

   Read data from /tmp/input.txt and send it to Kafka topic 'test-topic' -> Option B

4. Quick Check:

   FileStreamSource reads file to Kafka = C [OK]

Hint: FileStreamSource reads file and sends to Kafka topic [OK]

Common Mistakes:

• Confusing source with sink connectors
• Thinking it writes to the file instead of reading
• Assuming it creates topics automatically

4. You wrote this sink connector config:

{
  "name": "my-sink-connector",
  "config": {
    "connector.class": "FileStreamSink",
    "tasks.max": "1",
    "file": "/tmp/output.txt",
    "topic": "input-topic"
  }
}

But no data appears in /tmp/output.txt after running. What is the likely problem?

medium

A. The connector class name is incorrect; it should be FileStreamSource

B. The file path /tmp/output.txt is not writable by the connector

C. The topic name 'input-topic' does not exist or has no data

D. The tasks.max value must be greater than 1

Solution

1. Step 1: Check connector class correctness

   The class FileStreamSink is correct for writing Kafka data to a file.

2. Step 2: Verify topic existence and data flow

   If the topic input-topic does not exist or has no data, no output will be written.

3. Final Answer:

   The topic name 'input-topic' does not exist or has no data -> Option C

4. Quick Check:

   Sink needs data in topic to write = B [OK]

Hint: Check topic exists and has data for sink connectors [OK]

Common Mistakes:

• Assuming connector class is wrong when it is correct
• Ignoring topic data availability
• Thinking tasks.max affects data flow directly

5. You want to integrate a database with Kafka using connectors. Which approach best explains why connectors are used for this integration?

hard

A. Connectors only work with file systems, not databases

B. Connectors replace the database with Kafka topics

C. Connectors require manual polling of the database by the user

D. Connectors automate data transfer between the database and Kafka without custom code

Solution

1. Step 1: Understand connector purpose for databases

   Connectors automate moving data between Kafka and external systems like databases.

2. Step 2: Eliminate incorrect options

   Connectors do not replace databases, do not require manual polling, and support databases, not just files.

3. Final Answer:

   Connectors automate data transfer between the database and Kafka without custom code -> Option D

4. Quick Check:

   Connectors automate integration without manual code = D [OK]

Hint: Connectors automate data flow without manual coding [OK]

Common Mistakes:

• Thinking connectors replace databases
• Assuming manual polling is needed
• Believing connectors only work with files