Why connectors integrate external systems in Kafka - Quick Recap

Recall & Review

beginner

What is the main purpose of connectors in Kafka?

Connectors in Kafka are used to integrate external systems by moving data into and out of Kafka topics automatically.

beginner

How do connectors help with external system integration?

Connectors simplify data flow by automatically reading from or writing to external systems like databases, file systems, or cloud services without manual coding.

intermediate

Why is it beneficial to use connectors instead of custom code for integration?

Using connectors reduces errors, saves time, and provides a reusable, standardized way to connect Kafka with external systems.

beginner

Name two types of Kafka connectors and their roles.

Source connectors import data from external systems into Kafka. Sink connectors export data from Kafka to external systems.

beginner

What real-life situation is similar to using connectors in Kafka?

Connectors are like adapters that let different devices talk to each other easily, such as a phone charger that fits various plugs.

What is the role of a Kafka connector?

A To store data permanently inside Kafka

B To replace Kafka brokers

C To connect Kafka with external systems automatically

D To monitor Kafka cluster health

Connectors automate data movement between Kafka and external systems.

Which connector type imports data into Kafka?

A Source connector

B Broker connector

C Stream connector

D Sink connector

Source connectors bring data from external systems into Kafka.

Why use connectors instead of writing custom integration code?

A Connectors only work with databases

B Connectors are slower but more complex

C Custom code is always better

D Connectors reduce errors and save time

Connectors provide a reliable, reusable way to integrate systems efficiently.

Which of these is NOT a typical external system for Kafka connectors?

A Databases

B Kafka brokers

C File systems

D Cloud storage

Kafka brokers are part of Kafka itself, not external systems.

What does a sink connector do?

A Exports data from Kafka to external systems

B Imports data into Kafka

C Manages Kafka topics

D Monitors Kafka performance

Sink connectors send data from Kafka to outside systems.

Explain why connectors are important for integrating external systems with Kafka.

Think about how connectors help move data without writing extra code.

Describe the difference between source and sink connectors in Kafka.

Consider the direction of data flow for each connector type.

Practice

1. Why do Kafka connectors integrate external systems?

easy

A. To automatically move data between Kafka and other systems

B. To create new Kafka topics manually

C. To write custom Kafka broker code

D. To monitor Kafka cluster health

Solution

1. Step 1: Understand the role of connectors

   Connectors are designed to link Kafka with external systems to automate data movement.

2. Step 2: Identify the main purpose

   The main purpose is to move data automatically, not to create topics or monitor health.

3. Final Answer:

   To automatically move data between Kafka and other systems -> Option A

4. Quick Check:

   Connectors move data automatically = A [OK]

Hint: Connectors move data automatically between systems [OK]

Common Mistakes:

• Thinking connectors create Kafka topics
• Confusing connectors with monitoring tools
• Assuming connectors require manual data transfer

2. Which of the following is the correct way to specify a connector configuration file in Kafka Connect?

easy

A. connect-standalone.sh config/connect-standalone.properties config/my-connector.properties

B. kafka-topics.sh --create --config my-connector.properties

C. kafka-console-producer.sh --broker-list localhost:9092 --topic my-connector.properties

D. connect-distributed.sh --config my-connector.properties

Solution

1. Step 1: Identify the correct command for standalone mode

   The command connect-standalone.sh is used with two config files: worker and connector.

2. Step 2: Check the options

   connect-standalone.sh config/connect-standalone.properties config/my-connector.properties correctly shows the worker config and connector config files together.

3. Final Answer:

   connect-standalone.sh config/connect-standalone.properties config/my-connector.properties -> Option A

4. Quick Check:

   Standalone mode uses two config files = A [OK]

Hint: Standalone mode needs worker and connector config files [OK]

Common Mistakes:

• Using topic commands instead of connector commands
• Passing only one config file
• Confusing distributed and standalone commands

3. Given this connector configuration snippet:

{
  "name": "my-source-connector",
  "config": {
    "connector.class": "FileStreamSource",
    "tasks.max": "1",
    "file": "/tmp/input.txt",
    "topic": "test-topic"
  }
}

What will this connector do when started?

medium

A. Create a new Kafka topic named '/tmp/input.txt'

B. Read data from /tmp/input.txt and send it to Kafka topic 'test-topic'

C. Write data from Kafka topic 'test-topic' to /tmp/input.txt

D. Monitor the file /tmp/input.txt for changes but do not send data

Solution

1. Step 1: Identify connector type and direction

   The connector class is FileStreamSource, which reads from a file and sends data to Kafka.

2. Step 2: Understand configuration parameters

   The file parameter points to the input file, and topic specifies the Kafka topic to send data to.

3. Final Answer:

   Read data from /tmp/input.txt and send it to Kafka topic 'test-topic' -> Option B

4. Quick Check:

   FileStreamSource reads file to Kafka = C [OK]

Hint: FileStreamSource reads file and sends to Kafka topic [OK]

Common Mistakes:

• Confusing source with sink connectors
• Thinking it writes to the file instead of reading
• Assuming it creates topics automatically

4. You wrote this sink connector config:

{
  "name": "my-sink-connector",
  "config": {
    "connector.class": "FileStreamSink",
    "tasks.max": "1",
    "file": "/tmp/output.txt",
    "topic": "input-topic"
  }
}

But no data appears in /tmp/output.txt after running. What is the likely problem?

medium

A. The connector class name is incorrect; it should be FileStreamSource

B. The file path /tmp/output.txt is not writable by the connector

C. The topic name 'input-topic' does not exist or has no data

D. The tasks.max value must be greater than 1

Solution

1. Step 1: Check connector class correctness

   The class FileStreamSink is correct for writing Kafka data to a file.

2. Step 2: Verify topic existence and data flow

   If the topic input-topic does not exist or has no data, no output will be written.

3. Final Answer:

   The topic name 'input-topic' does not exist or has no data -> Option C

4. Quick Check:

   Sink needs data in topic to write = B [OK]

Hint: Check topic exists and has data for sink connectors [OK]

Common Mistakes:

• Assuming connector class is wrong when it is correct
• Ignoring topic data availability
• Thinking tasks.max affects data flow directly

5. You want to integrate a database with Kafka using connectors. Which approach best explains why connectors are used for this integration?

hard

A. Connectors only work with file systems, not databases

B. Connectors replace the database with Kafka topics

C. Connectors require manual polling of the database by the user

D. Connectors automate data transfer between the database and Kafka without custom code

Solution

1. Step 1: Understand connector purpose for databases

   Connectors automate moving data between Kafka and external systems like databases.

2. Step 2: Eliminate incorrect options

   Connectors do not replace databases, do not require manual polling, and support databases, not just files.

3. Final Answer:

   Connectors automate data transfer between the database and Kafka without custom code -> Option D

4. Quick Check:

   Connectors automate integration without manual code = D [OK]

Hint: Connectors automate data flow without manual coding [OK]

Common Mistakes:

• Thinking connectors replace databases
• Assuming manual polling is needed
• Believing connectors only work with files