Why connectors integrate external systems in Kafka - Performance Analysis

Time Complexity: Why connectors integrate external systems

O(n)

Understanding Time Complexity

When connectors link Kafka to outside systems, we want to know how the work grows as data grows.

How does the time to move data change when more data or systems are involved?

Scenario Under Consideration

Analyze the time complexity of the following code snippet.

connector.poll() {
  records = externalSystem.fetchRecords()
  for (record in records) {
    kafkaTopic.send(record)
  }
}

This code fetches records from an external system and sends each one to a Kafka topic.

Identify Repeating Operations

Identify the loops, recursion, array traversals that repeat.

• Primary operation: Looping over each record fetched from the external system.
• How many times: Once for every record received in each poll call.

How Execution Grows With Input

As the number of records fetched grows, the time to process them grows too.

Input Size (n)	Approx. Operations
10	10 sends to Kafka
100	100 sends to Kafka
1000	1000 sends to Kafka

Pattern observation: The work grows directly with the number of records fetched.

Final Time Complexity

Time Complexity: O(n)

This means the time to process grows in a straight line with the number of records.

Common Mistake

[X] Wrong: "The connector time stays the same no matter how many records come in."

[OK] Correct: Each record needs to be handled, so more records mean more work and more time.

Interview Connect

Understanding how connectors scale with data helps you explain real-world system behavior clearly and confidently.

Self-Check

What if the connector batches records before sending? How would the time complexity change?

Practice

1. Why do Kafka connectors integrate external systems?

easy

A. To automatically move data between Kafka and other systems

B. To create new Kafka topics manually

C. To write custom Kafka broker code

D. To monitor Kafka cluster health

Solution

1. Step 1: Understand the role of connectors

   Connectors are designed to link Kafka with external systems to automate data movement.

2. Step 2: Identify the main purpose

   The main purpose is to move data automatically, not to create topics or monitor health.

3. Final Answer:

   To automatically move data between Kafka and other systems -> Option A

4. Quick Check:

   Connectors move data automatically = A [OK]

Hint: Connectors move data automatically between systems [OK]

Common Mistakes:

• Thinking connectors create Kafka topics
• Confusing connectors with monitoring tools
• Assuming connectors require manual data transfer

2. Which of the following is the correct way to specify a connector configuration file in Kafka Connect?

easy

A. connect-standalone.sh config/connect-standalone.properties config/my-connector.properties

B. kafka-topics.sh --create --config my-connector.properties

C. kafka-console-producer.sh --broker-list localhost:9092 --topic my-connector.properties

D. connect-distributed.sh --config my-connector.properties

Solution

1. Step 1: Identify the correct command for standalone mode

   The command connect-standalone.sh is used with two config files: worker and connector.

2. Step 2: Check the options

   connect-standalone.sh config/connect-standalone.properties config/my-connector.properties correctly shows the worker config and connector config files together.

3. Final Answer:

   connect-standalone.sh config/connect-standalone.properties config/my-connector.properties -> Option A

4. Quick Check:

   Standalone mode uses two config files = A [OK]

Hint: Standalone mode needs worker and connector config files [OK]

Common Mistakes:

• Using topic commands instead of connector commands
• Passing only one config file
• Confusing distributed and standalone commands

3. Given this connector configuration snippet:

{
  "name": "my-source-connector",
  "config": {
    "connector.class": "FileStreamSource",
    "tasks.max": "1",
    "file": "/tmp/input.txt",
    "topic": "test-topic"
  }
}

What will this connector do when started?

medium

A. Create a new Kafka topic named '/tmp/input.txt'

B. Read data from /tmp/input.txt and send it to Kafka topic 'test-topic'

C. Write data from Kafka topic 'test-topic' to /tmp/input.txt

D. Monitor the file /tmp/input.txt for changes but do not send data

Solution

1. Step 1: Identify connector type and direction

   The connector class is FileStreamSource, which reads from a file and sends data to Kafka.

2. Step 2: Understand configuration parameters

   The file parameter points to the input file, and topic specifies the Kafka topic to send data to.

3. Final Answer:

   Read data from /tmp/input.txt and send it to Kafka topic 'test-topic' -> Option B

4. Quick Check:

   FileStreamSource reads file to Kafka = C [OK]

Hint: FileStreamSource reads file and sends to Kafka topic [OK]

Common Mistakes:

• Confusing source with sink connectors
• Thinking it writes to the file instead of reading
• Assuming it creates topics automatically

4. You wrote this sink connector config:

{
  "name": "my-sink-connector",
  "config": {
    "connector.class": "FileStreamSink",
    "tasks.max": "1",
    "file": "/tmp/output.txt",
    "topic": "input-topic"
  }
}

But no data appears in /tmp/output.txt after running. What is the likely problem?

medium

A. The connector class name is incorrect; it should be FileStreamSource

B. The file path /tmp/output.txt is not writable by the connector

C. The topic name 'input-topic' does not exist or has no data

D. The tasks.max value must be greater than 1

Solution

1. Step 1: Check connector class correctness

   The class FileStreamSink is correct for writing Kafka data to a file.

2. Step 2: Verify topic existence and data flow

   If the topic input-topic does not exist or has no data, no output will be written.

3. Final Answer:

   The topic name 'input-topic' does not exist or has no data -> Option C

4. Quick Check:

   Sink needs data in topic to write = B [OK]

Hint: Check topic exists and has data for sink connectors [OK]

Common Mistakes:

• Assuming connector class is wrong when it is correct
• Ignoring topic data availability
• Thinking tasks.max affects data flow directly

5. You want to integrate a database with Kafka using connectors. Which approach best explains why connectors are used for this integration?

hard

A. Connectors only work with file systems, not databases

B. Connectors replace the database with Kafka topics

C. Connectors require manual polling of the database by the user

D. Connectors automate data transfer between the database and Kafka without custom code

Solution

1. Step 1: Understand connector purpose for databases

   Connectors automate moving data between Kafka and external systems like databases.

2. Step 2: Eliminate incorrect options

   Connectors do not replace databases, do not require manual polling, and support databases, not just files.

3. Final Answer:

   Connectors automate data transfer between the database and Kafka without custom code -> Option D

4. Quick Check:

   Connectors automate integration without manual code = D [OK]

Hint: Connectors automate data flow without manual coding [OK]

Common Mistakes:

• Thinking connectors replace databases
• Assuming manual polling is needed
• Believing connectors only work with files