Source connectors in Kafka - Time & Space Complexity

Time Complexity: Source connectors

O(n)

Understanding Time Complexity

When using source connectors in Kafka, it's important to understand how the time to process data grows as the input size increases.

We want to know how the connector's work changes when more data arrives.

Scenario Under Consideration

Analyze the time complexity of the following source connector poll method.

public List<SourceRecord> poll() throws InterruptedException {
    List<SourceRecord> records = new ArrayList<>();
    while (hasMoreData()) {
        SourceRecord record = readNextRecord();
        records.add(record);
    }
    return records;
}

This code reads all available data from the source system and collects it into a list to send to Kafka.

Identify Repeating Operations

Identify the loops, recursion, array traversals that repeat.

• Primary operation: The while loop that reads each record one by one.
• How many times: Once for every available record in the source system.

How Execution Grows With Input

As the number of records to read grows, the time to run this method grows in a similar way.

Input Size (n)	Approx. Operations
10	About 10 reads and adds
100	About 100 reads and adds
1000	About 1000 reads and adds

Pattern observation: The work grows directly with the number of records to process.

Final Time Complexity

Time Complexity: O(n)

This means the time to read records grows in a straight line as the number of records increases.

Common Mistake

[X] Wrong: "The connector reads all data instantly regardless of size."

[OK] Correct: Each record must be read one by one, so more data means more time.

Interview Connect

Understanding how source connectors scale with data size helps you explain real-world data flow and performance in Kafka systems.

Self-Check

"What if the connector batches records in groups instead of one by one? How would the time complexity change?"

Practice

1. What is the main purpose of a Kafka source connector?

easy

A. To automatically import data from external systems into Kafka topics

B. To export data from Kafka topics to external systems

C. To monitor Kafka cluster health

D. To manage Kafka topic partitions

Solution

1. Step 1: Understand the role of source connectors

   Source connectors are designed to bring data from outside systems into Kafka topics automatically.

2. Step 2: Differentiate from sink connectors

   Sink connectors do the opposite by exporting data from Kafka to external systems, so they are not the correct answer here.

3. Final Answer:

   To automatically import data from external systems into Kafka topics -> Option A

4. Quick Check:

   Source connector = import data [OK]

Hint: Source connectors bring data into Kafka topics [OK]

Common Mistakes:

• Confusing source connectors with sink connectors
• Thinking connectors manage Kafka internals like partitions
• Assuming connectors monitor Kafka health

2. Which of the following is the correct JSON field to specify the connector class in a Kafka source connector configuration?

easy

A. "sourceConnector"

B. "source.class"

C. "connectorType"

D. "connector.class"

Solution

1. Step 1: Recall Kafka Connect configuration syntax

   The key to specify the connector implementation class is always "connector.class" in the JSON config.

2. Step 2: Eliminate incorrect options

   Other options like "source.class", "connectorType", or "sourceConnector" are not valid Kafka Connect configuration keys.

3. Final Answer:

   "connector.class" -> Option D

4. Quick Check:

   Connector class key = "connector.class" [OK]

Hint: Use "connector.class" to specify connector type [OK]

Common Mistakes:

• Using incorrect or made-up JSON keys
• Confusing source connector config keys with sink connector keys
• Omitting quotes around JSON keys

3. Given this Kafka Connect source connector configuration snippet:

{
  "name": "my-source-connector",
  "connector.class": "FileStreamSource",
  "tasks.max": "1",
  "file": "/tmp/input.txt",
  "topic": "test-topic"
}

What will happen when this connector runs?

medium

A. It fails because 'tasks.max' must be an integer, not a string

B. It reads data from /tmp/input.txt and writes it to the Kafka topic 'test-topic'

C. It creates a new Kafka topic named '/tmp/input.txt'

D. It writes data from Kafka topic 'test-topic' to /tmp/input.txt

Solution

1. Step 1: Identify connector type and purpose

   The "FileStreamSource" connector reads data from a file and sends it to a Kafka topic.

2. Step 2: Understand configuration keys

   "file" specifies the input file, "topic" specifies the Kafka topic to write to, and "tasks.max" as a string is acceptable in JSON configs.

3. Final Answer:

   It reads data from /tmp/input.txt and writes it to the Kafka topic 'test-topic' -> Option B

4. Quick Check:

   FileStreamSource reads file to topic [OK]

Hint: FileStreamSource reads file into Kafka topic [OK]

Common Mistakes:

• Confusing source and sink connectors
• Thinking 'tasks.max' must be integer type (string is valid in JSON)
• Assuming connector creates topics automatically

4. You configured a source connector with this JSON:

{
  "name": "my-connector",
  "connector.class": "JdbcSourceConnector",
  "tasks.max": 2,
  "connection.url": "jdbc:mysql://localhost:3306/mydb",
  "topic.prefix": "db-"
}

But the connector fails to start. What is the most likely cause?

medium

A. Missing required property 'mode' for JdbcSourceConnector

B. Incorrect 'tasks.max' value; must be a string

C. The 'topic.prefix' should be 'topic.name' instead

D. The 'connection.url' must use PostgreSQL syntax

Solution

1. Step 1: Review JdbcSourceConnector required properties

   JdbcSourceConnector requires a 'mode' property to specify how to query the database (e.g., 'incrementing', 'timestamp+incrementing').

2. Step 2: Check other properties

   'tasks.max' as integer is valid, 'topic.prefix' is correct, and 'connection.url' can be MySQL syntax if using MySQL.

3. Final Answer:

   Missing required property 'mode' for JdbcSourceConnector -> Option A

4. Quick Check:

   JdbcSourceConnector needs 'mode' property [OK]

Hint: JdbcSourceConnector requires 'mode' property [OK]

Common Mistakes:

• Assuming 'tasks.max' must be string
• Confusing 'topic.prefix' with 'topic.name'
• Thinking connection URL must be PostgreSQL always

5. You want to ingest data from multiple CSV files located in a directory into Kafka using a source connector. Which approach best applies?

hard

A. Use JdbcSourceConnector with CSV files as database tables

B. Use the FileStreamSource connector with the directory path as 'file' property

C. Use a FilePulse source connector configured with the directory path to read all CSV files

D. Manually write a Kafka producer to read files and send data

Solution

1. Step 1: Identify connector capabilities

   FilePulse source connector supports reading multiple files from a directory and processing CSV files.

2. Step 2: Evaluate other options

   FileStreamSource reads only a single file, JdbcSourceConnector is for databases, and manual coding is not using source connectors.

3. Final Answer:

   Use a FilePulse source connector configured with the directory path to read all CSV files -> Option C

4. Quick Check:

   FilePulse reads multiple files from directory [OK]

Hint: FilePulse connector handles multiple files ingestion [OK]

Common Mistakes:

• Using FileStreamSource for multiple files
• Trying to use JdbcSourceConnector for CSV files
• Ignoring built-in connectors and coding manually