Source connectors in Kafka - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

What is the output of this Kafka Connect source connector configuration?

Given this JSON configuration for a Kafka Connect source connector, what will be the value of the tasks.max property when the connector starts?

Kafka

{
  "name": "my-source-connector",
  "config": {
    "connector.class": "FileStreamSource",
    "tasks.max": "3",
    "file": "/tmp/input.txt",
    "topic": "test-topic"
  }
}

A Error: tasks.max must be an integer

B 1

C 0

D 3

💡 Hint

Check the value assigned to tasks.max in the config JSON.

The tasks.max property is set to the string "3" which Kafka Connect parses as integer 3, so the connector will start with 3 tasks.

🧠 Conceptual

intermediate

Which Kafka Connect source connector property controls the maximum number of tasks?

In Kafka Connect source connectors, which configuration property defines the maximum number of tasks that can be created to read data?

A offset.flush.interval.ms

B max.poll.records

C tasks.max

D batch.size

💡 Hint

It controls parallelism of the connector.

The tasks.max property controls how many tasks the connector can create to read data in parallel.

❓ Predict Output

advanced

What error does this source connector configuration cause?

What error will Kafka Connect raise when starting this source connector?

Kafka

{
  "name": "invalid-connector",
  "config": {
    "connector.class": "FileStreamSource",
    "tasks.max": "zero",
    "file": "/tmp/input.txt",
    "topic": "test-topic"
  }
}

A No error, tasks.max defaults to 1

B Error: tasks.max must be a positive integer

C Error: file path not found

D Error: topic name missing

💡 Hint

Check the value type of tasks.max.

The tasks.max property must be a positive integer. The string "zero" is invalid and causes a configuration error.

🚀 Application

advanced

How many tasks will be created with this source connector config?

If a source connector is configured with tasks.max set to 5, but the connector's taskConfigs method returns only 3 task configurations, how many tasks will Kafka Connect create?

A 3

B 5

C 0

D Error: mismatch between tasks.max and taskConfigs

💡 Hint

The number of tasks is limited by the number of task configurations returned.

Kafka Connect creates as many tasks as the number of task configurations returned by the connector, up to tasks.max. Here, only 3 task configs are returned, so 3 tasks are created.

🔧 Debug

expert

Why does this source connector fail to start?

This source connector configuration fails to start. What is the cause?

Kafka

{
  "name": "my-source",
  "config": {
    "connector.class": "org.apache.kafka.connect.file.FileStreamSourceConnector",
    "tasks.max": "2",
    "file": "/tmp/input.txt"
  }
}

A Missing required property: topic

B Invalid connector.class value

C tasks.max must be an integer, not string

D File path does not exist

💡 Hint

Check if all required properties for FileStreamSourceConnector are present.

The FileStreamSourceConnector requires the 'topic' property to specify the Kafka topic to write to. Without it, the connector fails to start.

Practice

1. What is the main purpose of a Kafka source connector?

easy

A. To automatically import data from external systems into Kafka topics

B. To export data from Kafka topics to external systems

C. To monitor Kafka cluster health

D. To manage Kafka topic partitions

Solution

1. Step 1: Understand the role of source connectors

   Source connectors are designed to bring data from outside systems into Kafka topics automatically.

2. Step 2: Differentiate from sink connectors

   Sink connectors do the opposite by exporting data from Kafka to external systems, so they are not the correct answer here.

3. Final Answer:

   To automatically import data from external systems into Kafka topics -> Option A

4. Quick Check:

   Source connector = import data [OK]

Hint: Source connectors bring data into Kafka topics [OK]

Common Mistakes:

• Confusing source connectors with sink connectors
• Thinking connectors manage Kafka internals like partitions
• Assuming connectors monitor Kafka health

2. Which of the following is the correct JSON field to specify the connector class in a Kafka source connector configuration?

easy

A. "sourceConnector"

B. "source.class"

C. "connectorType"

D. "connector.class"

Solution

1. Step 1: Recall Kafka Connect configuration syntax

   The key to specify the connector implementation class is always "connector.class" in the JSON config.

2. Step 2: Eliminate incorrect options

   Other options like "source.class", "connectorType", or "sourceConnector" are not valid Kafka Connect configuration keys.

3. Final Answer:

   "connector.class" -> Option D

4. Quick Check:

   Connector class key = "connector.class" [OK]

Hint: Use "connector.class" to specify connector type [OK]

Common Mistakes:

• Using incorrect or made-up JSON keys
• Confusing source connector config keys with sink connector keys
• Omitting quotes around JSON keys

3. Given this Kafka Connect source connector configuration snippet:

{
  "name": "my-source-connector",
  "connector.class": "FileStreamSource",
  "tasks.max": "1",
  "file": "/tmp/input.txt",
  "topic": "test-topic"
}

What will happen when this connector runs?

medium

A. It fails because 'tasks.max' must be an integer, not a string

B. It reads data from /tmp/input.txt and writes it to the Kafka topic 'test-topic'

C. It creates a new Kafka topic named '/tmp/input.txt'

D. It writes data from Kafka topic 'test-topic' to /tmp/input.txt

Solution

1. Step 1: Identify connector type and purpose

   The "FileStreamSource" connector reads data from a file and sends it to a Kafka topic.

2. Step 2: Understand configuration keys

   "file" specifies the input file, "topic" specifies the Kafka topic to write to, and "tasks.max" as a string is acceptable in JSON configs.

3. Final Answer:

   It reads data from /tmp/input.txt and writes it to the Kafka topic 'test-topic' -> Option B

4. Quick Check:

   FileStreamSource reads file to topic [OK]

Hint: FileStreamSource reads file into Kafka topic [OK]

Common Mistakes:

• Confusing source and sink connectors
• Thinking 'tasks.max' must be integer type (string is valid in JSON)
• Assuming connector creates topics automatically

4. You configured a source connector with this JSON:

{
  "name": "my-connector",
  "connector.class": "JdbcSourceConnector",
  "tasks.max": 2,
  "connection.url": "jdbc:mysql://localhost:3306/mydb",
  "topic.prefix": "db-"
}

But the connector fails to start. What is the most likely cause?

medium

A. Missing required property 'mode' for JdbcSourceConnector

B. Incorrect 'tasks.max' value; must be a string

C. The 'topic.prefix' should be 'topic.name' instead

D. The 'connection.url' must use PostgreSQL syntax

Solution

1. Step 1: Review JdbcSourceConnector required properties

   JdbcSourceConnector requires a 'mode' property to specify how to query the database (e.g., 'incrementing', 'timestamp+incrementing').

2. Step 2: Check other properties

   'tasks.max' as integer is valid, 'topic.prefix' is correct, and 'connection.url' can be MySQL syntax if using MySQL.

3. Final Answer:

   Missing required property 'mode' for JdbcSourceConnector -> Option A

4. Quick Check:

   JdbcSourceConnector needs 'mode' property [OK]

Hint: JdbcSourceConnector requires 'mode' property [OK]

Common Mistakes:

• Assuming 'tasks.max' must be string
• Confusing 'topic.prefix' with 'topic.name'
• Thinking connection URL must be PostgreSQL always

5. You want to ingest data from multiple CSV files located in a directory into Kafka using a source connector. Which approach best applies?

hard

A. Use JdbcSourceConnector with CSV files as database tables

B. Use the FileStreamSource connector with the directory path as 'file' property

C. Use a FilePulse source connector configured with the directory path to read all CSV files

D. Manually write a Kafka producer to read files and send data

Solution

1. Step 1: Identify connector capabilities

   FilePulse source connector supports reading multiple files from a directory and processing CSV files.

2. Step 2: Evaluate other options

   FileStreamSource reads only a single file, JdbcSourceConnector is for databases, and manual coding is not using source connectors.

3. Final Answer:

   Use a FilePulse source connector configured with the directory path to read all CSV files -> Option C

4. Quick Check:

   FilePulse reads multiple files from directory [OK]

Hint: FilePulse connector handles multiple files ingestion [OK]

Common Mistakes:

• Using FileStreamSource for multiple files
• Trying to use JdbcSourceConnector for CSV files
• Ignoring built-in connectors and coding manually