Source connectors in Kafka - Cheat Sheet & Quick Revision

Recall & Review

beginner

What is a Source Connector in Kafka Connect?

A Source Connector is a component that imports data from an external system into Kafka topics. It acts like a bridge, continuously pulling data from the source and pushing it into Kafka.

beginner

Name two common external systems from which Kafka Source Connectors can import data.

Common external systems include databases (like MySQL, PostgreSQL) and file systems (like CSV files or logs). Source connectors read data from these and send it to Kafka.

intermediate

How does a Source Connector handle data format conversion?

Source Connectors convert the external data into Kafka's message format, often using serializers like JSON or Avro, so Kafka can store and process the data efficiently.

intermediate

What is the role of Kafka Connect in managing Source Connectors?

Kafka Connect manages the lifecycle of Source Connectors, including starting, stopping, and scaling them. It also handles fault tolerance and configuration management.

beginner

Explain the difference between a Source Connector and a Sink Connector.

A Source Connector imports data into Kafka from external systems, while a Sink Connector exports data from Kafka to external systems. They work in opposite directions.

What does a Kafka Source Connector do?

A Imports data from external systems into Kafka

B Exports data from Kafka to external systems

C Manages Kafka cluster nodes

D Monitors Kafka topic sizes

A Source Connector imports data from external systems into Kafka topics.

Which of these is NOT typically a source for a Kafka Source Connector?

A Relational databases

B File systems

C Kafka topics

D Message queues

Kafka topics are the destination, not a source, for Source Connectors.

How does Kafka Connect help with Source Connectors?

A By storing data permanently

B By managing their lifecycle and configuration

C By encrypting Kafka messages

D By creating Kafka topics automatically

Kafka Connect manages starting, stopping, and configuring Source Connectors.

What data format might a Source Connector use to send data to Kafka?

A Markdown

B HTML

C CSS

D JSON

JSON is a common data format used by Source Connectors to serialize data for Kafka.

Which statement is true about Source and Sink Connectors?

A Source Connectors send data into Kafka; Sink Connectors send data out

B Both send data into Kafka

C Both send data out of Kafka

D They perform the same function

Source Connectors bring data into Kafka; Sink Connectors export data from Kafka.

Describe what a Kafka Source Connector is and how it works in simple terms.

Think of it as a data pipeline from outside systems into Kafka.

Explain the difference between Source Connectors and Sink Connectors in Kafka Connect.

One brings data in, the other sends data out.

Practice

1. What is the main purpose of a Kafka source connector?

easy

A. To automatically import data from external systems into Kafka topics

B. To export data from Kafka topics to external systems

C. To monitor Kafka cluster health

D. To manage Kafka topic partitions

Solution

1. Step 1: Understand the role of source connectors

   Source connectors are designed to bring data from outside systems into Kafka topics automatically.

2. Step 2: Differentiate from sink connectors

   Sink connectors do the opposite by exporting data from Kafka to external systems, so they are not the correct answer here.

3. Final Answer:

   To automatically import data from external systems into Kafka topics -> Option A

4. Quick Check:

   Source connector = import data [OK]

Hint: Source connectors bring data into Kafka topics [OK]

Common Mistakes:

• Confusing source connectors with sink connectors
• Thinking connectors manage Kafka internals like partitions
• Assuming connectors monitor Kafka health

2. Which of the following is the correct JSON field to specify the connector class in a Kafka source connector configuration?

easy

A. "sourceConnector"

B. "source.class"

C. "connectorType"

D. "connector.class"

Solution

1. Step 1: Recall Kafka Connect configuration syntax

   The key to specify the connector implementation class is always "connector.class" in the JSON config.

2. Step 2: Eliminate incorrect options

   Other options like "source.class", "connectorType", or "sourceConnector" are not valid Kafka Connect configuration keys.

3. Final Answer:

   "connector.class" -> Option D

4. Quick Check:

   Connector class key = "connector.class" [OK]

Hint: Use "connector.class" to specify connector type [OK]

Common Mistakes:

• Using incorrect or made-up JSON keys
• Confusing source connector config keys with sink connector keys
• Omitting quotes around JSON keys

3. Given this Kafka Connect source connector configuration snippet:

{
  "name": "my-source-connector",
  "connector.class": "FileStreamSource",
  "tasks.max": "1",
  "file": "/tmp/input.txt",
  "topic": "test-topic"
}

What will happen when this connector runs?

medium

A. It fails because 'tasks.max' must be an integer, not a string

B. It reads data from /tmp/input.txt and writes it to the Kafka topic 'test-topic'

C. It creates a new Kafka topic named '/tmp/input.txt'

D. It writes data from Kafka topic 'test-topic' to /tmp/input.txt

Solution

1. Step 1: Identify connector type and purpose

   The "FileStreamSource" connector reads data from a file and sends it to a Kafka topic.

2. Step 2: Understand configuration keys

   "file" specifies the input file, "topic" specifies the Kafka topic to write to, and "tasks.max" as a string is acceptable in JSON configs.

3. Final Answer:

   It reads data from /tmp/input.txt and writes it to the Kafka topic 'test-topic' -> Option B

4. Quick Check:

   FileStreamSource reads file to topic [OK]

Hint: FileStreamSource reads file into Kafka topic [OK]

Common Mistakes:

• Confusing source and sink connectors
• Thinking 'tasks.max' must be integer type (string is valid in JSON)
• Assuming connector creates topics automatically

4. You configured a source connector with this JSON:

{
  "name": "my-connector",
  "connector.class": "JdbcSourceConnector",
  "tasks.max": 2,
  "connection.url": "jdbc:mysql://localhost:3306/mydb",
  "topic.prefix": "db-"
}

But the connector fails to start. What is the most likely cause?

medium

A. Missing required property 'mode' for JdbcSourceConnector

B. Incorrect 'tasks.max' value; must be a string

C. The 'topic.prefix' should be 'topic.name' instead

D. The 'connection.url' must use PostgreSQL syntax

Solution

1. Step 1: Review JdbcSourceConnector required properties

   JdbcSourceConnector requires a 'mode' property to specify how to query the database (e.g., 'incrementing', 'timestamp+incrementing').

2. Step 2: Check other properties

   'tasks.max' as integer is valid, 'topic.prefix' is correct, and 'connection.url' can be MySQL syntax if using MySQL.

3. Final Answer:

   Missing required property 'mode' for JdbcSourceConnector -> Option A

4. Quick Check:

   JdbcSourceConnector needs 'mode' property [OK]

Hint: JdbcSourceConnector requires 'mode' property [OK]

Common Mistakes:

• Assuming 'tasks.max' must be string
• Confusing 'topic.prefix' with 'topic.name'
• Thinking connection URL must be PostgreSQL always

5. You want to ingest data from multiple CSV files located in a directory into Kafka using a source connector. Which approach best applies?

hard

A. Use JdbcSourceConnector with CSV files as database tables

B. Use the FileStreamSource connector with the directory path as 'file' property

C. Use a FilePulse source connector configured with the directory path to read all CSV files

D. Manually write a Kafka producer to read files and send data

Solution

1. Step 1: Identify connector capabilities

   FilePulse source connector supports reading multiple files from a directory and processing CSV files.

2. Step 2: Evaluate other options

   FileStreamSource reads only a single file, JdbcSourceConnector is for databases, and manual coding is not using source connectors.

3. Final Answer:

   Use a FilePulse source connector configured with the directory path to read all CSV files -> Option C

4. Quick Check:

   FilePulse reads multiple files from directory [OK]

Hint: FilePulse connector handles multiple files ingestion [OK]

Common Mistakes:

• Using FileStreamSource for multiple files
• Trying to use JdbcSourceConnector for CSV files
• Ignoring built-in connectors and coding manually