Connector configuration in Kafka - Cheat Sheet & Quick Revision

Recall & Review

beginner

What is a Kafka Connector configuration?

It is a set of settings that tell Kafka Connect how to connect to a data source or sink, what data to move, and how to handle it.

beginner

Name two essential properties in a Kafka Connector configuration.

1. "connector.class": specifies the connector plugin to use.
2. "tasks.max": defines the maximum number of tasks to run in parallel.

beginner

What does the topics property specify in a source connector configuration?

It lists the Kafka topics where the connector will write the data it reads from the source system.

intermediate

How do you specify the connection details to a database in a JDBC source connector configuration?

You use properties like connection.url, connection.user, and connection.password to provide the database URL, username, and password.

intermediate

Why is the tasks.max property important in connector configuration?

It controls how many parallel tasks the connector can run, which affects throughput and resource use. Setting it too low limits speed; too high may overload the system.

Which property defines the connector plugin to use in Kafka Connect?

A connector.class

B tasks.max

C topics

D connection.url

The property 'connector.class' tells Kafka Connect which connector plugin to load.

What does the 'tasks.max' property control?

A Kafka topic name

B Database connection timeout

C Connector plugin version

D Maximum number of parallel tasks

'tasks.max' sets how many tasks can run at the same time for the connector.

In a source connector, the 'topics' property specifies:

A Maximum tasks

B Where to write data in Kafka

C Connector class name

D Where to read data from Kafka

'topics' lists the Kafka topics where the connector writes data it reads from the source.

Which property is NOT typically part of a JDBC source connector configuration?

A topics.max

B connection.user

C connection.password

D connection.url

'topics.max' is not a valid property; the correct property is 'tasks.max'.

Why should you carefully set 'tasks.max'?

A To specify the connector class

B To set the Kafka topic name

C To balance throughput and resource use

D To define the database URL

'tasks.max' affects how many tasks run in parallel, impacting speed and system load.

Explain the role of the 'connector.class' and 'tasks.max' properties in Kafka Connector configuration.

Think about what tells Kafka Connect what to do and how many workers to use.

Describe how you would configure a JDBC source connector to connect to a database and write data to Kafka topics.

Focus on connection details and where data goes.

Practice

1. What is the main purpose of a connector configuration in Kafka Connect?

easy

A. To specify how data moves between Kafka and external systems

B. To define the Kafka broker's internal settings

C. To configure Kafka topic partitions

D. To set the Kafka consumer group ID

Solution

1. Step 1: Understand Kafka Connect role

   Kafka Connect uses connectors to move data between Kafka and other systems.

2. Step 2: Identify configuration purpose

   The connector configuration tells Kafka Connect how to perform this data movement.

3. Final Answer:

   To specify how data moves between Kafka and external systems -> Option A

4. Quick Check:

   Connector configuration = data movement setup [OK]

Hint: Connector config defines data flow direction [OK]

Common Mistakes:

• Confusing connector config with Kafka broker settings
• Thinking it configures Kafka topic partitions
• Mixing up consumer group ID with connector config

2. Which of the following is the correct way to specify the connector class in a Kafka Connect configuration JSON?

easy

A. "connectorClassName": "org.apache.kafka.connect.file.FileStreamSourceConnector"

B. "connector.class": "org.apache.kafka.connect.file.FileStreamSourceConnector"

C. "connectorClass": "org.apache.kafka.connect.file.FileStreamSourceConnector"

D. "connector-class": "org.apache.kafka.connect.file.FileStreamSourceConnector"

Solution

1. Step 1: Recall Kafka Connect JSON key format

   Kafka Connect uses keys with dot notation, e.g., "connector.class".

2. Step 2: Match correct key name

   Only "connector.class": "org.apache.kafka.connect.file.FileStreamSourceConnector" uses the correct key "connector.class" with quotes and colon.

3. Final Answer:

   "connector.class": "org.apache.kafka.connect.file.FileStreamSourceConnector" -> Option B

4. Quick Check:

   Correct key is "connector.class" [OK]

Hint: Use dot notation keys like "connector.class" [OK]

Common Mistakes:

• Using hyphens instead of dots in keys
• Using camelCase keys instead of dot notation
• Omitting quotes around keys or values

3. Given this connector configuration snippet:

{
  "name": "my-source-connector",
  "connector.class": "org.apache.kafka.connect.file.FileStreamSourceConnector",
  "tasks.max": "2",
  "file": "/tmp/input.txt",
  "topic": "my-topic"
}

What does the tasks.max setting control?

medium

A. Maximum number of Kafka topics to consume

B. Maximum number of retries for failed tasks

C. Maximum size of each task's data batch

D. Maximum number of tasks to run in parallel for this connector

Solution

1. Step 1: Understand tasks.max meaning

   In Kafka Connect, tasks.max sets how many parallel tasks the connector can run.

2. Step 2: Eliminate other options

   It does not control topic count, batch size, or retry count.

3. Final Answer:

   Maximum number of tasks to run in parallel for this connector -> Option D

4. Quick Check:

   tasks.max = parallel task limit [OK]

Hint: tasks.max limits parallel connector tasks [OK]

Common Mistakes:

• Confusing tasks.max with topic count
• Thinking it controls batch size
• Assuming it sets retry attempts

4. You have this connector config JSON:

{
  "name": "my-sink-connector",
  "connector.class": "org.apache.kafka.connect.file.FileStreamSinkConnector",
  "tasks.max": 1,
  "topics": "my-topic"
}

What is wrong with this configuration?

medium

A. "connector.class" value is incorrect for a sink connector

B. "topics" key should be "topic" (singular)

C. Missing "file" property to specify output file

D. "tasks.max" should be a string, not a number

Solution

1. Step 1: Check required properties for FileStreamSinkConnector

   This connector needs a "file" property to specify the output file path.

2. Step 2: Validate other properties

   "tasks.max" can be a number or string, "topics" is valid plural, and "connector.class" is correct.

3. Final Answer:

   Missing "file" property to specify output file -> Option C

4. Quick Check:

   FileStreamSinkConnector requires "file" property [OK]

Hint: FileStreamSinkConnector needs "file" path set [OK]

Common Mistakes:

• Assuming "tasks.max" must be string
• Confusing "topic" and "topics" keys
• Changing connector.class unnecessarily

5. You want to configure a Kafka Connect source connector to read from a database table and write to a Kafka topic. Which configuration property is essential to specify the Kafka topic where data will be published?

hard

A. "topic"

B. "topics"

C. "topics.regex"

D. "kafka.topic"

Solution

1. Step 1: Understand topic properties in connectors

   Source connectors typically use "topic" (singular) to specify the Kafka topic to publish data to.

2. Step 2: Differentiate from other keys

   "topics" is common for sink connectors; "topics.regex" is for pattern matching; "kafka.topic" is not standard.

3. Final Answer:

   "topic" -> Option A

4. Quick Check:

   Use "topic" to specify Kafka topic for sources [OK]

Hint: Use "topic" for source connector Kafka topic [OK]

Common Mistakes:

• Using plural "topics" instead of "topic"
• Confusing "topics.regex" with direct topic specification
• Inventing non-standard keys like "kafka.topic"