Connector configuration in Kafka - Mini Project: Build & Apply

Connector configuration

📖 Scenario: You are setting up a Kafka connector to move data from a database to Kafka topics. This is a common task in data engineering to keep data pipelines running smoothly.

🎯 Goal: Build a simple Kafka connector configuration step-by-step. You will create the connector settings, add a topic prefix, configure the connector class, and finally print the full configuration.

📋 What You'll Learn

Create a dictionary called connector_config with initial connector settings

Add a configuration variable topic_prefix with the value db_

Update connector_config to include the topic.prefix key with the value from topic_prefix

Print the final connector_config dictionary

💡 Why This Matters

🌍 Real World

Kafka connectors are used to move data between databases and Kafka topics automatically, enabling real-time data pipelines.

💼 Career

Data engineers and developers often configure Kafka connectors to integrate data sources and sinks efficiently in production systems.

1

Create initial connector configuration

Create a dictionary called connector_config with these exact entries: "name": "my-connector", "connector.class": "io.confluent.connect.jdbc.JdbcSourceConnector", and "tasks.max": "1".

Kafka

# Create the initial connector_config dictionary with name, connector.class, and tasks.max
# Your code here

Hint

Use curly braces {} to create a dictionary and separate key-value pairs with commas.

2

Add topic prefix configuration variable

Create a variable called topic_prefix and set it to the string "db_".

Kafka

connector_config = {
    "name": "my-connector",
    "connector.class": "io.confluent.connect.jdbc.JdbcSourceConnector",
    "tasks.max": "1"
}
# Create the topic_prefix variable and set it to "db_"
# Your code here

Hint

Assign the string "db_" to the variable topic_prefix using the equals sign.

3

Add topic prefix to connector configuration

Add a new key-value pair to connector_config where the key is "topic.prefix" and the value is the variable topic_prefix.

Kafka

connector_config = {
    "name": "my-connector",
    "connector.class": "io.confluent.connect.jdbc.JdbcSourceConnector",
    "tasks.max": "1"
}
topic_prefix = "db_"
# Add the key "topic.prefix" with value topic_prefix to connector_config
# Your code here

Hint

Use square brackets to add a new key to the dictionary and assign it the value of topic_prefix.

4

Print the final connector configuration

Write a print statement to display the connector_config dictionary.

Kafka

connector_config = {
    "name": "my-connector",
    "connector.class": "io.confluent.connect.jdbc.JdbcSourceConnector",
    "tasks.max": "1"
}
topic_prefix = "db_"
connector_config["topic.prefix"] = topic_prefix
# Print the connector_config dictionary
# Your code here

Hint

Use print(connector_config) to show the dictionary on the screen.

Practice

1. What is the main purpose of a connector configuration in Kafka Connect?

easy

A. To specify how data moves between Kafka and external systems

B. To define the Kafka broker's internal settings

C. To configure Kafka topic partitions

D. To set the Kafka consumer group ID

Solution

1. Step 1: Understand Kafka Connect role

   Kafka Connect uses connectors to move data between Kafka and other systems.

2. Step 2: Identify configuration purpose

   The connector configuration tells Kafka Connect how to perform this data movement.

3. Final Answer:

   To specify how data moves between Kafka and external systems -> Option A

4. Quick Check:

   Connector configuration = data movement setup [OK]

Hint: Connector config defines data flow direction [OK]

Common Mistakes:

• Confusing connector config with Kafka broker settings
• Thinking it configures Kafka topic partitions
• Mixing up consumer group ID with connector config

2. Which of the following is the correct way to specify the connector class in a Kafka Connect configuration JSON?

easy

A. "connectorClassName": "org.apache.kafka.connect.file.FileStreamSourceConnector"

B. "connector.class": "org.apache.kafka.connect.file.FileStreamSourceConnector"

C. "connectorClass": "org.apache.kafka.connect.file.FileStreamSourceConnector"

D. "connector-class": "org.apache.kafka.connect.file.FileStreamSourceConnector"

Solution

1. Step 1: Recall Kafka Connect JSON key format

   Kafka Connect uses keys with dot notation, e.g., "connector.class".

2. Step 2: Match correct key name

   Only "connector.class": "org.apache.kafka.connect.file.FileStreamSourceConnector" uses the correct key "connector.class" with quotes and colon.

3. Final Answer:

   "connector.class": "org.apache.kafka.connect.file.FileStreamSourceConnector" -> Option B

4. Quick Check:

   Correct key is "connector.class" [OK]

Hint: Use dot notation keys like "connector.class" [OK]

Common Mistakes:

• Using hyphens instead of dots in keys
• Using camelCase keys instead of dot notation
• Omitting quotes around keys or values

3. Given this connector configuration snippet:

{
  "name": "my-source-connector",
  "connector.class": "org.apache.kafka.connect.file.FileStreamSourceConnector",
  "tasks.max": "2",
  "file": "/tmp/input.txt",
  "topic": "my-topic"
}

What does the tasks.max setting control?

medium

A. Maximum number of Kafka topics to consume

B. Maximum number of retries for failed tasks

C. Maximum size of each task's data batch

D. Maximum number of tasks to run in parallel for this connector

Solution

1. Step 1: Understand tasks.max meaning

   In Kafka Connect, tasks.max sets how many parallel tasks the connector can run.

2. Step 2: Eliminate other options

   It does not control topic count, batch size, or retry count.

3. Final Answer:

   Maximum number of tasks to run in parallel for this connector -> Option D

4. Quick Check:

   tasks.max = parallel task limit [OK]

Hint: tasks.max limits parallel connector tasks [OK]

Common Mistakes:

• Confusing tasks.max with topic count
• Thinking it controls batch size
• Assuming it sets retry attempts

4. You have this connector config JSON:

{
  "name": "my-sink-connector",
  "connector.class": "org.apache.kafka.connect.file.FileStreamSinkConnector",
  "tasks.max": 1,
  "topics": "my-topic"
}

What is wrong with this configuration?

medium

A. "connector.class" value is incorrect for a sink connector

B. "topics" key should be "topic" (singular)

C. Missing "file" property to specify output file

D. "tasks.max" should be a string, not a number

Solution

1. Step 1: Check required properties for FileStreamSinkConnector

   This connector needs a "file" property to specify the output file path.

2. Step 2: Validate other properties

   "tasks.max" can be a number or string, "topics" is valid plural, and "connector.class" is correct.

3. Final Answer:

   Missing "file" property to specify output file -> Option C

4. Quick Check:

   FileStreamSinkConnector requires "file" property [OK]

Hint: FileStreamSinkConnector needs "file" path set [OK]

Common Mistakes:

• Assuming "tasks.max" must be string
• Confusing "topic" and "topics" keys
• Changing connector.class unnecessarily

5. You want to configure a Kafka Connect source connector to read from a database table and write to a Kafka topic. Which configuration property is essential to specify the Kafka topic where data will be published?

hard

A. "topic"

B. "topics"

C. "topics.regex"

D. "kafka.topic"

Solution

1. Step 1: Understand topic properties in connectors

   Source connectors typically use "topic" (singular) to specify the Kafka topic to publish data to.

2. Step 2: Differentiate from other keys

   "topics" is common for sink connectors; "topics.regex" is for pattern matching; "kafka.topic" is not standard.

3. Final Answer:

   "topic" -> Option A

4. Quick Check:

   Use "topic" to specify Kafka topic for sources [OK]

Hint: Use "topic" for source connector Kafka topic [OK]

Common Mistakes:

• Using plural "topics" instead of "topic"
• Confusing "topics.regex" with direct topic specification
• Inventing non-standard keys like "kafka.topic"