Connector configuration in Kafka - Mini Project: Build & Apply

Connector configuration

📖 Scenario: You are setting up a Kafka connector to move data from a database to Kafka topics. This is a common task in data engineering to keep data pipelines running smoothly.

🎯 Goal: Build a simple Kafka connector configuration step-by-step. You will create the connector settings, add a topic prefix, configure the connector class, and finally print the full configuration.

📋 What You'll Learn

Create a dictionary called connector_config with initial connector settings

Add a configuration variable topic_prefix with the value db_

Update connector_config to include the topic.prefix key with the value from topic_prefix

Print the final connector_config dictionary

💡 Why This Matters

🌍 Real World

Kafka connectors are used to move data between databases and Kafka topics automatically, enabling real-time data pipelines.

💼 Career

Data engineers and developers often configure Kafka connectors to integrate data sources and sinks efficiently in production systems.

1

Create initial connector configuration

Create a dictionary called connector_config with these exact entries: "name": "my-connector", "connector.class": "io.confluent.connect.jdbc.JdbcSourceConnector", and "tasks.max": "1".

Kafka

# Create the initial connector_config dictionary with name, connector.class, and tasks.max
# Your code here

Need a hint?

Use curly braces {} to create a dictionary and separate key-value pairs with commas.

2

Add topic prefix configuration variable

Create a variable called topic_prefix and set it to the string "db_".

Kafka

connector_config = {
    "name": "my-connector",
    "connector.class": "io.confluent.connect.jdbc.JdbcSourceConnector",
    "tasks.max": "1"
}
# Create the topic_prefix variable and set it to "db_"
# Your code here

Need a hint?

Assign the string "db_" to the variable topic_prefix using the equals sign.

3

Add topic prefix to connector configuration

Add a new key-value pair to connector_config where the key is "topic.prefix" and the value is the variable topic_prefix.

Kafka

connector_config = {
    "name": "my-connector",
    "connector.class": "io.confluent.connect.jdbc.JdbcSourceConnector",
    "tasks.max": "1"
}
topic_prefix = "db_"
# Add the key "topic.prefix" with value topic_prefix to connector_config
# Your code here

Need a hint?

Use square brackets to add a new key to the dictionary and assign it the value of topic_prefix.

4

Print the final connector configuration

Write a print statement to display the connector_config dictionary.

Kafka

connector_config = {
    "name": "my-connector",
    "connector.class": "io.confluent.connect.jdbc.JdbcSourceConnector",
    "tasks.max": "1"
}
topic_prefix = "db_"
connector_config["topic.prefix"] = topic_prefix
# Print the connector_config dictionary
# Your code here

Need a hint?

Use print(connector_config) to show the dictionary on the screen.