Sink connectors in Kafka - Mini Project: Build & Apply

Sink Connectors with Kafka

📖 Scenario: You are working with Apache Kafka to move data from a Kafka topic to an external system using sink connectors. This is common in real-world data pipelines where data collected in Kafka needs to be stored or processed elsewhere.

🎯 Goal: Build a simple Kafka sink connector configuration step-by-step to send messages from a Kafka topic to a file system.

📋 What You'll Learn

Create a Kafka topic configuration

Set up a sink connector configuration variable

Write the connector configuration JSON with required fields

Print the final connector configuration JSON

💡 Why This Matters

🌍 Real World

Kafka sink connectors are used to move data from Kafka topics to external systems like files, databases, or search engines automatically.

💼 Career

Understanding sink connectors is important for data engineers and developers who build data pipelines and integrate Kafka with other systems.

1

Create Kafka topic configuration

Create a variable called topic_config as a dictionary with these exact entries: "name": "test-topic", "partitions": 3, "replication_factor": 1.

Kafka

# Your code here

Need a hint?

Use a Python dictionary with keys exactly as shown.

2

Set up sink connector configuration variable

Create a variable called sink_connector_config as an empty dictionary to hold the sink connector settings.

Kafka

topic_config = {
    "name": "test-topic",
    "partitions": 3,
    "replication_factor": 1
}
# Your code here

Need a hint?

Just create an empty dictionary named sink_connector_config.

3

Write sink connector configuration JSON

Add these exact key-value pairs to sink_connector_config: "name": "file-sink-connector", "connector.class": "FileStreamSink", "tasks.max": "1", "topics": "test-topic", "file": "/tmp/output.txt".

Kafka

topic_config = {
    "name": "test-topic",
    "partitions": 3,
    "replication_factor": 1
}
sink_connector_config = {}
# Your code here

Need a hint?

Assign all required keys and values exactly as shown inside sink_connector_config.

4

Print the sink connector configuration

Write a print statement to display the sink_connector_config dictionary.

Kafka

topic_config = {
    "name": "test-topic",
    "partitions": 3,
    "replication_factor": 1
}
sink_connector_config = {
    "name": "file-sink-connector",
    "connector.class": "FileStreamSink",
    "tasks.max": "1",
    "topics": "test-topic",
    "file": "/tmp/output.txt"
}
# Your code here

Need a hint?

Use print(sink_connector_config) to show the dictionary.