Kafka Connect architecture - Mini Project: Build & Apply

Understanding Kafka Connect Architecture

📖 Scenario: You are working in a company that needs to move data between different systems automatically. Kafka Connect helps you do this by connecting Kafka with other data sources and sinks without writing complex code.

🎯 Goal: Build a simple Kafka Connect configuration to understand how connectors work and how data flows through Kafka Connect architecture.

📋 What You'll Learn

Create a basic Kafka Connect source connector configuration

Define a connector name and connector class

Set up the connector to read from a file

Print the connector configuration to verify setup

💡 Why This Matters

🌍 Real World

Kafka Connect is used in companies to move data automatically between databases, files, and Kafka topics without writing custom code.

💼 Career

Understanding Kafka Connect architecture is important for data engineers and developers working with real-time data pipelines and integrations.

1

Create a Kafka Connect source connector configuration dictionary

Create a dictionary called source_connector_config with these exact entries: "name": "file-source-connector", "connector.class": "org.apache.kafka.connect.file.FileStreamSourceConnector", "tasks.max": "1", "file": "/tmp/test.txt", and "topic": "connect-test-topic".

Kafka

# Create the source connector configuration dictionary
# Your code here

Need a hint?

Think of this dictionary as the settings that tell Kafka Connect what to do.

2

Add a configuration variable for maximum tasks

Add a variable called max_tasks and set it to the string "1".

Kafka

source_connector_config = {
    "name": "file-source-connector",
    "connector.class": "org.apache.kafka.connect.file.FileStreamSourceConnector",
    "tasks.max": "1",
    "file": "/tmp/test.txt",
    "topic": "connect-test-topic"
}
# Define max_tasks variable
# Your code here

Need a hint?

This variable helps control how many tasks the connector can run.

3

Update the connector configuration to use the max_tasks variable

Update the source_connector_config dictionary so that the value for the key "tasks.max" is set to the variable max_tasks.

Kafka

source_connector_config = {
    "name": "file-source-connector",
    "connector.class": "org.apache.kafka.connect.file.FileStreamSourceConnector",
    "tasks.max": max_tasks,
    "file": "/tmp/test.txt",
    "topic": "connect-test-topic"
}
max_tasks = "1"
# Your code here

Need a hint?

Use the variable instead of the fixed string for tasks.max.

4

Print the connector configuration

Write a print statement to display the source_connector_config dictionary.

Kafka

max_tasks = "1"
source_connector_config = {
    "name": "file-source-connector",
    "connector.class": "org.apache.kafka.connect.file.FileStreamSourceConnector",
    "tasks.max": max_tasks,
    "file": "/tmp/test.txt",
    "topic": "connect-test-topic"
}
# Print the connector configuration
# Your code here

Need a hint?

Use print(source_connector_config) to see the configuration.