Bird
Raised Fist0
Kafkadevops~30 mins

Kafka Connect architecture - Mini Project: Build & Apply

Choose your learning style10 modes available
LearnWhyDeepVisualTryChallengeProjectRecallTime
Understanding Kafka Connect Architecture
📖 Scenario: You are working in a company that needs to move data between different systems automatically. Kafka Connect helps you do this by connecting Kafka with other data sources and sinks without writing complex code.
🎯 Goal: Build a simple Kafka Connect configuration to understand how connectors work and how data flows through Kafka Connect architecture.
📋 What You'll Learn
Create a basic Kafka Connect source connector configuration
Define a connector name and connector class
Set up the connector to read from a file
Print the connector configuration to verify setup
💡 Why This Matters
🌍 Real World
Kafka Connect is used in companies to move data automatically between databases, files, and Kafka topics without writing custom code.
💼 Career
Understanding Kafka Connect architecture is important for data engineers and developers working with real-time data pipelines and integrations.
Progress0 / 4 steps
1
Create a Kafka Connect source connector configuration dictionary
Create a dictionary called source_connector_config with these exact entries: "name": "file-source-connector", "connector.class": "org.apache.kafka.connect.file.FileStreamSourceConnector", "tasks.max": "1", "file": "/tmp/test.txt", and "topic": "connect-test-topic".
Kafka
Hint

Think of this dictionary as the settings that tell Kafka Connect what to do.

2
Add a configuration variable for maximum tasks
Add a variable called max_tasks and set it to the string "1".
Kafka
Hint

This variable helps control how many tasks the connector can run.

3
Update the connector configuration to use the max_tasks variable
Update the source_connector_config dictionary so that the value for the key "tasks.max" is set to the variable max_tasks.
Kafka
Hint

Use the variable instead of the fixed string for tasks.max.

4
Print the connector configuration
Write a print statement to display the source_connector_config dictionary.
Kafka
Hint

Use print(source_connector_config) to see the configuration.

Practice

(1/5)
1. What is the main role of a Kafka Connect worker in the Kafka Connect architecture?
easy
A. To run connectors and tasks that move data between Kafka and external systems
B. To store Kafka topic data permanently
C. To produce messages to Kafka topics
D. To manage Kafka broker configurations

Solution

  1. Step 1: Understand Kafka Connect components

    Kafka Connect architecture includes workers, connectors, and tasks. Workers run connectors and tasks.

  2. Step 2: Identify worker's role

    Workers execute connectors and tasks to move data between Kafka and external systems.

  3. Final Answer:

    To run connectors and tasks that move data between Kafka and external systems -> Option A

  4. Quick Check:

    Worker = runs connectors/tasks [OK]
Hint: Workers run connectors and tasks to move data [OK]
Common Mistakes:
  • Confusing workers with Kafka brokers
  • Thinking workers store data permanently
  • Mixing up producers and workers
2. Which of the following is the correct way to start Kafka Connect in standalone mode using the command line?
easy
A. connect-distributed.sh worker.properties connector.properties
B. connect-standalone.sh worker.properties connector.properties
C. kafka-server-start.sh connect-standalone.properties
D. kafka-connect-start.sh standalone.properties

Solution

  1. Step 1: Recall Kafka Connect startup commands

    Standalone mode uses connect-standalone.sh with worker and connector config files.

  2. Step 2: Match correct command syntax

    connect-standalone.sh worker.properties connector.properties matches the correct command format: connect-standalone.sh worker.properties connector.properties

  3. Final Answer:

    connect-standalone.sh worker.properties connector.properties -> Option B

  4. Quick Check:

    Standalone mode uses connect-standalone.sh [OK]
Hint: Standalone mode uses connect-standalone.sh command [OK]
Common Mistakes:
  • Using connect-distributed.sh for standalone mode
  • Confusing Kafka broker start scripts with Connect scripts
  • Wrong file names or missing config files
3. Given the following Kafka Connect distributed mode configuration snippet:
{
  "name": "my-source-connector",
  "config": {
    "connector.class": "FileStreamSource",
    "tasks.max": "2",
    "file": "/tmp/input.txt",
    "topic": "test-topic"
  }
}

What does the tasks.max setting control?
medium
A. The maximum number of tasks to run for this connector
B. The maximum number of Kafka brokers to connect
C. The maximum size of each message sent
D. The maximum number of topics to read from

Solution

  1. Step 1: Understand the tasks.max parameter

    In Kafka Connect, tasks.max sets how many tasks a connector can run in parallel.

  2. Step 2: Apply to given config

    Here, tasks.max is 2, so the connector can run up to 2 tasks to increase throughput.

  3. Final Answer:

    The maximum number of tasks to run for this connector -> Option A

  4. Quick Check:

    tasks.max = max tasks per connector [OK]
Hint: tasks.max limits parallel tasks for connector [OK]
Common Mistakes:
  • Confusing tasks.max with broker count
  • Thinking it limits message size
  • Assuming it controls topic count
4. You have a distributed Kafka Connect cluster but notice that connectors are not starting tasks as expected. Which of the following is the most likely cause?
medium
A. The connector configuration file is missing the tasks.max property
B. Kafka brokers are offline
C. The worker.properties file is empty
D. The connector class name is misspelled in the configuration

Solution

  1. Step 1: Analyze why tasks don't start

    If connectors don't start tasks, a common cause is an invalid connector class name.

  2. Step 2: Check other options

    Missing tasks.max defaults to 1 task, so not likely cause. Brokers offline cause other errors. Empty worker.properties would prevent worker start. Misspelled connector class prevents task creation.

  3. Final Answer:

    The connector class name is misspelled in the configuration -> Option D

  4. Quick Check:

    Misspelled connector class blocks tasks [OK]
Hint: Check connector class spelling if tasks don't start [OK]
Common Mistakes:
  • Assuming missing tasks.max stops tasks
  • Ignoring connector class name correctness
  • Not verifying worker config completeness
5. You want to build a fault-tolerant data pipeline using Kafka Connect that can scale horizontally and recover from worker failures. Which Kafka Connect mode should you choose and why?
hard
A. Standalone mode, because it is simpler and easier to configure
B. Distributed mode, because it requires no configuration files
C. Distributed mode, because it supports multiple workers and fault tolerance
D. Standalone mode, because it automatically balances tasks across workers

Solution

  1. Step 1: Understand Kafka Connect modes

    Standalone mode runs a single worker, no fault tolerance or scaling. Distributed mode runs multiple workers, supports scaling and fault tolerance.

  2. Step 2: Match requirements to mode

    For fault tolerance and horizontal scaling, distributed mode is required.

  3. Final Answer:

    Distributed mode, because it supports multiple workers and fault tolerance -> Option C

  4. Quick Check:

    Fault tolerance needs distributed mode [OK]
Hint: Use distributed mode for scaling and fault tolerance [OK]
Common Mistakes:
  • Choosing standalone for fault tolerance
  • Thinking standalone balances tasks
  • Believing distributed mode needs no config
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard