Kafka Connect architecture - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

What is the role of a Kafka Connect Worker?

In Kafka Connect architecture, what is the primary role of a Worker?

A It runs connectors and tasks to move data between Kafka and external systems.

B It stores the Kafka topics and manages partitions.

C It acts as a Kafka broker to route messages between producers and consumers.

D It provides a user interface for managing Kafka topics.

💡 Hint

Think about which component actually executes the data movement jobs.

A Kafka Connect Worker runs connectors and tasks that handle data transfer between Kafka and other systems. It does not store topics or act as a broker.

❓ Predict Output

intermediate

What is the output of this Kafka Connect REST API call?

Given a Kafka Connect cluster with two workers, what is the expected output of the REST API call GET /connectors if there are three connectors named source1, sink1, and sink2?

Kafka

curl -X GET http://localhost:8083/connectors

A {"connectors": ["source1", "sink1", "sink2"]}

B ["source1", "sink1", "sink2"]

C [{"name": "source1"}, {"name": "sink1"}, {"name": "sink2"}]

D source1, sink1, sink2

💡 Hint

Check the Kafka Connect REST API documentation for the format of the connectors list response.

The GET /connectors endpoint returns a JSON array of connector names as strings.

❓ Predict Output

advanced

What happens when a Kafka Connect task fails repeatedly?

Consider a Kafka Connect task that fails to start due to a configuration error. What is the expected behavior of the Kafka Connect framework regarding this task?

A The task is paused and requires manual restart from the user interface.

B The task immediately stops and the connector is removed from the cluster.

C The task logs the error once and then continues running with default settings.

D The task retries starting indefinitely with exponential backoff until fixed or manually stopped.

💡 Hint

Think about fault tolerance and automatic recovery in Kafka Connect.

Kafka Connect retries failed tasks with exponential backoff to handle transient errors automatically.

🧠 Conceptual

advanced

How does Kafka Connect achieve fault tolerance in distributed mode?

In distributed mode, how does Kafka Connect ensure fault tolerance and high availability of connectors and tasks?

A By using a single leader worker that handles all connectors and tasks exclusively.

B By replicating all connector configurations to each worker's local disk.

C By running multiple workers that coordinate via Kafka topics to share task assignments and state.

D By storing all data in an external database separate from Kafka.

💡 Hint

Consider how Kafka Connect uses Kafka itself for coordination.

Kafka Connect distributed mode uses Kafka topics to coordinate multiple workers, sharing task assignments and state for fault tolerance.

🚀 Application

expert

Determine the number of tasks for a source connector

You configure a Kafka Connect source connector to read from 5 partitions of an external system. The connector's tasks.max is set to 3. How many tasks will Kafka Connect create to read data?

A 3 tasks, because <code>tasks.max</code> limits the maximum tasks created.

B 5 tasks, one per partition regardless of <code>tasks.max</code>.

C 1 task, because Kafka Connect uses a single task by default.

D 8 tasks, combining partitions and <code>tasks.max</code>.

💡 Hint

Remember that tasks.max limits the number of tasks, but tasks can handle multiple partitions.

The connector creates up to tasks.max tasks. Since there are 5 partitions but tasks.max is 3, only 3 tasks are created. Each task can handle multiple partitions.

Practice

1. What is the main role of a Kafka Connect worker in the Kafka Connect architecture?

easy

A. To run connectors and tasks that move data between Kafka and external systems

B. To store Kafka topic data permanently

C. To produce messages to Kafka topics

D. To manage Kafka broker configurations

Solution

1. Step 1: Understand Kafka Connect components

   Kafka Connect architecture includes workers, connectors, and tasks. Workers run connectors and tasks.

2. Step 2: Identify worker's role

   Workers execute connectors and tasks to move data between Kafka and external systems.

3. Final Answer:

   To run connectors and tasks that move data between Kafka and external systems -> Option A

4. Quick Check:

   Worker = runs connectors/tasks [OK]

Hint: Workers run connectors and tasks to move data [OK]

Common Mistakes:

• Confusing workers with Kafka brokers
• Thinking workers store data permanently
• Mixing up producers and workers

2. Which of the following is the correct way to start Kafka Connect in standalone mode using the command line?

easy

A. connect-distributed.sh worker.properties connector.properties

B. connect-standalone.sh worker.properties connector.properties

C. kafka-server-start.sh connect-standalone.properties

D. kafka-connect-start.sh standalone.properties

Solution

1. Step 1: Recall Kafka Connect startup commands

   Standalone mode uses connect-standalone.sh with worker and connector config files.

2. Step 2: Match correct command syntax

   connect-standalone.sh worker.properties connector.properties matches the correct command format: connect-standalone.sh worker.properties connector.properties

3. Final Answer:

   connect-standalone.sh worker.properties connector.properties -> Option B

4. Quick Check:

   Standalone mode uses connect-standalone.sh [OK]

Hint: Standalone mode uses connect-standalone.sh command [OK]

Common Mistakes:

• Using connect-distributed.sh for standalone mode
• Confusing Kafka broker start scripts with Connect scripts
• Wrong file names or missing config files

3. Given the following Kafka Connect distributed mode configuration snippet:

{
  "name": "my-source-connector",
  "config": {
    "connector.class": "FileStreamSource",
    "tasks.max": "2",
    "file": "/tmp/input.txt",
    "topic": "test-topic"
  }
}

What does the tasks.max setting control?

medium

A. The maximum number of tasks to run for this connector

B. The maximum number of Kafka brokers to connect

C. The maximum size of each message sent

D. The maximum number of topics to read from

Solution

1. Step 1: Understand the tasks.max parameter

   In Kafka Connect, tasks.max sets how many tasks a connector can run in parallel.

2. Step 2: Apply to given config

   Here, tasks.max is 2, so the connector can run up to 2 tasks to increase throughput.

3. Final Answer:

   The maximum number of tasks to run for this connector -> Option A

4. Quick Check:

   tasks.max = max tasks per connector [OK]

Hint: tasks.max limits parallel tasks for connector [OK]

Common Mistakes:

• Confusing tasks.max with broker count
• Thinking it limits message size
• Assuming it controls topic count

4. You have a distributed Kafka Connect cluster but notice that connectors are not starting tasks as expected. Which of the following is the most likely cause?

medium

A. The connector configuration file is missing the tasks.max property

B. Kafka brokers are offline

C. The worker.properties file is empty

D. The connector class name is misspelled in the configuration

Solution

1. Step 1: Analyze why tasks don't start

   If connectors don't start tasks, a common cause is an invalid connector class name.

2. Step 2: Check other options

   Missing tasks.max defaults to 1 task, so not likely cause. Brokers offline cause other errors. Empty worker.properties would prevent worker start. Misspelled connector class prevents task creation.

3. Final Answer:

   The connector class name is misspelled in the configuration -> Option D

4. Quick Check:

   Misspelled connector class blocks tasks [OK]

Hint: Check connector class spelling if tasks don't start [OK]

Common Mistakes:

• Assuming missing tasks.max stops tasks
• Ignoring connector class name correctness
• Not verifying worker config completeness

5. You want to build a fault-tolerant data pipeline using Kafka Connect that can scale horizontally and recover from worker failures. Which Kafka Connect mode should you choose and why?

hard

A. Standalone mode, because it is simpler and easier to configure

B. Distributed mode, because it requires no configuration files

C. Distributed mode, because it supports multiple workers and fault tolerance

D. Standalone mode, because it automatically balances tasks across workers

Solution

1. Step 1: Understand Kafka Connect modes

   Standalone mode runs a single worker, no fault tolerance or scaling. Distributed mode runs multiple workers, supports scaling and fault tolerance.

2. Step 2: Match requirements to mode

   For fault tolerance and horizontal scaling, distributed mode is required.

3. Final Answer:

   Distributed mode, because it supports multiple workers and fault tolerance -> Option C

4. Quick Check:

   Fault tolerance needs distributed mode [OK]

Hint: Use distributed mode for scaling and fault tolerance [OK]

Common Mistakes:

• Choosing standalone for fault tolerance
• Thinking standalone balances tasks
• Believing distributed mode needs no config