Kafka Connect architecture - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

What is the role of a Kafka Connect Worker?

In Kafka Connect architecture, what is the primary role of a Worker?

A It runs connectors and tasks to move data between Kafka and external systems.

B It stores the Kafka topics and manages partitions.

C It acts as a Kafka broker to route messages between producers and consumers.

D It provides a user interface for managing Kafka topics.

💡 Hint

Think about which component actually executes the data movement jobs.

A Kafka Connect Worker runs connectors and tasks that handle data transfer between Kafka and other systems. It does not store topics or act as a broker.

❓ Predict Output

intermediate

What is the output of this Kafka Connect REST API call?

Given a Kafka Connect cluster with two workers, what is the expected output of the REST API call GET /connectors if there are three connectors named source1, sink1, and sink2?

Kafka

curl -X GET http://localhost:8083/connectors

A {"connectors": ["source1", "sink1", "sink2"]}

B ["source1", "sink1", "sink2"]

C [{"name": "source1"}, {"name": "sink1"}, {"name": "sink2"}]

D source1, sink1, sink2

💡 Hint

Check the Kafka Connect REST API documentation for the format of the connectors list response.

The GET /connectors endpoint returns a JSON array of connector names as strings.

❓ Predict Output

advanced

What happens when a Kafka Connect task fails repeatedly?

Consider a Kafka Connect task that fails to start due to a configuration error. What is the expected behavior of the Kafka Connect framework regarding this task?

A The task is paused and requires manual restart from the user interface.

B The task immediately stops and the connector is removed from the cluster.

C The task logs the error once and then continues running with default settings.

D The task retries starting indefinitely with exponential backoff until fixed or manually stopped.

💡 Hint

Think about fault tolerance and automatic recovery in Kafka Connect.

Kafka Connect retries failed tasks with exponential backoff to handle transient errors automatically.

🧠 Conceptual

advanced

How does Kafka Connect achieve fault tolerance in distributed mode?

In distributed mode, how does Kafka Connect ensure fault tolerance and high availability of connectors and tasks?

A By using a single leader worker that handles all connectors and tasks exclusively.

B By replicating all connector configurations to each worker's local disk.

C By running multiple workers that coordinate via Kafka topics to share task assignments and state.

D By storing all data in an external database separate from Kafka.

💡 Hint

Consider how Kafka Connect uses Kafka itself for coordination.

Kafka Connect distributed mode uses Kafka topics to coordinate multiple workers, sharing task assignments and state for fault tolerance.

🚀 Application

expert

Determine the number of tasks for a source connector

You configure a Kafka Connect source connector to read from 5 partitions of an external system. The connector's tasks.max is set to 3. How many tasks will Kafka Connect create to read data?

A 3 tasks, because <code>tasks.max</code> limits the maximum tasks created.

B 5 tasks, one per partition regardless of <code>tasks.max</code>.

C 1 task, because Kafka Connect uses a single task by default.

D 8 tasks, combining partitions and <code>tasks.max</code>.

💡 Hint

Remember that tasks.max limits the number of tasks, but tasks can handle multiple partitions.

The connector creates up to tasks.max tasks. Since there are 5 partitions but tasks.max is 3, only 3 tasks are created. Each task can handle multiple partitions.