Connector configuration in Kafka - Step-by-Step Execution

Process Flow - Connector configuration

Start: Define Connector Config

↓

Set Connector Name

↓

Specify Connector Class

↓

Configure Connection Details

↓

Set Topic and Data Format

↓

Apply Additional Settings

↓

Deploy Connector

↓

Connector Runs and Transfers Data

↓

End

This flow shows how to create and deploy a Kafka connector by defining its configuration step-by-step.

Execution Sample

Kafka

{
  "name": "my-source-connector",
  "connector.class": "FileStreamSource",
  "tasks.max": "1",
  "file": "/tmp/input.txt",
  "topic": "my-topic"
}

This JSON configures a source connector that reads from a file and writes to a Kafka topic.

Process Table

Step	Config Key	Value	Action	Result
1	name	my-source-connector	Set connector name	Connector identified as 'my-source-connector'
2	connector.class	FileStreamSource	Specify connector type	Connector will read from a file source
3	tasks.max	1	Set max tasks	Only one task will run for this connector
4	file	/tmp/input.txt	Set input file path	Connector reads data from '/tmp/input.txt'
5	topic	my-topic	Set Kafka topic	Data will be sent to 'my-topic'
6	Deploy	-	Deploy connector with above config	Connector starts running and transferring data
7	Run	-	Connector reads file and sends data	Data appears in Kafka topic 'my-topic'
8	-	-	End of configuration and deployment	Connector active and working

💡 Connector deployed and running; configuration complete.

Status Tracker

Config Key	Initial	After Step 1	After Step 2	After Step 3	After Step 4	After Step 5	Final
name	undefined	my-source-connector	my-source-connector	my-source-connector	my-source-connector	my-source-connector	my-source-connector
connector.class	undefined	undefined	FileStreamSource	FileStreamSource	FileStreamSource	FileStreamSource	FileStreamSource
tasks.max	undefined	undefined	undefined	1	1	1	1
file	undefined	undefined	undefined	undefined	/tmp/input.txt	/tmp/input.txt	/tmp/input.txt
topic	undefined	undefined	undefined	undefined	undefined	my-topic	my-topic

Key Moments - 3 Insights

Why do we need to specify 'connector.class' in the configuration?

The 'connector.class' tells Kafka Connect what type of connector to run, as shown in step 2 of the execution_table. Without it, Kafka Connect won't know how to handle the data source.

What happens if 'tasks.max' is set to more than 1?

Setting 'tasks.max' to more than 1 allows multiple tasks to run in parallel, increasing throughput. In the execution_table at step 3, it is set to 1, meaning only one task runs.

How does the connector know where to send the data?

The 'topic' config key (step 5) tells the connector which Kafka topic to send data to. Without this, data would have no destination.

Visual Quiz - 3 Questions

Look at the execution_table, what is the value of 'file' after step 4?

A "my-topic"

B "/tmp/input.txt"

C "FileStreamSource"

D "my-source-connector"

💡 Hint

Check the 'Value' column at step 4 in the execution_table.

At which step does the connector start running and transferring data?

A Step 3

B Step 5

C Step 6

D Step 2

💡 Hint

Look for the 'Action' column mentioning deployment in the execution_table.

If we change 'tasks.max' to 3, what will change in the variable_tracker?

A 'tasks.max' will show '3' after step 3 and onwards

B 'connector.class' will change to '3'

C 'file' will change to '3'

D 'name' will change to '3'

💡 Hint

Look at the 'tasks.max' row in variable_tracker and imagine changing its value.

Concept Snapshot

Connector configuration in Kafka Connect:
- Define JSON with keys like 'name', 'connector.class', 'tasks.max', 'file', 'topic'
- 'name' identifies the connector
- 'connector.class' specifies connector type
- 'tasks.max' controls parallelism
- 'file' or other source config sets input
- 'topic' sets Kafka destination
- Deploy config to start connector

Full Transcript

This visual execution shows how to configure a Kafka connector step-by-step. We start by setting the connector name, then specify the connector class to tell Kafka Connect what type of connector to run. Next, we set the maximum number of tasks to control parallelism. Then, we provide connection details like the input file path and the Kafka topic to send data to. After applying all settings, we deploy the connector, which then runs and transfers data from the source to the Kafka topic. The variable tracker shows how each configuration key is set over time. Key moments clarify why each setting is important. The quiz tests understanding of configuration values and deployment steps.

Practice

1. What is the main purpose of a connector configuration in Kafka Connect?

easy

A. To specify how data moves between Kafka and external systems

B. To define the Kafka broker's internal settings

C. To configure Kafka topic partitions

D. To set the Kafka consumer group ID

Solution

1. Step 1: Understand Kafka Connect role

   Kafka Connect uses connectors to move data between Kafka and other systems.

2. Step 2: Identify configuration purpose

   The connector configuration tells Kafka Connect how to perform this data movement.

3. Final Answer:

   To specify how data moves between Kafka and external systems -> Option A

4. Quick Check:

   Connector configuration = data movement setup [OK]

Hint: Connector config defines data flow direction [OK]

Common Mistakes:

• Confusing connector config with Kafka broker settings
• Thinking it configures Kafka topic partitions
• Mixing up consumer group ID with connector config

2. Which of the following is the correct way to specify the connector class in a Kafka Connect configuration JSON?

easy

A. "connectorClassName": "org.apache.kafka.connect.file.FileStreamSourceConnector"

B. "connector.class": "org.apache.kafka.connect.file.FileStreamSourceConnector"

C. "connectorClass": "org.apache.kafka.connect.file.FileStreamSourceConnector"

D. "connector-class": "org.apache.kafka.connect.file.FileStreamSourceConnector"

Solution

1. Step 1: Recall Kafka Connect JSON key format

   Kafka Connect uses keys with dot notation, e.g., "connector.class".

2. Step 2: Match correct key name

   Only "connector.class": "org.apache.kafka.connect.file.FileStreamSourceConnector" uses the correct key "connector.class" with quotes and colon.

3. Final Answer:

   "connector.class": "org.apache.kafka.connect.file.FileStreamSourceConnector" -> Option B

4. Quick Check:

   Correct key is "connector.class" [OK]

Hint: Use dot notation keys like "connector.class" [OK]

Common Mistakes:

• Using hyphens instead of dots in keys
• Using camelCase keys instead of dot notation
• Omitting quotes around keys or values

3. Given this connector configuration snippet:

{
  "name": "my-source-connector",
  "connector.class": "org.apache.kafka.connect.file.FileStreamSourceConnector",
  "tasks.max": "2",
  "file": "/tmp/input.txt",
  "topic": "my-topic"
}

What does the tasks.max setting control?

medium

A. Maximum number of Kafka topics to consume

B. Maximum number of retries for failed tasks

C. Maximum size of each task's data batch

D. Maximum number of tasks to run in parallel for this connector

Solution

1. Step 1: Understand tasks.max meaning

   In Kafka Connect, tasks.max sets how many parallel tasks the connector can run.

2. Step 2: Eliminate other options

   It does not control topic count, batch size, or retry count.

3. Final Answer:

   Maximum number of tasks to run in parallel for this connector -> Option D

4. Quick Check:

   tasks.max = parallel task limit [OK]

Hint: tasks.max limits parallel connector tasks [OK]

Common Mistakes:

• Confusing tasks.max with topic count
• Thinking it controls batch size
• Assuming it sets retry attempts

4. You have this connector config JSON:

{
  "name": "my-sink-connector",
  "connector.class": "org.apache.kafka.connect.file.FileStreamSinkConnector",
  "tasks.max": 1,
  "topics": "my-topic"
}

What is wrong with this configuration?

medium

A. "connector.class" value is incorrect for a sink connector

B. "topics" key should be "topic" (singular)

C. Missing "file" property to specify output file

D. "tasks.max" should be a string, not a number

Solution

1. Step 1: Check required properties for FileStreamSinkConnector

   This connector needs a "file" property to specify the output file path.

2. Step 2: Validate other properties

   "tasks.max" can be a number or string, "topics" is valid plural, and "connector.class" is correct.

3. Final Answer:

   Missing "file" property to specify output file -> Option C

4. Quick Check:

   FileStreamSinkConnector requires "file" property [OK]

Hint: FileStreamSinkConnector needs "file" path set [OK]

Common Mistakes:

• Assuming "tasks.max" must be string
• Confusing "topic" and "topics" keys
• Changing connector.class unnecessarily

5. You want to configure a Kafka Connect source connector to read from a database table and write to a Kafka topic. Which configuration property is essential to specify the Kafka topic where data will be published?

hard

A. "topic"

B. "topics"

C. "topics.regex"

D. "kafka.topic"

Solution

1. Step 1: Understand topic properties in connectors

   Source connectors typically use "topic" (singular) to specify the Kafka topic to publish data to.

2. Step 2: Differentiate from other keys

   "topics" is common for sink connectors; "topics.regex" is for pattern matching; "kafka.topic" is not standard.

3. Final Answer:

   "topic" -> Option A

4. Quick Check:

   Use "topic" to specify Kafka topic for sources [OK]

Hint: Use "topic" for source connector Kafka topic [OK]

Common Mistakes:

• Using plural "topics" instead of "topic"
• Confusing "topics.regex" with direct topic specification
• Inventing non-standard keys like "kafka.topic"