Why connectors integrate external systems in Kafka - Visual Breakdown

Process Flow - Why connectors integrate external systems

External System Data

↓

Connector reads data

↓

Transforms data if needed

↓

Sends data to Kafka topics

↓

Kafka processes and stores data

↓

Consumers read from Kafka topics

Connectors act as bridges that read data from external systems, optionally transform it, and send it into Kafka topics for processing and consumption.

Execution Sample

Kafka

connector.read()
transform(data)
kafka_topic.send(data)

This code shows a connector reading data from an external system, transforming it, and sending it to a Kafka topic.

Process Table

Step	Action	Data State	Result
1	Connector reads data from external system	Raw data from system	Data captured by connector
2	Transform data if needed	Raw data	Data formatted for Kafka
3	Send data to Kafka topic	Formatted data	Data stored in Kafka topic
4	Kafka stores data	Data in topic	Data ready for consumers
5	Consumers read data	Data in topic	Data processed by consumers
6	End of process	No new data	Waiting for new data

💡 Process waits for new data after current batch is processed

Status Tracker

Variable	Start	After Step 1	After Step 2	After Step 3	After Step 4	Final
data	None	Raw data from external system	Transformed data	Sent to Kafka topic	Stored in Kafka topic	Consumed by clients

Key Moments - 2 Insights

Why does the connector transform data before sending it to Kafka?

Because external systems may have different data formats, the connector transforms data to match Kafka's expected format, as shown in step 2 of the execution_table.

What happens if the connector does not send data to Kafka?

If data is not sent to Kafka (step 3), the data won't be stored or available for consumers, so the integration fails to deliver data downstream.

Visual Quiz - 3 Questions

Look at the execution_table, what is the data state after step 2?

A Raw data from external system

B Data formatted for Kafka

C Data stored in Kafka topic

D Data processed by consumers

💡 Hint

Check the 'Data State' column in row for step 2 in execution_table

At which step does the connector send data to Kafka?

A Step 3

B Step 2

C Step 1

D Step 4

💡 Hint

Look at the 'Action' column in execution_table to find when data is sent to Kafka

If the external system sends no new data, what happens according to the exit_note?

A Connector retries sending old data

B Kafka deletes old data

C Process waits for new data

D Consumers stop working

💡 Hint

Refer to the exit_note in execution_table for process behavior when no new data arrives

Concept Snapshot

Connectors link external systems to Kafka.
They read data, transform it if needed,
and send it to Kafka topics.
Kafka stores data for consumers.
This enables smooth data integration.

Full Transcript

Connectors are tools that connect external systems to Kafka. They read data from these systems, sometimes change the data format to fit Kafka's needs, and then send the data into Kafka topics. Kafka stores this data so other programs, called consumers, can read and use it. This process helps different systems work together smoothly by sharing data through Kafka.