Why connectors integrate external systems in Kafka - See It in Action

Why connectors integrate external systems

📖 Scenario: Imagine you have different apps and databases that need to share information smoothly. Connectors help these separate systems talk to each other without you writing complex code.

🎯 Goal: Learn why connectors are used to link external systems with Kafka, making data flow easy and automatic.

📋 What You'll Learn

Create a dictionary called external_systems with these exact entries: 'Database': 'Stores data', 'CRM': 'Manages customer info', 'WebApp': 'User interface'

Create a variable called connector_purpose and set it to the string 'Integrate external systems with Kafka'

Create a list called reasons with these exact strings: 'Automate data flow', 'Reduce manual coding', 'Ensure data consistency'

Print the connector_purpose and each reason from reasons on separate lines

💡 Why This Matters

🌍 Real World

Connectors are used in real companies to connect databases, apps, and services so data flows automatically without manual work.

💼 Career

Understanding connectors is important for roles in data engineering, system integration, and software development where smooth data exchange is needed.

1

Create the external systems dictionary

Create a dictionary called external_systems with these exact entries: 'Database': 'Stores data', 'CRM': 'Manages customer info', 'WebApp': 'User interface'

Kafka

# Create the external_systems dictionary with given entries
# Your code here

Hint

Use curly braces {} to create a dictionary with keys and values.

2

Add the connector purpose variable

Create a variable called connector_purpose and set it to the string 'Integrate external systems with Kafka'

Kafka

external_systems = {'Database': 'Stores data', 'CRM': 'Manages customer info', 'WebApp': 'User interface'}
# Create connector_purpose variable with the given string
# Your code here

Hint

Use an equals sign = to assign the string to the variable.

3

Create the reasons list

Create a list called reasons with these exact strings: 'Automate data flow', 'Reduce manual coding', 'Ensure data consistency'

Kafka

external_systems = {'Database': 'Stores data', 'CRM': 'Manages customer info', 'WebApp': 'User interface'}
connector_purpose = 'Integrate external systems with Kafka'
# Create reasons list with the given strings
# Your code here

Hint

Use square brackets [] to create a list of strings.

4

Print the connector purpose and reasons

Print the variable connector_purpose and then print each item in the list reasons on separate lines using a for loop with variable reason

Kafka

external_systems = {'Database': 'Stores data', 'CRM': 'Manages customer info', 'WebApp': 'User interface'}
connector_purpose = 'Integrate external systems with Kafka'
reasons = ['Automate data flow', 'Reduce manual coding', 'Ensure data consistency']
# Print connector_purpose and each reason on separate lines
# Your code here

Hint

Use print() for the variable, then a for loop to print each reason.

Practice

1. Why do Kafka connectors integrate external systems?

easy

A. To automatically move data between Kafka and other systems

B. To create new Kafka topics manually

C. To write custom Kafka broker code

D. To monitor Kafka cluster health

Solution

1. Step 1: Understand the role of connectors

   Connectors are designed to link Kafka with external systems to automate data movement.

2. Step 2: Identify the main purpose

   The main purpose is to move data automatically, not to create topics or monitor health.

3. Final Answer:

   To automatically move data between Kafka and other systems -> Option A

4. Quick Check:

   Connectors move data automatically = A [OK]

Hint: Connectors move data automatically between systems [OK]

Common Mistakes:

• Thinking connectors create Kafka topics
• Confusing connectors with monitoring tools
• Assuming connectors require manual data transfer

2. Which of the following is the correct way to specify a connector configuration file in Kafka Connect?

easy

A. connect-standalone.sh config/connect-standalone.properties config/my-connector.properties

B. kafka-topics.sh --create --config my-connector.properties

C. kafka-console-producer.sh --broker-list localhost:9092 --topic my-connector.properties

D. connect-distributed.sh --config my-connector.properties

Solution

1. Step 1: Identify the correct command for standalone mode

   The command connect-standalone.sh is used with two config files: worker and connector.

2. Step 2: Check the options

   connect-standalone.sh config/connect-standalone.properties config/my-connector.properties correctly shows the worker config and connector config files together.

3. Final Answer:

   connect-standalone.sh config/connect-standalone.properties config/my-connector.properties -> Option A

4. Quick Check:

   Standalone mode uses two config files = A [OK]

Hint: Standalone mode needs worker and connector config files [OK]

Common Mistakes:

• Using topic commands instead of connector commands
• Passing only one config file
• Confusing distributed and standalone commands

3. Given this connector configuration snippet:

{
  "name": "my-source-connector",
  "config": {
    "connector.class": "FileStreamSource",
    "tasks.max": "1",
    "file": "/tmp/input.txt",
    "topic": "test-topic"
  }
}

What will this connector do when started?

medium

A. Create a new Kafka topic named '/tmp/input.txt'

B. Read data from /tmp/input.txt and send it to Kafka topic 'test-topic'

C. Write data from Kafka topic 'test-topic' to /tmp/input.txt

D. Monitor the file /tmp/input.txt for changes but do not send data

Solution

1. Step 1: Identify connector type and direction

   The connector class is FileStreamSource, which reads from a file and sends data to Kafka.

2. Step 2: Understand configuration parameters

   The file parameter points to the input file, and topic specifies the Kafka topic to send data to.

3. Final Answer:

   Read data from /tmp/input.txt and send it to Kafka topic 'test-topic' -> Option B

4. Quick Check:

   FileStreamSource reads file to Kafka = C [OK]

Hint: FileStreamSource reads file and sends to Kafka topic [OK]

Common Mistakes:

• Confusing source with sink connectors
• Thinking it writes to the file instead of reading
• Assuming it creates topics automatically

4. You wrote this sink connector config:

{
  "name": "my-sink-connector",
  "config": {
    "connector.class": "FileStreamSink",
    "tasks.max": "1",
    "file": "/tmp/output.txt",
    "topic": "input-topic"
  }
}

But no data appears in /tmp/output.txt after running. What is the likely problem?

medium

A. The connector class name is incorrect; it should be FileStreamSource

B. The file path /tmp/output.txt is not writable by the connector

C. The topic name 'input-topic' does not exist or has no data

D. The tasks.max value must be greater than 1

Solution

1. Step 1: Check connector class correctness

   The class FileStreamSink is correct for writing Kafka data to a file.

2. Step 2: Verify topic existence and data flow

   If the topic input-topic does not exist or has no data, no output will be written.

3. Final Answer:

   The topic name 'input-topic' does not exist or has no data -> Option C

4. Quick Check:

   Sink needs data in topic to write = B [OK]

Hint: Check topic exists and has data for sink connectors [OK]

Common Mistakes:

• Assuming connector class is wrong when it is correct
• Ignoring topic data availability
• Thinking tasks.max affects data flow directly

5. You want to integrate a database with Kafka using connectors. Which approach best explains why connectors are used for this integration?

hard

A. Connectors only work with file systems, not databases

B. Connectors replace the database with Kafka topics

C. Connectors require manual polling of the database by the user

D. Connectors automate data transfer between the database and Kafka without custom code

Solution

1. Step 1: Understand connector purpose for databases

   Connectors automate moving data between Kafka and external systems like databases.

2. Step 2: Eliminate incorrect options

   Connectors do not replace databases, do not require manual polling, and support databases, not just files.

3. Final Answer:

   Connectors automate data transfer between the database and Kafka without custom code -> Option D

4. Quick Check:

   Connectors automate integration without manual code = D [OK]

Hint: Connectors automate data flow without manual coding [OK]

Common Mistakes:

• Thinking connectors replace databases
• Assuming manual polling is needed
• Believing connectors only work with files