Common connectors (JDBC, S3, Elasticsearch) in Kafka - Cheat Sheet & Quick Revision

Recall & Review

beginner

What is the purpose of the JDBC connector in Kafka?

The JDBC connector allows Kafka to connect to relational databases. It can read data from databases into Kafka topics or write data from Kafka topics back into databases.

beginner

How does the Kafka S3 connector help in data handling?

The Kafka S3 connector moves data between Kafka topics and Amazon S3 storage. It can save Kafka data as files in S3 or read files from S3 into Kafka.

beginner

What role does the Elasticsearch connector play in Kafka?

The Elasticsearch connector sends data from Kafka topics to an Elasticsearch cluster. This helps in searching and analyzing data quickly.

beginner

Which Kafka connector type is used to import data into Kafka topics?

Source connectors are used to import data into Kafka topics from external systems like databases or storage services.

beginner

What is a sink connector in Kafka?

A sink connector takes data from Kafka topics and sends it to external systems like databases, S3, or Elasticsearch for storage or processing.

Which Kafka connector would you use to export data from Kafka to a relational database?

A S3 source connector

B JDBC sink connector

C Elasticsearch source connector

D Kafka Streams

The JDBC sink connector writes data from Kafka topics into relational databases.

What does the Kafka S3 connector primarily do?

A Transfers data between Kafka and Amazon S3

B Indexes data in Elasticsearch

C Connects Kafka to relational databases

D Processes streams inside Kafka

The S3 connector moves data between Kafka topics and Amazon S3 storage.

Which connector type reads data from external systems into Kafka?

A Sink connector

B Stream connector

C Processor connector

D Source connector

Source connectors import data into Kafka topics from outside systems.

What is the main use of the Elasticsearch sink connector in Kafka?

A To send Kafka data to Elasticsearch for search and analysis

B To read data from Elasticsearch into Kafka

C To store Kafka data in Amazon S3

D To connect Kafka to relational databases

The Elasticsearch sink connector sends Kafka topic data to Elasticsearch clusters.

Which of the following is NOT a typical Kafka connector?

A Elasticsearch connector

B S3 connector

C FTP connector

D JDBC connector

FTP connector is not a common Kafka connector; JDBC, S3, and Elasticsearch connectors are standard.

Explain how Kafka connectors help integrate Kafka with databases, cloud storage, and search engines.

Think about how data moves between Kafka and other systems.

Describe the difference between source and sink connectors in Kafka with examples.

Focus on the direction of data flow.

Practice

1. What is the main purpose of a JDBC connector in Kafka?

easy

A. To monitor Kafka cluster health

B. To store Kafka data in cloud object storage

C. To enable full-text search on Kafka data

D. To move data between Kafka and relational databases

Solution

1. Step 1: Understand JDBC connector role

   JDBC connectors are designed to connect Kafka with relational databases using JDBC protocol.

2. Step 2: Match purpose with options

   To move data between Kafka and relational databases correctly states moving data between Kafka and databases, which is the JDBC connector's main use.

3. Final Answer:

   To move data between Kafka and relational databases -> Option D

4. Quick Check:

   JDBC connector = database integration [OK]

Hint: JDBC always relates to databases and Kafka [OK]

Common Mistakes:

• Confusing JDBC with S3 or Elasticsearch connectors
• Thinking JDBC stores data in cloud storage
• Assuming JDBC is for search functionality

2. Which of the following is the correct configuration property to specify the S3 bucket name in a Kafka S3 sink connector?

easy

A. s3.bucket.name

B. bucket.name.s3

C. s3.bucket

D. bucket.s3.name

Solution

1. Step 1: Recall S3 connector config syntax

   The standard property to specify the bucket is s3.bucket.name.

2. Step 2: Compare options with standard property

   Only s3.bucket.name matches the correct property name exactly.

3. Final Answer:

   s3.bucket.name -> Option A

4. Quick Check:

   Bucket name property = s3.bucket.name [OK]

Hint: S3 config keys start with 's3.' prefix [OK]

Common Mistakes:

• Swapping order of words in property name
• Using incorrect prefixes like 'bucket.s3'
• Missing the '.name' suffix

3. Given the following Kafka Elasticsearch sink connector config snippet:

{
  "connector.class": "io.confluent.connect.elasticsearch.ElasticsearchSinkConnector",
  "topics": "orders",
  "connection.url": "http://localhost:9200",
  "type.name": "_doc",
  "key.ignore": "false"
}

What will happen when a message with key 'order123' and value '{"item":"book"}' is sent to topic 'orders'?

medium

A. The message will be stored in Kafka only, not sent to Elasticsearch

B. The document will be indexed with a random ID ignoring the key

C. The document will be indexed in Elasticsearch with ID 'order123'

D. The connector will fail due to missing authentication

Solution

1. Step 1: Analyze key.ignore setting

   Since "key.ignore" is set to "false", the connector uses the Kafka message key as the Elasticsearch document ID.

2. Step 2: Understand indexing behavior

   The message with key 'order123' will be indexed in Elasticsearch with that key as the document ID.

3. Final Answer:

   The document will be indexed in Elasticsearch with ID 'order123' -> Option C

4. Quick Check:

   key.ignore=false means use key as ID [OK]

Hint: key.ignore=false means use Kafka key as ES document ID [OK]

Common Mistakes:

• Assuming key is ignored by default
• Thinking connector fails without auth (not always true)
• Confusing Kafka storage with Elasticsearch indexing

4. You configured a Kafka JDBC source connector but it is not importing any data. Which of the following is the most likely cause?

medium

A. The Kafka topic name is missing in the connector config

B. The JDBC URL is incorrect or unreachable

C. The connector class is set to ElasticsearchSinkConnector

D. The S3 bucket name is not specified

Solution

1. Step 1: Check connectivity issues

   If the JDBC URL is wrong or unreachable, the connector cannot read data from the database.

2. Step 2: Evaluate other options

   The Kafka topic name is missing in the connector config would cause errors but usually the connector won't start; The connector class is set to ElasticsearchSinkConnector is wrong connector type; The S3 bucket name is not specified is unrelated to JDBC source.

3. Final Answer:

   The JDBC URL is incorrect or unreachable -> Option B

4. Quick Check:

   Bad JDBC URL = no data import [OK]

Hint: Always verify JDBC URL connectivity first [OK]

Common Mistakes:

• Confusing connector class for source vs sink
• Ignoring missing topic config errors
• Checking unrelated S3 settings for JDBC issues

5. You want to archive Kafka topic data to S3 daily and also enable fast search on the same data using Elasticsearch. Which connector setup is the best approach?

hard

A. Use an S3 sink connector for archiving and an Elasticsearch sink connector for search, both consuming from the same Kafka topic

B. Use only an S3 sink connector and query S3 directly for search

C. Use a JDBC sink connector to store data in a database, then export to S3 and Elasticsearch from there

D. Use an Elasticsearch source connector to send data to S3

Solution

1. Step 1: Understand connector roles

   S3 sink connector archives data to cloud storage; Elasticsearch sink connector indexes data for search.

2. Step 2: Choose correct architecture

   Using both sink connectors on the same Kafka topic allows parallel archiving and search capabilities efficiently.

3. Final Answer:

   Use an S3 sink connector for archiving and an Elasticsearch sink connector for search, both consuming from the same Kafka topic -> Option A

4. Quick Check:

   Separate sinks for archive and search = best practice [OK]

Hint: Use separate sink connectors for storage and search [OK]

Common Mistakes:

• Trying to search data directly in S3 without indexing
• Using JDBC sink unnecessarily for this use case
• Confusing source and sink connector roles