Common connectors (JDBC, S3, Elasticsearch) in Kafka - Deep Dive

Overview - Common connectors (JDBC, S3, Elasticsearch)

What is it?

Common connectors are tools that help Kafka connect with other systems like databases, storage, or search engines. JDBC connector links Kafka to databases using standard database protocols. S3 connector moves data between Kafka and Amazon S3 storage. Elasticsearch connector sends Kafka data to Elasticsearch for fast searching. These connectors make data flow smooth and automatic between Kafka and other platforms.

Why it matters

Without these connectors, moving data between Kafka and other systems would be slow, manual, and error-prone. They solve the problem of integrating different technologies easily, so data can be stored, searched, or analyzed without extra work. This helps businesses react faster and keep data organized across many tools.

Where it fits

Before learning connectors, you should understand Kafka basics like topics and producers/consumers. After connectors, you can explore Kafka Streams for data processing or advanced Kafka security and scaling. Connectors are a bridge between Kafka and the outside world.

Mental Model

Core Idea

Connectors are bridges that automatically move data between Kafka and other systems without manual coding.

Think of it like...

Imagine a conveyor belt that carries packages from one factory to another without workers needing to carry them by hand. Connectors are like those conveyor belts for data.

┌─────────────┐      ┌─────────────┐      ┌─────────────┐
│  Database   │◄─────│ JDBC Source │      │ JDBC Sink   │────►│  Database   │
└─────────────┘      └─────────────┘      └─────────────┘
       ▲                    ▲                    ▲
       │                    │                    │
       │                    │                    │
┌─────────────┐      ┌─────────────┐      ┌─────────────┐
│   Kafka     │─────►│  S3 Sink    │      │ Elasticsearch│
└─────────────┘      └─────────────┘      └─────────────┘

Build-Up - 7 Steps

1

Foundation

What is a Kafka Connector

Concept: Introduces the basic idea of a connector as a tool to link Kafka with other systems.

Kafka connectors are pre-built or custom tools that move data into or out of Kafka automatically. They save you from writing code to connect Kafka with databases, storage, or search engines.

Result

You understand that connectors automate data movement between Kafka and other systems.

Knowing connectors exist helps you see Kafka as part of a bigger data ecosystem, not just a messaging tool.

2

Foundation

Types of Connectors: Source and Sink

Concept: Explains the two main connector roles: source (bringing data into Kafka) and sink (sending data out).

Source connectors read data from external systems and write it into Kafka topics. Sink connectors read data from Kafka topics and write it to external systems like databases or storage.

Result

You can classify connectors by their direction of data flow.

Understanding source vs sink clarifies how data moves through Kafka pipelines.

3

Intermediate

JDBC Connector for Databases

🤔Before reading on: do you think JDBC connectors can both read from and write to databases? Commit to your answer.

Concept: Introduces the JDBC connector that connects Kafka with relational databases using standard database protocols.

The JDBC source connector pulls data from databases into Kafka topics by running SQL queries. The JDBC sink connector writes Kafka topic data back into databases by inserting or updating rows. It supports many databases like MySQL, PostgreSQL, and Oracle.

Result

You can move data between Kafka and databases automatically using JDBC connectors.

Knowing JDBC connectors handle both directions helps design flexible data pipelines involving databases.

4

Intermediate

S3 Connector for Cloud Storage

🤔Before reading on: do you think S3 connectors only send data to S3 or can they also read from it? Commit to your answer.

Concept: Explains how S3 connectors move data between Kafka and Amazon S3 cloud storage.

The S3 sink connector writes Kafka topic data as files into S3 buckets, useful for backups or batch processing. The S3 source connector reads files from S3 and sends them into Kafka topics. This helps integrate Kafka with cloud storage for long-term data retention.

Result

You can automatically archive Kafka data to S3 or load data from S3 into Kafka.

Understanding S3 connectors enables building scalable, cloud-based data lakes with Kafka.

5

Intermediate

Elasticsearch Connector for Search

🤔Before reading on: do you think Elasticsearch connectors transform data or just move it? Commit to your answer.

Concept: Describes the Elasticsearch sink connector that sends Kafka data to Elasticsearch for fast searching and analytics.

The Elasticsearch sink connector reads Kafka topic data and indexes it into Elasticsearch. It can transform data formats and map fields to Elasticsearch indexes. This allows real-time search and visualization of streaming data.

Result

You can make Kafka data searchable and analyzable in Elasticsearch automatically.

Knowing how connectors transform and index data helps build real-time search applications.

6

Advanced

Configuring and Managing Connectors

🤔Before reading on: do you think connectors run inside Kafka brokers or separately? Commit to your answer.

Concept: Explains how connectors are configured, deployed, and managed using Kafka Connect framework.

Connectors run in Kafka Connect workers, which can be standalone or distributed. You configure connectors with JSON files specifying connection details, topics, and data formats. Kafka Connect handles scaling, fault tolerance, and monitoring of connectors.

Result

You can deploy and manage connectors reliably in production environments.

Understanding Kafka Connect architecture is key to running connectors at scale and with high availability.

7

Expert

Advanced Connector Tuning and Error Handling

🤔Before reading on: do you think connectors automatically retry on all errors or do some errors require manual fixes? Commit to your answer.

Concept: Covers advanced topics like tuning performance, handling data errors, and customizing connector behavior.

Connectors can be tuned with batch sizes, retry policies, and error handling strategies. Some errors like schema mismatches require manual intervention. Custom converters and transformations can be added to connectors for complex data needs. Monitoring connector metrics helps detect and fix issues early.

Result

You can optimize connectors for performance and reliability in complex production scenarios.

Knowing how to tune and troubleshoot connectors prevents data loss and downtime in real systems.

Under the Hood

Kafka Connect runs connectors as separate worker processes that poll source systems or consume Kafka topics. Connectors use APIs like JDBC to query databases or SDKs to interact with S3 and Elasticsearch. Data is converted between Kafka's internal format and external system formats using converters and transformations. Kafka Connect manages offsets to track progress and ensure exactly-once or at-least-once delivery.

Why designed this way?

Kafka Connect was designed to simplify integration by standardizing how connectors run and are managed. Running connectors separately avoids overloading Kafka brokers and allows scaling connectors independently. Using pluggable converters and transformations makes connectors flexible for many data formats. This design balances ease of use, scalability, and reliability.

┌───────────────┐       ┌───────────────┐       ┌───────────────┐
│ External     │◄──────│ Source        │       │ Sink          │─────►│ External     │
│ System (DB,  │       │ Connector     │       │ Connector     │     │ System (S3,  │
│ S3, ES)      │       └───────┬───────┘       └───────┬───────┘     │ ES)          │
└───────────────┘               │                       │             └───────────────┘
                                │                       │
                          ┌─────▼─────┐           ┌─────▼─────┐
                          │ Kafka    │           │ Kafka    │
                          │ Topics   │           │ Topics   │
                          └──────────┘           └──────────┘

Myth Busters - 4 Common Misconceptions

Quick: Do you think Kafka connectors can only send data out of Kafka, not bring data in? Commit yes or no.

Common Belief: Connectors only move data from Kafka to other systems, not the other way around.

Reality: Connectors can be both source (bringing data into Kafka) and sink (sending data out).

Why it matters: Believing this limits your design options and may cause you to build unnecessary custom code to ingest data.

Quick: Do you think connectors run inside Kafka brokers? Commit yes or no.

Common Belief: Connectors run inside Kafka brokers themselves.

Reality: Connectors run in separate Kafka Connect worker processes, not inside brokers.

Why it matters: Misunderstanding this can lead to wrong deployment setups and performance issues.

Quick: Do you think connectors automatically fix all data errors without manual help? Commit yes or no.

Common Belief: Connectors handle all data errors automatically and never fail.

Reality: Some errors like schema mismatches require manual fixes or configuration changes.

Why it matters: Assuming automatic error handling can cause unnoticed data loss or pipeline failures.

Quick: Do you think S3 connectors only write data to S3 and cannot read from it? Commit yes or no.

Common Belief: S3 connectors only send data from Kafka to S3, not the other way.

Reality: S3 connectors can also read data from S3 and send it into Kafka topics.

Why it matters: Missing this limits your ability to build flexible data pipelines involving cloud storage.

Expert Zone

1

Connectors can be chained with Kafka Streams or KSQL for complex data transformations before reaching the sink.

2

Offset management in Kafka Connect ensures exactly-once delivery but requires careful configuration to avoid duplicates.

3

Custom Single Message Transforms (SMTs) allow lightweight data changes inside connectors without full stream processing.

When NOT to use

Avoid connectors when you need ultra-low latency or complex transformations; use Kafka Streams or custom consumers instead. Also, if your external system lacks a connector, consider writing a custom producer/consumer.

Production Patterns

In production, connectors run in distributed mode for fault tolerance. Teams use monitoring tools like Confluent Control Center or Prometheus to track connector health. Connectors are often combined with schema registries to enforce data formats.

Connections

ETL (Extract, Transform, Load)

Connectors automate the Extract and Load parts of ETL pipelines.

Understanding connectors clarifies how modern data pipelines automate data movement without manual scripts.

Microservices Architecture

Connectors enable event-driven communication between microservices and external systems.

Knowing connectors helps design loosely coupled systems that share data reliably.

Supply Chain Logistics

Connectors are like automated shipping routes moving goods between warehouses (systems).

Seeing data flow as logistics helps grasp the importance of reliable, automated connectors.

Common Pitfalls

#1 Trying to run connectors inside Kafka brokers directly.

Wrong approach:

Starting connectors as part of Kafka broker process or installing connector plugins only on brokers.

Correct approach:

Run connectors in Kafka Connect worker processes separate from brokers, configured via REST API or config files.

Root cause:

Misunderstanding Kafka Connect architecture and deployment model.

#2 Using default connector settings without tuning for large data volumes.

Wrong approach:

Deploying connectors with default batch sizes and retry policies on heavy workloads.

Correct approach:

Adjust batch sizes, retries, and parallelism in connector configs to match data volume and system capacity.

Root cause:

Assuming default configs are optimal for all scenarios.

#3 Ignoring schema compatibility and data format mismatches.

Wrong approach:

Sending data with incompatible schemas to sink connectors without validation.

Correct approach:

Use schema registry and validate schemas before sending data to connectors.

Root cause:

Underestimating the importance of data format consistency.

Key Takeaways

Kafka connectors automate data movement between Kafka and external systems, saving manual coding.

Connectors come in two types: source connectors bring data into Kafka, sink connectors send data out.

JDBC, S3, and Elasticsearch connectors are common tools to integrate databases, cloud storage, and search engines with Kafka.

Kafka Connect framework runs connectors separately for scalability and reliability, managing configuration and offsets.

Advanced tuning and error handling are essential for stable, high-performance connector operation in production.

Practice

1. What is the main purpose of a JDBC connector in Kafka?

easy

A. To monitor Kafka cluster health

B. To store Kafka data in cloud object storage

C. To enable full-text search on Kafka data

D. To move data between Kafka and relational databases

Solution

1. Step 1: Understand JDBC connector role

   JDBC connectors are designed to connect Kafka with relational databases using JDBC protocol.

2. Step 2: Match purpose with options

   To move data between Kafka and relational databases correctly states moving data between Kafka and databases, which is the JDBC connector's main use.

3. Final Answer:

   To move data between Kafka and relational databases -> Option D

4. Quick Check:

   JDBC connector = database integration [OK]

Hint: JDBC always relates to databases and Kafka [OK]

Common Mistakes:

• Confusing JDBC with S3 or Elasticsearch connectors
• Thinking JDBC stores data in cloud storage
• Assuming JDBC is for search functionality

2. Which of the following is the correct configuration property to specify the S3 bucket name in a Kafka S3 sink connector?

easy

A. s3.bucket.name

B. bucket.name.s3

C. s3.bucket

D. bucket.s3.name

Solution

1. Step 1: Recall S3 connector config syntax

   The standard property to specify the bucket is s3.bucket.name.

2. Step 2: Compare options with standard property

   Only s3.bucket.name matches the correct property name exactly.

3. Final Answer:

   s3.bucket.name -> Option A

4. Quick Check:

   Bucket name property = s3.bucket.name [OK]

Hint: S3 config keys start with 's3.' prefix [OK]

Common Mistakes:

• Swapping order of words in property name
• Using incorrect prefixes like 'bucket.s3'
• Missing the '.name' suffix

3. Given the following Kafka Elasticsearch sink connector config snippet:

{
  "connector.class": "io.confluent.connect.elasticsearch.ElasticsearchSinkConnector",
  "topics": "orders",
  "connection.url": "http://localhost:9200",
  "type.name": "_doc",
  "key.ignore": "false"
}

What will happen when a message with key 'order123' and value '{"item":"book"}' is sent to topic 'orders'?

medium

A. The message will be stored in Kafka only, not sent to Elasticsearch

B. The document will be indexed with a random ID ignoring the key

C. The document will be indexed in Elasticsearch with ID 'order123'

D. The connector will fail due to missing authentication

Solution

1. Step 1: Analyze key.ignore setting

   Since "key.ignore" is set to "false", the connector uses the Kafka message key as the Elasticsearch document ID.

2. Step 2: Understand indexing behavior

   The message with key 'order123' will be indexed in Elasticsearch with that key as the document ID.

3. Final Answer:

   The document will be indexed in Elasticsearch with ID 'order123' -> Option C

4. Quick Check:

   key.ignore=false means use key as ID [OK]

Hint: key.ignore=false means use Kafka key as ES document ID [OK]

Common Mistakes:

• Assuming key is ignored by default
• Thinking connector fails without auth (not always true)
• Confusing Kafka storage with Elasticsearch indexing

4. You configured a Kafka JDBC source connector but it is not importing any data. Which of the following is the most likely cause?

medium

A. The Kafka topic name is missing in the connector config

B. The JDBC URL is incorrect or unreachable

C. The connector class is set to ElasticsearchSinkConnector

D. The S3 bucket name is not specified

Solution

1. Step 1: Check connectivity issues

   If the JDBC URL is wrong or unreachable, the connector cannot read data from the database.

2. Step 2: Evaluate other options

   The Kafka topic name is missing in the connector config would cause errors but usually the connector won't start; The connector class is set to ElasticsearchSinkConnector is wrong connector type; The S3 bucket name is not specified is unrelated to JDBC source.

3. Final Answer:

   The JDBC URL is incorrect or unreachable -> Option B

4. Quick Check:

   Bad JDBC URL = no data import [OK]

Hint: Always verify JDBC URL connectivity first [OK]

Common Mistakes:

• Confusing connector class for source vs sink
• Ignoring missing topic config errors
• Checking unrelated S3 settings for JDBC issues

5. You want to archive Kafka topic data to S3 daily and also enable fast search on the same data using Elasticsearch. Which connector setup is the best approach?

hard

A. Use an S3 sink connector for archiving and an Elasticsearch sink connector for search, both consuming from the same Kafka topic

B. Use only an S3 sink connector and query S3 directly for search

C. Use a JDBC sink connector to store data in a database, then export to S3 and Elasticsearch from there

D. Use an Elasticsearch source connector to send data to S3

Solution

1. Step 1: Understand connector roles

   S3 sink connector archives data to cloud storage; Elasticsearch sink connector indexes data for search.

2. Step 2: Choose correct architecture

   Using both sink connectors on the same Kafka topic allows parallel archiving and search capabilities efficiently.

3. Final Answer:

   Use an S3 sink connector for archiving and an Elasticsearch sink connector for search, both consuming from the same Kafka topic -> Option A

4. Quick Check:

   Separate sinks for archive and search = best practice [OK]

Hint: Use separate sink connectors for storage and search [OK]

Common Mistakes:

• Trying to search data directly in S3 without indexing
• Using JDBC sink unnecessarily for this use case
• Confusing source and sink connector roles