What is CDC with Kafka: Change Data Capture Explained

Change Data Capture (CDC) with Kafka is a method to track and stream database changes in real time using Kafka topics. It captures inserts, updates, and deletes from a database and sends them as events to Kafka for processing or syncing with other systems.

How It Works

Imagine you have a notebook where you write down every change you make to your bank account balance. CDC with Kafka works similarly by recording every change made to a database. Instead of checking the whole database repeatedly, it only notes what changed.

Kafka acts like a post office that delivers these change messages to different places that need them, such as analytics tools or other databases. This way, systems stay updated instantly without heavy data copying.

Technically, CDC tools read the database's transaction log or use triggers to detect changes, then send these changes as messages to Kafka topics. Consumers can then read these messages to update their own data stores or trigger actions.

Example

This example shows how to use Debezium, a popular CDC connector, with Kafka to capture changes from a MySQL database and stream them to a Kafka topic.

json

{
  "name": "mysql-connector",
  "config": {
    "connector.class": "io.debezium.connector.mysql.MySqlConnector",
    "database.hostname": "localhost",
    "database.port": "3306",
    "database.user": "debezium",
    "database.password": "dbz",
    "database.server.id": 184054,
    "database.server.name": "dbserver1",
    "database.include.list": "inventory",
    "table.include.list": "inventory.customers",
    "database.history.kafka.bootstrap.servers": "localhost:9092",
    "database.history.kafka.topic": "schema-changes.inventory"
  }
}

Output

Connector mysql-connector started and capturing changes from MySQL database 'inventory' on table 'customers'. Changes appear as JSON messages in Kafka topic 'dbserver1.inventory.customers'.

When to Use

Use CDC with Kafka when you need real-time data synchronization between databases or systems without heavy batch jobs. It is ideal for keeping data warehouses, caches, search indexes, or microservices up to date instantly.

For example, an e-commerce site can use CDC to update inventory and order status across multiple services as soon as a customer places an order. It also helps in auditing changes or replicating data to cloud platforms.

Key Points

• CDC captures only data changes, not full data snapshots.
• Kafka streams these changes as events to multiple consumers.
• It reduces load on source databases by avoiding full data reads.
• Popular CDC tools like Debezium integrate easily with Kafka.
• Useful for real-time analytics, syncing, and microservices communication.

Key Takeaways

CDC with Kafka streams database changes in real time as events.

It improves efficiency by sending only changes, not full data copies.

Debezium is a common tool to capture changes from databases into Kafka.

Use CDC for real-time syncing, analytics, and event-driven systems.

Kafka acts as a reliable message broker delivering change events to consumers.