Event sourcing pattern in Kafka - Commands & Configuration

Introduction

Event sourcing is a way to save all changes to data as a sequence of events. Instead of saving only the current state, it keeps every change so you can rebuild the state anytime. This helps track history and recover data easily.

When you want to keep a full history of changes for audit or debugging.

When you need to rebuild the current state from past events after a failure.

When multiple services need to react to changes in data asynchronously.

When you want to decouple your data storage from business logic.

When you want to implement complex workflows that depend on event sequences.

Config File - server.properties

server.properties

broker.id=1
listeners=PLAINTEXT://:9092
log.dirs=/tmp/kafka-logs
num.partitions=3
log.retention.hours=168
zookeeper.connect=localhost:2181

This is a basic Kafka broker configuration file.

broker.id: Unique ID for this Kafka broker.

listeners: Network address where Kafka listens for connections.

log.dirs: Directory to store Kafka logs (events).

num.partitions: Number of partitions per topic, enabling parallelism.

log.retention.hours: How long Kafka keeps event logs before deleting.

zookeeper.connect: Address of Zookeeper managing Kafka cluster metadata.

Commands

Create a Kafka topic named 'event-sourcing-topic' with 3 partitions to store event streams for event sourcing.

Terminal

kafka-topics.sh --create --topic event-sourcing-topic --bootstrap-server localhost:9092 --partitions 3 --replication-factor 1

Expected Output

Created topic event-sourcing-topic.

→

--topic - Name of the Kafka topic to create

→

--partitions - Number of partitions for parallelism

→

--replication-factor - Number of copies of data for fault tolerance

Start a producer to send events (messages) to the 'event-sourcing-topic'. Each message represents a change in the system.

Terminal

kafka-console-producer.sh --topic event-sourcing-topic --bootstrap-server localhost:9092

Expected Output

No output (command runs silently)

→

--topic - Topic to send messages to

→

--bootstrap-server - Kafka server address

Start a consumer to read all events from the beginning of the 'event-sourcing-topic'. This lets you rebuild the current state by processing all past events.

Terminal

kafka-console-consumer.sh --topic event-sourcing-topic --bootstrap-server localhost:9092 --from-beginning

Expected Output

event1

event2

event3

→

--topic - Topic to read messages from

→

--from-beginning - Read all messages from the start

→

--bootstrap-server - Kafka server address

Key Concept

If you remember nothing else from this pattern, remember: store every change as an event so you can rebuild the current state anytime by replaying those events.

Common Mistakes

Creating a topic with only one partition

Limits parallel processing and scalability of event streams.

Create topics with multiple partitions to allow parallel consumers.

Not consuming events from the beginning

You miss past events needed to rebuild the full state.

Use the --from-beginning flag to read all events from the start.

Not setting proper log retention

Events may be deleted too soon, losing history needed for event sourcing.

Configure log.retention.hours to keep events long enough for your use case.

Summary

Create a Kafka topic with multiple partitions to store event streams.

Use a producer to send events representing changes to the topic.

Use a consumer with --from-beginning to read all events and rebuild state.

Practice

1. What is the main purpose of the event sourcing pattern in Kafka?

easy

A. To save every change as an event for a full history

B. To delete old data to save storage space

C. To process only the latest state without history

D. To send events only to one consumer

Solution

1. Step 1: Understand event sourcing concept

   Event sourcing means saving every change as a separate event instead of just the current state.

2. Step 2: Relate to Kafka usage

   Kafka topics store these events so consumers can read the full history and rebuild state if needed.

3. Final Answer:

   To save every change as an event for a full history -> Option A

4. Quick Check:

   Event sourcing = full history saved as events [OK]

Hint: Event sourcing always stores all changes, not just current state [OK]

Common Mistakes:

• Thinking event sourcing deletes old data
• Believing it only keeps latest state
• Assuming events go to only one consumer

2. Which Kafka component is primarily used to store events in event sourcing?

easy

A. Kafka Topic

B. Kafka Producer

C. Kafka Consumer

D. Kafka Broker

Solution

1. Step 1: Identify Kafka components roles

   Kafka Producer sends data, Consumer reads data, Broker manages servers, Topic stores data.

2. Step 2: Match storage role

   Events are stored in Kafka Topics, which keep the event log for consumers.

3. Final Answer:

   Kafka Topic -> Option A

4. Quick Check:

   Events stored in Kafka Topic [OK]

Hint: Events are stored in topics, not producers or consumers [OK]

Common Mistakes:

• Confusing producer with storage
• Thinking consumer stores events
• Mixing broker with event storage

3. Given this Kafka event stream, what will the consumer state be after processing all events?

events = [
  {"type": "create", "id": 1, "value": 10},
  {"type": "update", "id": 1, "value": 20},
  {"type": "create", "id": 2, "value": 5},
  {"type": "delete", "id": 1}
]

state = {}
for event in events:
    if event["type"] == "create" or event["type"] == "update":
        state[event["id"]] = event["value"]
    elif event["type"] == "delete":
        state.pop(event["id"], None)

print(state)

medium

A. {}

B. {"1": 20, "2": 5}

C. {"1": 10, "2": 5}

D. {"2": 5}

Solution

1. Step 1: Process each event in order

   First create id 1 with value 10, then update id 1 to 20, create id 2 with 5, then delete id 1.

2. Step 2: Determine final state

   After deletion, id 1 is removed, only id 2 with value 5 remains.

3. Final Answer:

   {"2": 5} -> Option D

4. Quick Check:

   Delete removes id 1, only id 2 left [OK]

Hint: Track events step-by-step to update state correctly [OK]

Common Mistakes:

• Forgetting the delete event removes id 1
• Confusing update with create
• Assuming old values remain after update

4. Identify the bug in this Kafka event consumer code snippet for event sourcing:

state = {}
for event in events:
    if event["type"] == "create":
        state[event["id"]] = event["value"]
    elif event["type"] == "update":
        if event["id"] in state:
            state[event["id"]] = event["value"]
    elif event["type"] == "delete":
        del state[event["id"]]

medium

A. Missing else clause for unknown event types

B. Using del without checking if id exists causes KeyError

C. Create should not overwrite existing id

D. Update should create new id if not present

Solution

1. Step 1: Analyze delete operation

   Using del without checking if the key exists can cause a KeyError if id is missing.

2. Step 2: Check other conditions

   Update correctly checks if id exists before updating; create overwrites which is allowed; else clause is optional.

3. Final Answer:

   Using del without checking if id exists causes KeyError -> Option B

4. Quick Check:

   del without check = KeyError risk [OK]

Hint: Always check key exists before deleting from dict [OK]

Common Mistakes:

• Ignoring KeyError on delete
• Thinking update must create new id
• Believing create cannot overwrite

5. You want to rebuild the current state from a Kafka topic storing events using event sourcing. Which approach is best to handle a large number of events efficiently?

hard

A. Read all events from the beginning every time to rebuild state

B. Only read the latest event for each entity from the topic

C. Use snapshots periodically and then replay only recent events

D. Delete old events from the topic to reduce size

Solution

1. Step 1: Understand rebuilding state from events

   Replaying all events from the start is slow for large data; deleting events loses history.

2. Step 2: Use snapshots to optimize

   Snapshots save a recent state, so only new events after snapshot need replay, improving efficiency.

3. Final Answer:

   Use snapshots periodically and then replay only recent events -> Option C

4. Quick Check:

   Snapshots + replay recent events = efficient rebuild [OK]

Hint: Snapshots speed up rebuilding by avoiding full replay [OK]

Common Mistakes:

• Replaying all events every time (slow)
• Deleting events loses full history
• Reading only latest event misses changes