Change stream events (insert, update, delete) in MongoDB - Deep Dive

Overview - Change stream events (insert, update, delete)

What is it?

Change stream events in MongoDB let you watch for changes in your database in real time. These events include inserts (new data added), updates (existing data changed), and deletes (data removed). This helps applications react instantly when data changes without constantly checking the database. It works by opening a stream that sends notifications whenever these changes happen.

Why it matters

Without change streams, applications would need to repeatedly ask the database if anything changed, wasting time and resources. Change streams solve this by pushing updates only when something changes, making apps faster and more efficient. This is crucial for real-time features like notifications, live dashboards, or syncing data across systems.

Where it fits

Before learning change streams, you should understand basic MongoDB operations like insert, update, and delete. After mastering change streams, you can explore advanced topics like aggregation pipelines, real-time analytics, and building reactive applications that respond instantly to data changes.

Mental Model

Core Idea

Change streams are like a live news feed from your database that tells you immediately when data is added, changed, or removed.

Think of it like...

Imagine a security camera watching a store entrance. Instead of checking the store every hour, you get an alert whenever someone enters, leaves, or moves something. Change streams work the same way for your database.

┌───────────────────────────────┐
│        MongoDB Database       │
│ ┌─────────────┐ ┌───────────┐ │
│ │ Insert      │ │ Update    │ │
│ └─────────────┘ └───────────┘ │
│       │             │         │
│       ▼             ▼         │
│  ┌───────────────────────┐    │
│  │   Change Stream       │    │
│  │  (Event Listener)     │    │
│  └───────────────────────┘    │
│           │                   │
│           ▼                   │
│   Application reacts here     │
└───────────────────────────────┘

Build-Up - 8 Steps

1

Foundation

Basic MongoDB Data Changes

Concept: Learn what insert, update, and delete operations do in MongoDB.

Insert adds new documents to a collection. Update changes fields in existing documents. Delete removes documents from a collection. These are the core ways data changes in MongoDB.

Result

You can add, modify, and remove data in your database collections.

Understanding these basic operations is essential because change streams report on exactly these types of changes.

2

Foundation

What Are Change Streams?

Concept: Change streams provide a way to watch for data changes in real time.

Instead of asking the database repeatedly if data changed, change streams let you open a continuous feed that sends events when inserts, updates, or deletes happen.

Result

You get a live feed of database changes as they occur.

Knowing that change streams push updates instantly helps you build reactive applications that respond immediately to data changes.

3

Intermediate

Listening to Insert Events

🤔Before reading on: do you think insert events include the full new document or just the changed fields? Commit to your answer.

Concept: Insert events include the entire new document added to the collection.

When a new document is inserted, the change stream event contains the full document under the 'fullDocument' field. This lets you see exactly what was added.

Result

You receive the complete new document data as soon as it is inserted.

Understanding that insert events provide full documents helps you process new data without extra database queries.

4

Intermediate

Understanding Update Events

🤔Before reading on: do you think update events show the whole document or only the changed parts? Commit to your answer.

Concept: Update events show only the changed fields by default, not the entire document.

When a document is updated, the change event includes an 'updateDescription' field that lists which fields changed and how. The full document is not included unless you request it explicitly.

Result

You get a summary of changes without the full document, saving bandwidth and processing.

Knowing update events show only changes helps you optimize how you handle updates and decide when to fetch full documents.

5

Intermediate

Detecting Delete Events

Concept: Delete events notify you when documents are removed from the collection.

When a document is deleted, the change stream event includes the document's unique ID but not the full document since it no longer exists.

Result

You know exactly which document was deleted by its ID.

Recognizing that delete events provide the document ID helps you update your application state correctly without needing full data.

6

Advanced

Filtering Change Stream Events

🤔Before reading on: do you think you can listen to only insert events or must you receive all changes? Commit to your answer.

Concept: You can filter change streams to receive only specific event types like inserts, updates, or deletes.

Using aggregation pipelines, you can specify filters on the change stream to listen only for certain operations. For example, you can watch only insert events to reduce noise.

Result

Your application receives only the events it cares about, improving efficiency.

Filtering events reduces unnecessary processing and network traffic, making real-time apps more scalable.

7

Advanced

Resuming Change Streams After Interruptions

Concept: Change streams can resume from where they left off after a disconnect using resume tokens.

Each change event includes a resume token. If your application disconnects, you can restart the stream from the last token to avoid missing events.

Result

Your app maintains a continuous, reliable stream of changes even with network issues.

Understanding resume tokens is key to building robust real-time systems that never lose data updates.

8

Expert

Change Streams and Sharded Clusters

🤔Before reading on: do you think change streams work the same on single servers and sharded clusters? Commit to your answer.

Concept: Change streams work across sharded clusters but have special considerations for merging events from shards.

In sharded clusters, change streams aggregate events from multiple shards into a single stream. This requires internal coordination to maintain event order and consistency.

Result

You get a unified stream of changes from a distributed database setup.

Knowing how change streams handle sharding helps you design scalable real-time applications that work in complex environments.

Under the Hood

MongoDB's change streams use the database's internal oplog (operation log), which records every data change. The change stream API reads from this oplog and transforms raw operations into structured events like insert, update, and delete. It uses resume tokens to track progress and supports aggregation pipelines to filter or transform events before delivering them to the client.

Why designed this way?

Change streams were designed to provide a simple, efficient way to watch data changes without polling. Using the oplog leverages existing replication infrastructure, avoiding extra overhead. The design balances real-time responsiveness with scalability and fault tolerance, allowing applications to resume streams after failures and filter events to reduce noise.

┌───────────────┐      ┌───────────────┐      ┌───────────────┐
│  Client App   │◄─────│ Change Stream │◄─────│    Oplog      │
│ (Listener)   │      │   API Layer   │      │ (Operation Log)│
└───────────────┘      └───────────────┘      └───────────────┘
         ▲                      ▲                    ▲
         │                      │                    │
         │                      │                    │
         │                      │                    │
  Receives events       Filters & transforms    Records all DB
  (insert/update/delete)   events, manages       changes in order
                           resume tokens

Myth Busters - 4 Common Misconceptions

Quick: Do update events always include the full updated document? Commit yes or no.

Common Belief: Update events always send the entire updated document.

Reality: Update events only include the changed fields by default, not the full document unless explicitly requested.

Why it matters: Assuming full documents are sent can lead to inefficient processing and unexpected missing data in your application.

Quick: Can change streams miss events if your app disconnects? Commit yes or no.

Common Belief: Change streams cannot miss any events, even if the client disconnects.

Reality: If not using resume tokens properly, change streams can miss events during disconnects.

Why it matters: Missing events can cause your application to have outdated or inconsistent data, breaking real-time features.

Quick: Do delete events include the full deleted document? Commit yes or no.

Common Belief: Delete events include the full document that was deleted.

Reality: Delete events only include the document's ID, not the full document since it no longer exists.

Why it matters: Expecting full data on deletes can cause errors or confusion when trying to access deleted document details.

Quick: Are change streams only available on replica sets? Commit yes or no.

Common Belief: Change streams work on any MongoDB deployment, including standalone servers.

Reality: Change streams require a replica set or sharded cluster; they do not work on standalone servers.

Why it matters: Trying to use change streams on standalone servers will fail, leading to wasted time and confusion.

Expert Zone

1

Change streams can be combined with aggregation pipelines to perform complex filtering and transformation of events before your application receives them.

2

Resume tokens are opaque and must be stored exactly as received; modifying them breaks the ability to resume streams correctly.

3

In sharded clusters, change streams merge events from multiple shards, but event ordering is guaranteed only within each shard, not globally.

When NOT to use

Change streams are not suitable for standalone MongoDB servers or for use cases requiring historical change data beyond the oplog retention period. For full audit logs or long-term history, use MongoDB's built-in Change Data Capture tools or third-party log management systems.

Production Patterns

In production, change streams are used for real-time notifications, cache invalidation, syncing data between services, and triggering workflows. They are often combined with message queues or event buses to build scalable, reactive architectures.

Connections

Event-driven Architecture

Change streams provide the event source for event-driven systems.

Understanding change streams helps grasp how databases can emit events that trigger reactions in distributed systems, a core idea in event-driven design.

Observer Pattern (Software Design)

Change streams implement the observer pattern by notifying listeners of state changes.

Recognizing change streams as an observer pattern example clarifies how software components can react to changes without polling.

Real-time Messaging Systems

Change streams serve as a real-time data feed similar to messaging systems like Kafka or RabbitMQ.

Knowing this connection helps understand how databases can integrate with messaging platforms for scalable real-time data processing.

Common Pitfalls

#1 Expecting update events to include full documents by default.

Wrong approach:

changeStream.on('change', event => { console.log(event.fullDocument); }); // assumes fullDocument always present

Correct approach:

changeStream.on('change', event => { if(event.fullDocument) { console.log(event.fullDocument); } else { /* fetch full doc if needed */ } });

Root cause:

Misunderstanding that fullDocument is optional and only included if requested or available.

#2 Not handling resume tokens, causing missed events after disconnect.

Wrong approach:

const changeStream = collection.watch(); // no resume token handling

Correct approach:

const changeStream = collection.watch([], { resumeAfter: lastResumeToken });

Root cause:

Ignoring the need to store and use resume tokens to continue streams reliably.

#3 Trying to use change streams on a standalone MongoDB server.

Wrong approach:

const changeStream = standaloneCollection.watch(); // standalone server

Correct approach:

Use a replica set or sharded cluster deployment to enable change streams.

Root cause:

Not knowing that change streams require replica sets or sharded clusters.

Key Takeaways

Change streams let applications watch MongoDB data changes in real time without polling.

Insert events include the full new document, update events show only changed fields by default, and delete events provide the document ID.

Filtering and resume tokens make change streams efficient and reliable for production use.

Change streams rely on MongoDB's oplog and require replica sets or sharded clusters to function.

Understanding change streams unlocks building reactive, event-driven applications that respond instantly to data changes.