Capped collections for fixed-size data in MongoDB - Deep Dive

Overview - Capped collections for fixed-size data

What is it?

Capped collections are special types of collections in MongoDB that store data in a fixed size. They automatically overwrite the oldest entries when the allocated space is full, keeping the collection size constant. This makes them ideal for logging or caching where only recent data matters. They maintain insertion order and provide high performance for writes.

Why it matters

Without capped collections, managing fixed-size data would require manual cleanup or complex logic to remove old data, which can be slow and error-prone. Capped collections solve this by automatically controlling size and data lifecycle, ensuring efficient storage and fast access. This helps applications handle continuous data streams like logs or sensor data without growing storage endlessly.

Where it fits

Before learning capped collections, you should understand basic MongoDB collections and CRUD operations. After this, you can explore MongoDB's TTL indexes for automatic expiration and advanced data retention strategies. Capped collections fit into the broader topic of data lifecycle management in databases.

Mental Model

Core Idea

A capped collection is like a circular buffer that keeps only the newest data within a fixed storage size by overwriting the oldest entries automatically.

Think of it like...

Imagine a fixed-size whiteboard where you write notes in order. When the board is full, you erase the oldest notes at the top to make space for new ones at the bottom, always keeping the latest information visible.

┌───────────────────────────────┐
│ Capped Collection (Fixed Size) │
├───────────────────────────────┤
│ Entry 1 (Oldest)              │
│ Entry 2                      │
│ ...                         │
│ Entry N-1                   │
│ Entry N (Newest)             │
└───────────────────────────────┘
When full:
┌───────────────────────────────┐
│ Entry 2 (Now Oldest)          │
│ Entry 3                      │
│ ...                         │
│ Entry N                     │
│ New Entry (Overwrites Entry 1)│
└───────────────────────────────┘

Build-Up - 7 Steps

1

Foundation

Understanding MongoDB Collections

Concept: Learn what a MongoDB collection is and how it stores documents.

A MongoDB collection is like a folder that holds many documents (records). Each document is a set of key-value pairs, similar to a JSON object. Collections do not have a fixed size and can grow as you add more documents.

Result

You understand that collections are flexible containers for data in MongoDB.

Knowing what a collection is helps you appreciate why capped collections are a special, fixed-size variant.

2

Foundation

Basics of Fixed-Size Data Storage

Concept: Understand what fixed-size storage means and why it matters.

Fixed-size storage means the space used for data does not grow beyond a set limit. When new data arrives and the space is full, old data must be removed or overwritten. This is useful for logs or caches where only recent data is important.

Result

You grasp the idea of limiting storage size to control resource use.

Recognizing fixed-size storage needs prepares you to see why capped collections are useful.

3

Intermediate

Creating a Capped Collection in MongoDB

🤔Before reading on: do you think capped collections can be created on existing collections or only at creation? Commit to your answer.

Concept: Learn how to create a capped collection with a fixed size in MongoDB.

You create a capped collection using the command: db.createCollection('name', { capped: true, size: }). The size parameter sets the maximum storage in bytes. You cannot convert an existing collection to capped; it must be created as capped from the start.

Result

A capped collection is created that will hold data up to the specified size and then overwrite old data.

Understanding creation constraints prevents common mistakes and clarifies capped collections' fixed nature.

4

Intermediate

Behavior of Writes in Capped Collections

🤔Before reading on: do you think capped collections allow deleting individual documents? Commit to yes or no.

Concept: Explore how capped collections handle inserts and deletions.

In capped collections, inserts add new documents at the end. When full, the oldest documents are overwritten automatically. You cannot delete individual documents or update documents to increase their size. This preserves the fixed size and insertion order.

Result

You see that capped collections behave like a circular buffer with automatic overwriting and restricted operations.

Knowing these write rules helps avoid errors and explains why capped collections are fast and predictable.

5

Intermediate

Use Cases for Capped Collections

Concept: Identify practical scenarios where capped collections shine.

Capped collections are ideal for logs, caches, real-time data streams, or any scenario where only recent data matters and storage size must be controlled. They provide high write throughput and predictable storage use.

Result

You can match capped collections to real-world problems effectively.

Recognizing use cases guides you to choose capped collections wisely in projects.

6

Advanced

Performance Benefits and Limitations

🤔Before reading on: do you think capped collections support indexes other than the default _id? Commit to yes or no.

Concept: Understand performance characteristics and indexing in capped collections.

Capped collections provide high write performance because they avoid document moves and fragmentation. They support indexes on _id but have limited support for other indexes. Their fixed size prevents storage bloat but limits flexibility in data management.

Result

You appreciate the trade-offs between speed and flexibility in capped collections.

Knowing these limits helps you optimize performance and avoid pitfalls in production.

7

Expert

Internal Storage and Overwrite Mechanics

🤔Before reading on: do you think capped collections physically delete old documents or just overwrite them in place? Commit to your answer.

Concept: Dive into how MongoDB manages storage internally for capped collections.

Internally, capped collections use a circular buffer in the storage engine. When full, new inserts overwrite the oldest documents in place without moving data around. This avoids fragmentation and keeps insertion order intact. Deletions are not allowed to maintain this structure.

Result

You understand the low-level mechanism that makes capped collections efficient and predictable.

Understanding the internal circular buffer explains why capped collections have unique constraints and performance.

Under the Hood

Capped collections allocate a fixed amount of storage space on disk. MongoDB treats this space as a circular buffer where new documents are appended until the space is full. When full, new inserts overwrite the oldest documents in the same physical location. This avoids fragmentation and maintains insertion order. The storage engine disallows deletions and restricts updates that increase document size to preserve this structure.

Why designed this way?

Capped collections were designed to provide a high-performance, predictable storage mechanism for use cases like logging and caching. The circular buffer approach avoids costly data movement and fragmentation common in regular collections. Alternatives like manual cleanup or TTL indexes were less efficient or flexible for fixed-size needs. This design trades flexibility for speed and simplicity.

┌───────────────────────────────┐
│ Fixed Storage Space (Circular)│
├─────────────┬───────────────┤
│ Oldest Doc  │ Newest Doc    │
│ (to overwrite)│ (last inserted)│
├─────────────┴───────────────┤
│ Insert pointer moves forward │
│ Overwrites oldest documents  │
└───────────────────────────────┘

Myth Busters - 4 Common Misconceptions

Quick: Can you delete individual documents from a capped collection? Commit to yes or no.

Common Belief: You can delete any document from a capped collection like a normal collection.

Reality: Capped collections do not allow deletion of individual documents to maintain their fixed size and insertion order.

Why it matters: Trying to delete documents causes errors and breaks the fixed-size guarantee, leading to unexpected application failures.

Quick: Can you convert an existing normal collection into a capped collection? Commit to yes or no.

Common Belief: You can convert any existing collection into a capped collection easily.

Reality: You must create a capped collection from scratch; existing collections cannot be converted to capped.

Why it matters: Attempting conversion wastes time and causes confusion; understanding this avoids deployment mistakes.

Quick: Do capped collections support all types of indexes like normal collections? Commit to yes or no.

Common Belief: Capped collections support all indexes just like normal collections.

Reality: Capped collections only support the default _id index and have limited support for others.

Why it matters: Expecting full indexing can lead to poor query performance or errors in production.

Quick: Does the size parameter in capped collections limit the number of documents? Commit to yes or no.

Common Belief: The size parameter limits the number of documents stored.

Reality: The size parameter limits the total storage in bytes, not the document count, which varies by document size.

Why it matters: Misunderstanding this can cause unexpected data loss or storage issues when documents vary in size.

Expert Zone

1

Capped collections maintain insertion order strictly, which can be leveraged for efficient range queries on insertion time.

2

The internal circular buffer means that documents are overwritten in place, so document _id values remain unique but physical storage is reused.

3

Updates that increase document size are disallowed because they would break the fixed-size storage model and cause fragmentation.

When NOT to use

Avoid capped collections when you need flexible document deletion, complex indexing, or variable storage size. Use regular collections with TTL indexes or manual cleanup for those cases.

Production Patterns

In production, capped collections are often used for high-throughput logging systems, real-time analytics buffers, and caching layers where data freshness and write speed are critical. They are combined with tailable cursors to stream new data efficiently.

Connections

Circular Buffers (Data Structures)

Capped collections implement a circular buffer pattern at the database storage level.

Understanding circular buffers in programming helps grasp how capped collections overwrite old data efficiently without fragmentation.

Log Rotation (System Administration)

Capped collections automate log rotation by overwriting old entries when full.

Knowing log rotation concepts clarifies why capped collections are ideal for managing continuous data streams like logs.

Cache Eviction Policies (Computer Science)

Capped collections implement a fixed-size cache with automatic eviction of oldest data.

Recognizing capped collections as a form of cache with FIFO eviction helps understand their use in performance-critical applications.

Common Pitfalls

#1 Trying to delete documents from a capped collection.

Wrong approach:

db.cappedCollection.deleteOne({ _id: someId })

Correct approach:

// Do not delete; let capped collection overwrite old data automatically
// Instead, create a new capped collection if needed

Root cause:

Misunderstanding that capped collections disallow deletions to maintain fixed size and insertion order.

#2 Attempting to convert an existing collection to capped.

Wrong approach:

db.runCommand({ convertToCapped: 'existingCollection', size: 100000 })

Correct approach:

db.createCollection('newCappedCollection', { capped: true, size: 100000 })

Root cause:

Believing capped collections can be created by conversion rather than only at creation.

#3 Expecting capped collections to support all indexes.

Wrong approach:

db.cappedCollection.createIndex({ field: 1 })

Correct approach:

// Use only _id index or design queries accordingly
// Avoid creating unsupported indexes on capped collections

Root cause:

Assuming capped collections have the same indexing flexibility as normal collections.

Key Takeaways

Capped collections are fixed-size MongoDB collections that automatically overwrite oldest data when full, ensuring constant storage size.

They behave like circular buffers, maintaining insertion order and disallowing deletions or size-increasing updates to preserve structure.

You must create capped collections at the start; existing collections cannot be converted.

Capped collections provide high write performance and are ideal for logs, caches, and real-time data streams where only recent data matters.

Understanding their limitations in indexing and operations helps avoid common mistakes and use them effectively in production.