How MongoDB scans documents - Mechanics & Internals

Overview - How MongoDB scans documents

What is it?

MongoDB scans documents to find data that matches a query. It looks through the stored documents in collections to check if they meet the search conditions. This process can be fast or slow depending on how MongoDB searches through the data. Scanning means reading documents one by one or using indexes to jump directly to the right ones.

Why it matters

Without efficient scanning, searching data in MongoDB would be very slow, especially with large amounts of data. This would make apps and websites feel sluggish or unresponsive. Good scanning methods help MongoDB quickly find the right information, saving time and computing power. It makes data retrieval practical and scalable for real-world use.

Where it fits

Before learning how MongoDB scans documents, you should understand what documents and collections are in MongoDB. After this, you can learn about indexes and query optimization to improve scanning speed. Later, you can explore aggregation pipelines and performance tuning to handle complex data searches.

Mental Model

Core Idea

MongoDB scans documents by either checking each one in order or using indexes to jump directly to matching documents, balancing speed and resource use.

Think of it like...

Imagine looking for a book in a messy library. You can either check every book one by one (full scan) or use the library's catalog to find the exact shelf and book quickly (index scan).

┌───────────────┐        ┌───────────────┐
│   Query       │        │   Collection  │
└──────┬────────┘        └──────┬────────┘
       │                        │
       │                        │
       ▼                        ▼
┌───────────────┐        ┌───────────────┐
│ Index Exists? │──Yes──▶│ Use Index to  │
└──────┬────────┘        │ jump to docs  │
       │No               └──────┬────────┘
       ▼                        │
┌───────────────┐               │
│ Full Collection│◀────────────┘
│ Scan (Check all│
│ documents)    │
└───────────────┘

Build-Up - 7 Steps

1

Foundation

What is a MongoDB document scan

Concept: Introduces the basic idea of scanning documents in MongoDB collections.

In MongoDB, data is stored as documents inside collections. When you ask MongoDB to find data, it scans these documents to see which ones match your request. This scanning can be simple: checking each document one by one until it finds matches.

Result

You understand that scanning means looking through documents to find matches.

Understanding scanning as a basic search process helps you see why some queries are fast and others slow.

2

Foundation

Difference between full scan and index scan

Concept: Explains two main ways MongoDB scans documents: full collection scan and index scan.

A full scan means MongoDB reads every document in a collection to find matches. An index scan uses a special data structure (index) to jump directly to matching documents without checking all. Indexes are like shortcuts that speed up scanning.

Result

You can tell when MongoDB reads all documents versus when it uses an index.

Knowing these two scanning methods is key to understanding query performance.

3

Intermediate

How MongoDB uses indexes to scan

🤔Before reading on: do you think MongoDB always scans the whole collection even if an index exists? Commit to your answer.

Concept: Shows how MongoDB uses indexes to avoid scanning every document.

When a query matches an index, MongoDB uses the index to find document locations quickly. It reads the index entries, which are smaller and faster to search, then fetches only the matching documents. This reduces the number of documents scanned.

Result

Queries using indexes scan fewer documents and run faster.

Understanding index scans reveals why creating the right indexes drastically improves query speed.

4

Intermediate

When MongoDB falls back to full collection scan

🤔Before reading on: do you think MongoDB can use indexes for every query? Commit to your answer.

Concept: Explains conditions when MongoDB cannot use indexes and must scan all documents.

If a query has no matching index or uses fields not indexed, MongoDB scans every document. Also, some query types or operators prevent index use. This full scan reads all documents, which can be slow for large collections.

Result

Queries without suitable indexes cause full scans and slower performance.

Knowing when full scans happen helps you design better indexes and avoid slow queries.

5

Intermediate

Covered queries and index-only scans

🤔Before reading on: do you think MongoDB always fetches documents after using an index? Commit to your answer.

Concept: Introduces covered queries where MongoDB answers queries using only the index without reading documents.

If a query requests only fields stored in an index, MongoDB can return results directly from the index. This avoids fetching documents entirely, making scanning even faster. This is called a covered query.

Result

Covered queries scan only the index, improving speed and reducing resource use.

Understanding covered queries shows how index design can optimize scanning beyond just filtering.

6

Advanced

Impact of document size and storage on scanning

🤔Before reading on: do you think document size affects scanning speed? Commit to your answer.

Concept: Explores how document size and storage layout influence scan performance.

Larger documents take more time to read during scanning. MongoDB stores documents in data files with some fragmentation possible. Scanning many large or fragmented documents slows queries. Indexes help reduce this by limiting document reads.

Result

You see that scanning speed depends not just on indexes but also on document size and storage.

Knowing storage effects helps in optimizing schema and indexing for better scan performance.

7

Expert

How MongoDB query planner chooses scan method

🤔Before reading on: do you think MongoDB always picks the fastest scan method automatically? Commit to your answer.

Concept: Details how MongoDB decides between index scan and full scan using its query planner.

MongoDB's query planner tests different scan plans and estimates their cost based on statistics. It chooses the plan with the lowest estimated cost. Sometimes it picks a full scan if indexes are not selective or missing. Understanding this helps diagnose unexpected slow queries.

Result

You understand MongoDB's internal decision-making for scanning strategies.

Knowing the query planner's role enables advanced tuning and troubleshooting of query performance.

Under the Hood

MongoDB stores documents in collections as BSON objects on disk. When scanning, it either reads the entire collection sequentially (collection scan) or uses B-tree indexes to quickly locate matching documents. The query planner evaluates available indexes and their selectivity to choose the best scan method. Indexes store pointers to documents, allowing MongoDB to fetch only relevant data. Covered queries avoid fetching documents by returning data directly from indexes.

Why designed this way?

MongoDB was designed for flexibility and speed. Full scans are simple but slow for large data. Indexes were added to speed up common queries by avoiding unnecessary reads. The query planner balances between using indexes and full scans to optimize performance based on data distribution and query shape. This design allows MongoDB to handle diverse workloads efficiently.

┌───────────────┐
│   Client      │
└──────┬────────┘
       │ Query
       ▼
┌───────────────┐
│ Query Planner │
└──────┬────────┘
       │ Chooses plan
       ▼
┌───────────────┐        ┌───────────────┐
│ Index Scan    │◀──────▶│ Index (B-tree)│
└──────┬────────┘        └──────┬────────┘
       │ Fetch docs             │
       ▼                       ▼
┌───────────────┐        ┌───────────────┐
│ Collection    │        │ Documents     │
│ Scan          │        │ (BSON data)   │
└───────────────┘        └───────────────┘

Myth Busters - 4 Common Misconceptions

Quick: Does MongoDB always use indexes if they exist? Commit to yes or no.

Common Belief: MongoDB always uses indexes if they exist for a query.

Reality: MongoDB may choose a full collection scan if it estimates that scanning all documents is cheaper than using an index, especially if the index is not selective.

Why it matters: Assuming indexes are always used can lead to unexpected slow queries and wasted effort creating unnecessary indexes.

Quick: Can MongoDB answer queries without reading any documents? Commit to yes or no.

Common Belief: MongoDB always reads documents after using an index to answer queries.

Reality: If the query only requests fields stored in the index, MongoDB can answer directly from the index without reading documents (covered query).

Why it matters: Not knowing about covered queries misses opportunities to design indexes that speed up queries significantly.

Quick: Does document size not affect scan speed? Commit to yes or no.

Common Belief: Document size does not impact how fast MongoDB scans documents.

Reality: Larger documents take longer to read and process during scans, slowing down query performance.

Why it matters: Ignoring document size can cause poor schema design and slow queries on large datasets.

Quick: Is the query planner's choice always perfect? Commit to yes or no.

Common Belief: MongoDB's query planner always picks the fastest scanning method.

Reality: The planner uses estimates and sometimes picks suboptimal plans, especially with outdated statistics or complex queries.

Why it matters: Blind trust in the planner can hide performance issues that require manual index tuning or query rewriting.

Expert Zone

1

MongoDB's query planner caches plans but may re-plan queries if data distribution changes, affecting scan choices dynamically.

2

Index intersection allows MongoDB to combine multiple indexes to satisfy a query, influencing how scanning is performed.

3

The storage engine's data layout and compression affect how quickly documents can be scanned from disk or memory.

When NOT to use

Full collection scans are inefficient for large datasets and should be avoided by creating appropriate indexes. For complex queries, aggregation pipelines or specialized search engines like Elasticsearch may be better. When real-time analytics or full-text search is needed, MongoDB's scanning alone may not suffice.

Production Patterns

In production, developers monitor query plans using explain() to ensure index scans are used. They design compound indexes to cover frequent queries and avoid full scans. Sharding distributes data to reduce scan scope. Monitoring slow queries and adjusting indexes is a continuous process to maintain performance.

Connections

Database Indexing

Builds-on

Understanding how MongoDB scans documents is deeply connected to how indexes work, as indexes guide scanning to be efficient.

Operating System File Caching

Similar pattern

Just like OS caches frequently used files to speed access, MongoDB relies on memory caching to speed document scanning, showing how storage and memory interplay affects performance.

Library Catalog Systems

Analogous system

The way MongoDB uses indexes to jump to documents is like how a library catalog helps find books quickly, illustrating how organizing information reduces search time.

Common Pitfalls

#1 Running queries without indexes on large collections.

Wrong approach:

db.users.find({ age: { $gt: 30 } })

Correct approach:

db.users.createIndex({ age: 1 })
db.users.find({ age: { $gt: 30 } })

Root cause:

Not creating indexes leads MongoDB to scan every document, causing slow queries.

#2 Assuming indexes speed up all queries regardless of fields used.

Wrong approach:

db.orders.find({ status: 'shipped', total: { $gt: 100 } }) // no index on total

Correct approach:

db.orders.createIndex({ status: 1, total: 1 })
db.orders.find({ status: 'shipped', total: { $gt: 100 } })

Root cause:

Using queries on fields without indexes forces full scans or inefficient index use.

#3 Expecting covered queries without projecting only indexed fields.

Wrong approach:

db.products.find({ category: 'books' }, { name: 1, price: 1, description: 1 }) // description not indexed

Correct approach:

db.products.find({ category: 'books' }, { name: 1, price: 1 }) // only indexed fields

Root cause:

Requesting fields not in the index forces MongoDB to fetch documents, losing covered query benefits.

Key Takeaways

MongoDB scans documents either by checking all documents or using indexes to find matches faster.

Indexes act like shortcuts that help MongoDB avoid reading every document, improving query speed.

Not all queries use indexes; some require full collection scans which are slower on large data.

Covered queries let MongoDB answer from indexes alone without reading documents, boosting performance.

Understanding MongoDB's scanning helps you design better indexes and write faster queries.