Pagination pattern with skip and limit in MongoDB - Deep Dive

Overview - Pagination pattern with skip and limit

What is it?

Pagination with skip and limit is a way to divide a large list of data into smaller parts or pages. It helps you fetch only a portion of data at a time instead of loading everything at once. In MongoDB, skip tells the database how many items to ignore from the start, and limit tells how many items to return after skipping. This makes browsing large collections faster and easier.

Why it matters

Without pagination, loading all data at once can be slow and use too much memory, making apps lag or crash. Pagination lets users see data in chunks, improving speed and user experience. It also reduces the load on the database and network, making systems more efficient and scalable.

Where it fits

Before learning pagination, you should understand basic MongoDB queries and how to find documents. After mastering pagination with skip and limit, you can explore more advanced pagination methods like using range queries or cursors for better performance.

Mental Model

Core Idea

Pagination with skip and limit fetches a specific slice of data by skipping a number of items and then limiting the number returned.

Think of it like...

Imagine a book with many pages. Skip is like flipping past a certain number of pages, and limit is like reading only a set number of pages after that. You don’t read the whole book at once, just the part you want.

┌───────────────┐
│ Full dataset  │
│ [item1, ...]  │
└──────┬────────┘
       │ skip N items
       ▼
┌───────────────┐
│ After skip    │
│ [itemN+1, ...]│
└──────┬────────┘
       │ limit M items
       ▼
┌───────────────┐
│ Page of data  │
│ [itemN+1,...] │
└───────────────┘

Build-Up - 6 Steps

1

Foundation

Understanding basic skip and limit

Concept: Learn what skip and limit do in MongoDB queries.

In MongoDB, skip(N) tells the database to ignore the first N documents in the result. Limit(M) tells it to return only M documents after skipping. For example, db.collection.find().skip(5).limit(10) skips the first 5 documents and returns the next 10.

Result

You get a smaller set of documents starting from the 6th document, up to 10 documents.

Knowing how skip and limit work together is the foundation for fetching data in pages instead of all at once.

2

Foundation

Why pagination is needed

Concept: Understand the problem pagination solves with large data sets.

When a collection has thousands or millions of documents, fetching all at once is slow and uses a lot of memory. Pagination breaks data into pages so users and apps can load and display data bit by bit, improving speed and usability.

Result

Apps become faster and more responsive when showing data in pages.

Recognizing the performance and user experience benefits motivates using pagination.

3

Intermediate

Implementing skip and limit for pages

🤔Before reading on: Do you think skip counts from zero or one? Commit to your answer.

Concept: Learn how to calculate skip and limit values to get specific pages.

Page numbers start at 1. To get page P with page size S, skip = (P - 1) * S, limit = S. For example, page 3 with size 10 means skip 20, limit 10. Query: db.collection.find().skip(20).limit(10).

Result

You get documents 21 to 30 from the collection.

Understanding the math behind skip and limit lets you fetch any page correctly.

4

Intermediate

Sorting before pagination

🤔Before reading on: Does the order of documents affect pagination results? Commit to yes or no.

Concept: Learn why sorting data before applying skip and limit is important.

Without sorting, MongoDB returns documents in natural order, which can be inconsistent. Sorting ensures pages show data in a predictable order. Example: db.collection.find().sort({date: -1}).skip(10).limit(5) returns 5 newest documents after skipping 10.

Result

Pages show consistent, ordered data.

Sorting before pagination guarantees users see data in the expected order across pages.

5

Advanced

Performance issues with large skip values

🤔Before reading on: Do you think skip is fast even with very large numbers? Commit to yes or no.

Concept: Understand how skip works internally and why large skip values slow queries.

MongoDB must scan and discard skipped documents before returning results. Large skip means scanning many documents, causing slow queries and high resource use. For example, skip(1000000) is expensive.

Result

Queries with large skip values become slow and inefficient.

Knowing skip’s cost helps you avoid performance problems in big data sets.

6

Advanced

Alternatives to skip for efficient pagination

🤔Before reading on: Can you guess a faster way to paginate than skip? Commit to your answer.

Concept: Learn about using range queries or bookmarks instead of skip for better performance.

Instead of skip, use a filter on a sorted field (like _id or date) to get the next page. For example, db.collection.find({_id: {$gt: last_id}}).limit(10). This avoids scanning skipped documents.

Result

Pagination becomes faster and scales better with large data.

Understanding alternatives to skip unlocks scalable pagination strategies.

Under the Hood

MongoDB processes skip by scanning and discarding the first N documents matching the query, then returns the next M documents as per limit. This means skip requires reading all skipped documents internally, even if they are not returned. Limit simply caps the number of documents sent back to the client.

Why designed this way?

Skip and limit were designed for simple pagination and quick prototyping. They provide an easy way to slice data without complex queries. However, this simplicity trades off performance for large skips. Alternatives like range queries were developed later to address this.

┌───────────────┐
│ Query results │
│ [doc1, doc2,  │
│  doc3, ...,   │
│  docN]        │
└──────┬────────┘
       │ skip N docs (scan & discard)
       ▼
┌───────────────┐
│ Remaining docs│
│ [docN+1, ...] │
└──────┬────────┘
       │ limit M docs (return only M)
       ▼
┌───────────────┐
│ Returned page │
│ [docN+1,...]  │
└───────────────┘

Myth Busters - 4 Common Misconceptions

Quick: Does skip improve query speed by jumping directly to the page? Commit to yes or no.

Common Belief: Skip makes the database jump directly to the requested page, so it is always fast.

Reality: Skip actually scans and discards all skipped documents internally, so large skip values slow down queries.

Why it matters: Believing skip is always fast leads to slow, unresponsive apps when paginating deep into large collections.

Quick: If you don’t sort before skip and limit, will pagination still be consistent? Commit to yes or no.

Common Belief: Sorting is optional; skip and limit alone guarantee consistent pages.

Reality: Without sorting, document order is not guaranteed, so pages may show overlapping or missing data.

Why it matters: Skipping sorting causes confusing user experiences with inconsistent or repeated data across pages.

Quick: Can you use skip and limit for real-time data streams without issues? Commit to yes or no.

Common Belief: Skip and limit work well even if data changes frequently during pagination.

Reality: Data changes can cause skipped documents to shift, leading to missing or duplicated items between pages.

Why it matters: Ignoring this causes bugs in apps showing live data, confusing users with inconsistent results.

Quick: Is it always better to use skip and limit over other pagination methods? Commit to yes or no.

Common Belief: Skip and limit are the best and simplest way to paginate in all cases.

Reality: For large datasets or high page numbers, range-based pagination or cursor methods perform better and scale more efficiently.

Why it matters: Using skip and limit blindly can cause performance bottlenecks and poor user experience at scale.

Expert Zone

1

Using skip with large values can cause the query planner to choose inefficient indexes, slowing queries further.

2

Combining skip and limit with complex filters or aggregations can produce unexpected results if not carefully ordered.

3

MongoDB’s internal document storage order can change after compaction or updates, affecting pagination consistency without explicit sorting.

When NOT to use

Avoid skip and limit for deep pagination in large collections. Instead, use range queries on indexed fields or MongoDB’s native cursor-based pagination for better performance and consistency.

Production Patterns

In production, developers often use skip and limit for small page numbers or admin tools, but switch to range-based pagination for user-facing apps with large datasets. They also combine pagination with caching and prefetching to improve responsiveness.

Connections

Cursor-based pagination

Alternative method

Knowing skip and limit helps understand why cursor-based pagination is preferred for performance and consistency in large datasets.

SQL OFFSET and LIMIT

Similar pattern in relational databases

Understanding skip and limit in MongoDB parallels OFFSET and LIMIT in SQL, showing pagination is a common need across database types.

Memory paging in operating systems

Conceptual similarity

Pagination in databases is like memory paging in OS, where data is loaded in chunks to manage resources efficiently.

Common Pitfalls

#1 Using skip with very large numbers causing slow queries.

Wrong approach:

db.collection.find().skip(1000000).limit(10)

Correct approach:

db.collection.find({_id: {$gt: last_seen_id}}).limit(10)

Root cause:

Misunderstanding that skip scans all skipped documents internally, making large skips inefficient.

#2 Not sorting before applying skip and limit, causing inconsistent pages.

Wrong approach:

db.collection.find().skip(10).limit(5)

Correct approach:

db.collection.find().sort({createdAt: -1}).skip(10).limit(5)

Root cause:

Assuming natural order is stable, ignoring that MongoDB returns documents in arbitrary order without sort.

#3 Using skip and limit on rapidly changing data without handling duplicates or missing items.

Wrong approach:

db.collection.find().sort({date: -1}).skip(20).limit(10) // on live data

Correct approach:

Use range queries with stable sort keys and track last seen values for pagination.

Root cause:

Not accounting for data changes causing shifts in document positions between pages.

Key Takeaways

Pagination with skip and limit lets you fetch a specific slice of data by skipping some documents and limiting the number returned.

Always sort your data before paginating to ensure consistent and predictable pages.

Skip can cause performance problems with large values because MongoDB scans all skipped documents internally.

For large datasets or deep pages, use range-based pagination or cursor methods instead of skip and limit.

Understanding these patterns helps build fast, scalable, and user-friendly data browsing experiences.