Cursor-based pagination (startAfter, endBefore) in Firebase - Deep Dive

Overview - Cursor-based pagination (startAfter, endBefore)

What is it?

Cursor-based pagination is a way to load data in chunks by remembering a specific position in a list. Instead of counting pages, it uses a marker called a cursor to start or end the data retrieval. In Firebase, this is done using startAfter and endBefore methods to fetch data after or before a certain item. This helps in efficiently loading large lists without skipping or repeating items.

Why it matters

Without cursor-based pagination, loading large lists can be slow and unreliable because you might skip or repeat items when data changes. It solves the problem of navigating through data smoothly, especially when new items are added or removed. This makes apps faster and more user-friendly, avoiding confusing jumps or missing data.

Where it fits

Before learning cursor-based pagination, you should understand basic Firebase queries and how data is structured in collections. After this, you can learn about infinite scrolling, real-time updates, and optimizing data loading for better user experience.

Mental Model

Core Idea

Cursor-based pagination remembers a specific item in a list to fetch the next or previous set of items without counting pages.

Think of it like...

It's like using a bookmark in a book to remember where you left off reading, so you can continue from that exact spot next time without flipping pages randomly.

┌───────────────┐
│ Data List     │
├───────────────┤
│ Item 1        │
│ Item 2        │
│ Item 3        │ ← Cursor here
│ Item 4        │
│ Item 5        │
└───────────────┘

[startAfter(Item 3)] → fetches Item 4, Item 5...
[endBefore(Item 4)] → fetches Item 1, Item 2, Item 3

Build-Up - 7 Steps

1

Foundation

Understanding Basic Firebase Queries

Concept: Learn how to fetch data from Firebase collections using simple queries.

Firebase stores data in collections of documents. You can ask Firebase to give you documents that match certain rules, like all users or all messages. For example, to get the first 5 items, you use limit(5). This fetches the first chunk of data.

Result

You get a small list of documents from Firebase, limited to the number you asked for.

Knowing how to fetch data in chunks is the first step to loading data efficiently without overwhelming the app or network.

2

Foundation

Why Simple Offset Pagination Fails

Concept: Understand why counting pages or skipping items can cause problems with changing data.

Offset pagination uses skip and limit to jump pages, like skipping the first 10 items to get the next 10. But if data changes (items added or removed), the pages shift and you might see duplicates or miss items. This is unreliable for live data.

Result

You realize that offset pagination can cause confusing results when data updates frequently.

Understanding the limits of offset pagination motivates the need for cursor-based methods that track positions, not counts.

3

Intermediate

Introducing Cursor-Based Pagination

🤔Before reading on: do you think cursor-based pagination uses page numbers or item positions to fetch data? Commit to your answer.

Concept: Cursor pagination uses a specific item as a marker to fetch the next or previous set of items, avoiding counting pages.

Instead of saying 'give me page 3', you say 'give me items after this item'. In Firebase, you use startAfter(cursor) to get items after a known document, or endBefore(cursor) to get items before it. This keeps pagination stable even if data changes.

Result

You can fetch data chunks reliably, always continuing from where you left off without missing or repeating items.

Knowing that pagination can be based on item positions rather than page numbers helps avoid common bugs with changing data.

4

Intermediate

Using startAfter and endBefore in Firebase

🤔Before reading on: do you think startAfter includes the cursor item in results or skips it? Commit to your answer.

Concept: Learn how to apply startAfter and endBefore methods to paginate forward and backward through data.

startAfter(cursor) fetches documents after the cursor, excluding the cursor itself. endBefore(cursor) fetches documents before the cursor, excluding it. You get the cursor from a document snapshot, then use it in your next query to continue pagination.

Result

You can move forward and backward through data lists smoothly, fetching the right items each time.

Understanding that these methods exclude the cursor item prevents off-by-one errors and repeated items.

5

Intermediate

Combining Cursors with Ordering and Limits

Concept: Cursor pagination works best with consistent ordering and limits to control data chunks.

You must order your query by a field (like timestamp or name) to use cursors. Then apply limit(n) to get a fixed number of items. The cursor is a document snapshot from the ordered list. This ensures pagination is predictable and stable.

Result

Your paginated queries return consistent, ordered chunks of data without surprises.

Knowing that ordering is required for cursor pagination helps avoid confusing results and errors.

6

Advanced

Handling Edge Cases and Real-Time Updates

🤔Before reading on: do you think cursor pagination automatically updates if data changes? Commit to your answer.

Concept: Learn how cursor pagination behaves when data is added, removed, or updated during navigation.

If new items are added before the cursor, they won't appear in the current pagination unless you refresh. If items are deleted, cursors might point to missing documents, causing empty pages. You must handle these cases by refreshing cursors or using real-time listeners carefully.

Result

You understand how to keep pagination reliable even when data changes live.

Knowing the limits of cursor pagination with live data helps design better user experiences and avoid empty or repeated pages.

7

Expert

Optimizing Cursor Pagination for Performance

🤔Before reading on: do you think fetching only document IDs is enough for cursor pagination? Commit to your answer.

Concept: Explore advanced techniques to reduce data transfer and improve speed using cursors.

You can fetch only the fields needed for cursors (like timestamps or IDs) to minimize data size. Also, caching cursors and prefetching next pages improves responsiveness. Understanding Firestore's indexing and query costs helps optimize pagination queries for large datasets.

Result

Your app loads pages faster and uses fewer resources, improving user experience and cost efficiency.

Knowing how to optimize cursor pagination prevents slowdowns and high costs in production apps.

Under the Hood

Firebase stores documents in ordered indexes based on query fields. Cursor pagination uses document snapshots as pointers into these indexes. When you call startAfter or endBefore with a snapshot, Firebase uses the index to find the exact position and fetches documents after or before it efficiently without scanning the whole collection.

Why designed this way?

This design avoids the inefficiency and inconsistency of offset pagination, especially in distributed databases where data changes frequently. Using document snapshots as cursors leverages Firestore's strong indexing and real-time capabilities to provide stable, fast pagination.

┌───────────────┐
│ Collection    │
├───────────────┤
│ Document 1    │
│ Document 2    │
│ Document 3 ◄──┤ ← Cursor (snapshot)
│ Document 4    │
│ Document 5    │
└───────────────┘

Query with startAfter(Document 3) → fetch Document 4, Document 5

Index: [Doc1, Doc2, Doc3, Doc4, Doc5]
Cursor points to Doc3 position in index

Myth Busters - 4 Common Misconceptions

Quick: Does startAfter include the cursor item in the results? Commit to yes or no.

Common Belief: startAfter returns the cursor item and all items after it.

Reality: startAfter excludes the cursor item and returns only items after it.

Why it matters: Including the cursor item causes duplicate data in pagination, confusing users and breaking navigation.

Quick: Can you use cursor pagination without ordering your query? Commit to yes or no.

Common Belief: Cursor pagination works without ordering because it just remembers a document.

Reality: Ordering is required to define a consistent sequence; without it, cursors cannot reliably point to positions.

Why it matters: Without ordering, pagination results are unpredictable and may skip or repeat items.

Quick: Does cursor pagination automatically update when data changes? Commit to yes or no.

Common Belief: Cursor pagination always reflects live data changes instantly.

Reality: Cursor pagination queries return snapshots at query time; live updates require listeners and extra handling.

Why it matters: Assuming automatic updates can cause stale data views or missed new items in paginated lists.

Quick: Is offset pagination just as good as cursor pagination for large, changing datasets? Commit to yes or no.

Common Belief: Offset pagination is simpler and works fine for all cases.

Reality: Offset pagination is inefficient and unreliable with changing data, causing duplicates or missing items.

Why it matters: Using offset pagination in live apps leads to poor user experience and higher costs.

Expert Zone

1

Cursors are based on document snapshots, which include field values and metadata, not just IDs, enabling complex multi-field ordering.

2

Using composite indexes in Firestore can optimize cursor queries that order by multiple fields, improving performance.

3

Prefetching the next page using cursors can hide loading delays, but requires careful cache invalidation to avoid stale data.

When NOT to use

Cursor pagination is not ideal when you need random access to arbitrary pages by number; in such cases, consider offset pagination or maintaining your own page index. Also, for very small datasets, simple queries without pagination may be simpler.

Production Patterns

In real apps, cursor pagination is combined with infinite scrolling or 'Load More' buttons. Developers store the last visible document snapshot as the cursor. They handle edge cases like deleted documents by refreshing cursors or resetting pagination. Caching cursors and prefetching next pages are common for smooth UX.

Connections

Database Indexing

Cursor pagination relies on ordered indexes to find positions efficiently.

Understanding how indexes work helps grasp why cursor pagination is fast and stable compared to offset methods.

Stateful Navigation in Web Apps

Both use a remembered position or state to continue from where the user left off.

Knowing how apps remember user state clarifies why cursors are like bookmarks for data navigation.

Book Reading and Bookmarks

Cursor pagination is like placing a bookmark to remember your reading spot.

This cross-domain link shows how remembering a position is a universal solution to navigating large sequences.

Common Pitfalls

#1 Including the cursor item in the next page results, causing duplicates.

Wrong approach:

query.startAfter(cursor).limit(5) // but cursor is the last item fetched, expecting it to be included

Correct approach:

query.startAfter(cursor).limit(5) // cursor excluded, fetches next items only

Root cause:

Misunderstanding that startAfter excludes the cursor item, leading to repeated data.

#2 Using cursor pagination without ordering the query.

Wrong approach:

collectionRef.limit(5).startAfter(cursor) // no orderBy used

Correct approach:

collectionRef.orderBy('timestamp').limit(5).startAfter(cursor)

Root cause:

Not realizing that Firestore requires ordered queries for cursor methods to work.

#3 Not handling deleted documents that were used as cursors.

Wrong approach:

Using a deleted document snapshot as cursor without checking, causing empty pages.

Correct approach:

Check if cursor document exists; if not, reset pagination or fetch from start.

Root cause:

Assuming cursors always point to existing documents, ignoring data changes.

Key Takeaways

Cursor-based pagination uses a specific document as a marker to fetch the next or previous set of data reliably.

startAfter and endBefore exclude the cursor item, preventing duplicates in paginated results.

Ordering queries is essential for cursor pagination to work correctly and consistently.

Cursor pagination handles changing data better than offset pagination, making it ideal for live apps.

Advanced use includes optimizing queries with indexes, handling edge cases, and improving user experience with caching and prefetching.