Cursor-based pagination in Rest API - Deep Dive

Overview - Cursor-based pagination

What is it?

Cursor-based pagination is a way to split large lists of data into smaller parts, called pages, using a special marker called a cursor. Instead of counting pages by numbers, it uses a unique position in the data to know where to start the next page. This helps when data changes often or when there are many items to show. It is common in APIs to make loading data faster and smoother.

Why it matters

Without cursor-based pagination, APIs often use page numbers which can cause problems like missing or repeating items when data changes during browsing. Cursor-based pagination solves this by always knowing exactly where to continue, making user experiences more reliable and efficient. This is important for apps like social media feeds or product lists where data updates frequently.

Where it fits

Before learning cursor-based pagination, you should understand basic pagination concepts like offset and limit. After mastering cursor-based pagination, you can explore advanced API design topics like rate limiting, caching, and real-time data streaming.

Mental Model

Core Idea

Cursor-based pagination uses a unique marker to remember the last item seen, so the next page starts exactly after it, avoiding duplicates or missing data.

Think of it like...

Imagine reading a book with a bookmark. Instead of saying 'go to page 5', you say 'start reading after the bookmark'. No matter if pages are added or removed, you always know where to continue.

┌───────────────┐
│ Data List     │
│ ┌─────────┐   │
│ │ Item 1  │   │
│ │ Item 2  │   │
│ │ Item 3  │◄──┤ Cursor (bookmark)
│ │ Item 4  │   │
│ │ Item 5  │   │
│ └─────────┘   │
└───────────────┘

Next page starts after the cursor pointing to Item 3.

Build-Up - 7 Steps

1

Foundation

Understanding basic pagination

Concept: Learn what pagination is and why we split data into pages.

Pagination means dividing a big list of items into smaller groups called pages. For example, showing 10 products per page instead of all at once. This helps users load data faster and not get overwhelmed.

Result

You understand why APIs don’t send all data at once but in parts.

Knowing why we paginate helps you appreciate why cursor-based pagination exists.

2

Foundation

Offset-based pagination explained

Concept: Learn the common method of pagination using page numbers or offsets.

Offset pagination uses a number to say where to start, like 'skip 20 items, then show 10'. For example, page 3 with 10 items per page means skip 20 items and show next 10.

Result

You can request any page by calculating offset and limit.

Understanding offset pagination shows its limits and why cursor-based pagination is better for some cases.

3

Intermediate

Problems with offset pagination

🤔Before reading on: do you think offset pagination always shows consistent data even if items change? Commit to your answer.

Concept: Discover why offset pagination can cause missing or duplicate items when data changes.

If new items are added or removed while browsing pages, offset pagination can skip or repeat items. For example, if an item is deleted on page 1, page 2’s offset shifts and you might miss or see duplicates.

Result

You see that offset pagination is unreliable for frequently changing data.

Knowing these problems motivates the need for cursor-based pagination.

4

Intermediate

Cursor-based pagination basics

🤔Before reading on: do you think a cursor is a page number or a unique item marker? Commit to your answer.

Concept: Learn that cursor pagination uses a unique marker from the last item to fetch the next page.

Instead of page numbers, cursor pagination uses a cursor, often the ID or timestamp of the last item seen. The API returns items after this cursor, so it always continues from the right place.

Result

You understand how cursor pagination avoids skipping or repeating items.

Understanding the cursor as a position marker is key to grasping this pagination method.

5

Intermediate

Implementing cursor pagination in APIs

Concept: Learn how APIs accept and return cursors to paginate data.

An API request includes a cursor parameter, like ?cursor=abc123. The server finds items after that cursor and returns them with a new cursor for the next page. If no cursor is given, it returns the first page.

Result

You can call an API with a cursor to get the next set of items.

Knowing the request-response flow clarifies how cursor pagination works in practice.

6

Advanced

Handling edge cases in cursor pagination

🤔Before reading on: do you think cursor pagination can fail if data is deleted? Commit to your answer.

Concept: Explore what happens when items are deleted or duplicated and how to handle it.

If an item pointed by the cursor is deleted, the API should find the next closest item to continue. Also, cursors must be unique and stable, often using a combination of fields like timestamp and ID to avoid confusion.

Result

You understand how to keep pagination reliable even with data changes.

Knowing how to handle deletions and duplicates prevents bugs in real systems.

7

Expert

Performance and security considerations

🤔Before reading on: do you think cursors can expose sensitive data? Commit to your answer.

Concept: Learn about optimizing cursor pagination and protecting cursor data.

Cursors should be encoded or encrypted to avoid exposing internal IDs. Also, indexes on cursor fields improve query speed. Some systems use opaque cursors (random strings) to hide details. Efficient cursor pagination reduces database load and improves user experience.

Result

You know how to build fast and secure cursor pagination in production.

Understanding performance and security tradeoffs is essential for professional API design.

Under the Hood

Cursor-based pagination works by storing a unique identifier from the last item fetched, often a combination of fields like timestamp and ID. When the next page is requested, the database query uses this cursor to filter items greater than the cursor value, ensuring the next set is continuous and ordered. This avoids scanning all previous items, making queries efficient. The cursor is often encoded to keep it opaque to clients.

Why designed this way?

It was designed to solve the inconsistency and inefficiency of offset pagination, especially in dynamic datasets where items are added or removed frequently. Using a cursor tied to actual data positions ensures stable pagination. Alternatives like offset were simpler but caused data shifts and performance issues on large datasets.

┌───────────────┐
│ Client        │
│ sends cursor ─┼─▶ API Server
│               │   │
│               │   ▼
│               │ ┌───────────────┐
│               │ │ Database      │
│               │ │ Query: items  │
│               │ │ WHERE id > c  │
│               │ └───────────────┘
│               │   ▲
│               │   │
│               │ ◀─┼─ Returns items + new cursor
└───────────────┘

Myth Busters - 4 Common Misconceptions

Quick: Does cursor pagination always use simple IDs as cursors? Commit to yes or no.

Common Belief: Cursor pagination just uses the item ID as the cursor.

Reality: Cursors often combine multiple fields like timestamp and ID to ensure uniqueness and stable ordering.

Why it matters: Using only IDs can cause incorrect pagination if items have the same timestamp or if IDs are not sequential.

Quick: Can cursor pagination cause missing items if data changes? Commit to yes or no.

Common Belief: Cursor pagination completely prevents missing or duplicate items no matter what.

Reality: While it reduces these issues, if data changes between requests (like deletions), some edge cases can still cause skips unless carefully handled.

Why it matters: Ignoring edge cases can lead to confusing user experiences with missing data.

Quick: Is cursor pagination always slower than offset pagination? Commit to yes or no.

Common Belief: Cursor pagination is slower because it needs more complex queries.

Reality: Cursor pagination is usually faster because it uses indexed fields and avoids scanning large offsets.

Why it matters: Choosing offset pagination for performance reasons can cause scalability problems.

Quick: Can cursors expose sensitive information? Commit to yes or no.

Common Belief: Cursors are safe to expose as they are just IDs.

Reality: Raw cursors can reveal internal database structure or sensitive data, so they should be encoded or encrypted.

Why it matters: Exposing raw cursors can lead to security risks or data leaks.

Expert Zone

1

Cursor stability depends on consistent ordering; changing sort criteria breaks pagination continuity.

2

Opaque cursors improve security but require server-side decoding, adding complexity.

3

Combining multiple fields in cursors handles ties and ensures deterministic pagination.

When NOT to use

Avoid cursor pagination when users need random access to arbitrary pages, as it only supports sequential navigation. In such cases, offset pagination or keyset pagination with page numbers might be better.

Production Patterns

Real-world APIs use cursor pagination for infinite scrolling feeds, combining it with caching and rate limiting. They often encode cursors as base64 strings containing multiple fields and timestamps to ensure security and stability.

Connections

Keyset pagination

Cursor-based pagination is a form of keyset pagination using unique keys to navigate data.

Understanding keyset pagination helps grasp why cursors use unique fields for efficient queries.

Database indexing

Cursor pagination relies on database indexes on cursor fields for fast data retrieval.

Knowing how indexes work explains why cursor pagination scales better than offset pagination.

Bookmarks in reading

Both use a marker to remember position and continue later.

This cross-domain similarity shows how storing a position marker is a universal solution to resuming progress.

Common Pitfalls

#1 Using offset pagination for frequently changing data.

Wrong approach:

GET /items?page=3&limit=10

Correct approach:

GET /items?cursor=eyJpZCI6IjEyMyIsInRpbWUiOiIyMDIzLTA2LTAxIn0=

Root cause:

Misunderstanding that page numbers can cause data shifts when items are added or removed.

#2 Exposing raw database IDs as cursors.

Wrong approach:

cursor=12345

Correct approach:

cursor=eyJpZCI6IjEyMzQ1In0= (base64 encoded JSON)

Root cause:

Not considering security and information leakage risks.

#3 Using non-unique fields as cursors.

Wrong approach:

cursor=timestamp_only_value

Correct approach:

cursor=combined_timestamp_and_id_value

Root cause:

Ignoring that multiple items can share the same timestamp, causing pagination errors.

Key Takeaways

Cursor-based pagination uses a unique marker to remember the last item seen, enabling reliable and efficient data loading.

It solves problems of offset pagination like missing or duplicate items when data changes during browsing.

Cursors should be unique, stable, and often encoded to ensure security and correctness.

Understanding cursor pagination requires knowing basic pagination and database indexing concepts.

In production, cursor pagination is essential for smooth infinite scrolling and large dynamic datasets.