Page-based pagination in Rest API - Deep Dive

Overview - Page-based pagination

What is it?

Page-based pagination is a method used in APIs to split large sets of data into smaller, manageable chunks called pages. Instead of sending all data at once, the server sends one page at a time, and the client can request specific pages by number. This helps improve performance and user experience by loading data gradually.

Why it matters

Without pagination, APIs would try to send all data in one response, which can be very slow and overwhelming for both servers and users. Large responses can cause delays, crashes, or timeouts. Pagination solves this by breaking data into smaller pieces, making apps faster and more responsive, especially on slow networks or devices.

Where it fits

Before learning page-based pagination, you should understand basic API requests and responses. After mastering pagination, you can explore other data retrieval methods like cursor-based pagination or infinite scrolling techniques.

Mental Model

Core Idea

Page-based pagination divides data into numbered pages so clients can request and receive data in small, ordered chunks.

Think of it like...

It's like reading a book one page at a time instead of trying to read the whole book at once. You can jump to any page number to find the part you want without carrying the entire book.

┌───────────────┐
│   Full Data   │
└──────┬────────┘
       │ Split into pages
       ▼
┌──────┬───────┬───────┐
│Page 1│Page 2 │Page 3 │ ...
└──────┴───────┴───────┘
Client requests page by number → Server returns that page's data

Build-Up - 7 Steps

1

Foundation

Understanding API data overload

Concept: Large data sets can overwhelm API responses and clients.

Imagine an API that returns 10,000 user records in one response. This huge response can be slow to send, slow to receive, and hard for the client to process all at once. This overload can cause poor app performance or crashes.

Result

Sending all data at once is inefficient and can cause delays or failures.

Knowing why large responses are problematic motivates the need for pagination.

2

Foundation

Basic idea of splitting data into pages

Concept: Data can be divided into smaller groups called pages for easier handling.

Instead of sending all 10,000 records, the API can send 100 records per page. The client asks for page 1 to get the first 100, page 2 for the next 100, and so on. This reduces load and speeds up data transfer.

Result

Data is delivered in smaller, manageable chunks improving speed and usability.

Breaking data into pages is the core concept behind page-based pagination.

3

Intermediate

Requesting pages by number

🤔Before reading on: Do you think page numbers start at 0 or 1? Commit to your answer.

Concept: Clients specify which page number they want to receive from the server.

Most APIs use a query parameter like ?page=1 to request the first page. The server calculates which records to send based on the page number and page size (e.g., 100 items per page). For example, page 1 returns items 1-100, page 2 returns 101-200, etc.

Result

Clients can control which part of the data they get by changing the page number.

Understanding page numbering and calculation is key to using page-based pagination correctly.

4

Intermediate

Using page size to control data amount

🤔Before reading on: Should the page size be fixed by the server or flexible by the client? Commit to your answer.

Concept: Page size defines how many items each page contains and can be fixed or adjustable.

Some APIs fix the page size (e.g., always 50 items per page), while others let clients specify it with a parameter like ?page_size=20. Allowing clients to choose page size gives flexibility but requires limits to avoid abuse.

Result

Page size controls how much data is sent per request, balancing speed and completeness.

Knowing how page size works helps optimize data loading and user experience.

5

Intermediate

Handling total counts and last page

🤔Before reading on: Do you think the API always knows the total number of pages? Commit to your answer.

Concept: APIs often provide total item counts to help clients understand how many pages exist.

The server can include a total count of items in the response metadata. Clients use this to calculate total pages (total items ÷ page size). This helps clients show navigation controls like 'Page 3 of 50'. Sometimes, total counts are expensive to calculate and may be omitted.

Result

Clients get context about data size and can build better navigation UI.

Knowing total counts improves user interface design and user expectations.

6

Advanced

Dealing with data changes during pagination

🤔Before reading on: If data changes between page requests, do you think page-based pagination always shows consistent results? Commit to your answer.

Concept: Data can change while clients paginate, causing inconsistencies or missing items.

If new items are added or removed between page requests, clients might see duplicates or miss some data. To handle this, APIs can use stable sorting, timestamps, or snapshot versions to keep pagination consistent during a session.

Result

Without precautions, pagination can show confusing or incorrect data sequences.

Understanding data volatility during pagination is crucial for building reliable APIs.

7

Expert

Limitations and performance trade-offs

🤔Before reading on: Is page-based pagination always the best choice for large, frequently changing data? Commit to your answer.

Concept: Page-based pagination has limits in performance and consistency for very large or dynamic data sets.

Page-based pagination requires the server to skip records to reach the requested page, which can be slow on large data sets. Also, it can cause inconsistent views if data changes. Alternatives like cursor-based pagination solve some issues but add complexity.

Result

Page-based pagination is simple but may not scale well or guarantee perfect consistency.

Knowing the limits helps choose the right pagination method for your API's needs.

Under the Hood

When a client requests a page number, the server calculates the starting record by multiplying (page number - 1) by the page size. It then retrieves that slice of data from the database or data store. The server packages this slice along with metadata like total count or next page info and sends it back. Internally, the database query often uses OFFSET and LIMIT clauses to fetch the correct records.

Why designed this way?

Page-based pagination was designed for simplicity and ease of use. It allows clients to jump to any page by number without needing complex state. Early web apps and APIs used this method because it matched how users think about pages. Although alternatives exist, page-based pagination remains popular due to its straightforwardness and compatibility.

Client Request: GET /items?page=3&page_size=10
          │
          ▼
  Server calculates offset = (3-1)*10 = 20
          │
          ▼
  Database query: SELECT * FROM items LIMIT 10 OFFSET 20
          │
          ▼
  Server sends items 21-30 + metadata
          │
          ▼
  Client receives page 3 data

Myth Busters - 4 Common Misconceptions

Quick: Does requesting page 0 return the first page? Commit to yes or no.

Common Belief: Page numbering starts at 0, so page 0 is the first page.

Reality: Most APIs start page numbering at 1, so page 1 is the first page. Requesting page 0 often returns an error or empty data.

Why it matters: Using wrong page numbers causes failed requests or confusing empty results.

Quick: If you request page 2 twice, will you get the exact same data both times? Commit to yes or no.

Common Belief: Page-based pagination always returns consistent data for the same page number.

Reality: If data changes between requests, the content of a page can differ, causing duplicates or missing items.

Why it matters: Assuming consistency can lead to bugs in apps that rely on stable data views.

Quick: Does increasing page size always improve performance? Commit to yes or no.

Common Belief: Larger page sizes always make data loading faster because fewer requests are needed.

Reality: Larger pages mean bigger responses, which can slow down network transfer and client processing, hurting performance.

Why it matters: Choosing too large a page size can degrade user experience instead of improving it.

Quick: Is page-based pagination the best method for all API data retrieval? Commit to yes or no.

Common Belief: Page-based pagination is the best and only way to paginate API data.

Reality: Other methods like cursor-based pagination exist and can be better for large or frequently changing data sets.

Why it matters: Using page-based pagination blindly can cause scalability and consistency problems in complex systems.

Expert Zone

1

Page-based pagination can cause performance issues on large data sets because OFFSET queries require scanning skipped rows, which grows linearly with page number.

2

APIs sometimes omit total counts to improve performance, forcing clients to infer when they've reached the last page by receiving fewer items than the page size.

3

Combining page-based pagination with stable sorting keys or timestamps helps reduce data inconsistency during pagination but requires careful API design.

When NOT to use

Avoid page-based pagination for very large or rapidly changing data sets where consistency and performance are critical. Instead, use cursor-based pagination, which uses a pointer to the last item seen and fetches the next set without OFFSET. For infinite scrolling interfaces, cursor pagination provides smoother user experience.

Production Patterns

In real-world APIs, page-based pagination is often combined with filtering and sorting parameters. Servers enforce maximum page sizes to prevent abuse. Pagination metadata includes links to next, previous, first, and last pages for easy navigation. Some APIs provide both page and cursor pagination options to support different client needs.

Connections

Cursor-based pagination

Alternative pagination method with different trade-offs

Understanding page-based pagination helps appreciate why cursor-based pagination was created to solve performance and consistency issues.

Database OFFSET and LIMIT clauses

Underlying database operations that implement page-based pagination

Knowing how OFFSET and LIMIT work clarifies why large page numbers can slow down queries.

Library book indexing

Similar concept of dividing a large collection into numbered sections for easy access

Recognizing how libraries organize books into numbered shelves and sections helps understand how pagination organizes data for retrieval.

Common Pitfalls

#1 Requesting page 0 instead of page 1

Wrong approach:

GET /items?page=0&page_size=10

Correct approach:

GET /items?page=1&page_size=10

Root cause:

Misunderstanding that page numbering usually starts at 1, not 0.

#2 Assuming data is stable between page requests

Wrong approach:

Fetching page 1, then page 2, expecting no duplicates or missing items without handling data changes

Correct approach:

Implement stable sorting or use timestamps to ensure consistent pagination results despite data changes

Root cause:

Ignoring that data can change between requests causing inconsistent pagination views.

#3 Setting page size too large causing slow responses

Wrong approach:

GET /items?page=1&page_size=10000

Correct approach:

GET /items?page=1&page_size=100

Root cause:

Believing bigger page sizes always improve performance without considering network and client processing limits.

Key Takeaways

Page-based pagination splits large data sets into numbered pages to improve API performance and user experience.

Clients request specific pages by number, and servers calculate which data slice to return using page size and page number.

Page numbering usually starts at 1, and page size controls how many items each page contains.

Data changes between requests can cause inconsistent results, so stable sorting or other techniques are needed for reliability.

Page-based pagination is simple and widely used but has performance and consistency limits for very large or dynamic data sets.