Pagination patterns in FastAPI - Deep Dive

Overview - Pagination patterns

What is it?

Pagination patterns are ways to split large sets of data into smaller, manageable pieces called pages. This helps users and systems handle data without overload. In FastAPI, pagination controls how many items show per page and which page to display. It makes APIs faster and easier to use.

Why it matters

Without pagination, APIs would try to send all data at once, causing slow responses and crashes. Imagine trying to read a huge book all at once instead of chapter by chapter. Pagination solves this by breaking data into chunks, improving speed and user experience. It also reduces server load and bandwidth use.

Where it fits

Before learning pagination, you should understand how APIs work and how to handle data responses in FastAPI. After mastering pagination, you can learn about filtering, sorting, and caching to make APIs even more efficient.

Mental Model

Core Idea

Pagination is like dividing a long list into smaller pages so you can view and manage data step-by-step without overload.

Think of it like...

Think of a photo album with many pictures. Instead of looking at all photos at once, you flip through pages, seeing a few pictures per page. Pagination in APIs works the same way, showing data in small groups.

┌───────────────┐
│ Full Data Set │
└──────┬────────┘
       │ Split into pages
       ▼
┌───────────┐ ┌───────────┐ ┌───────────┐
│ Page 1   │ │ Page 2   │ │ Page 3   │
│ Items 1-5│ │ Items 6-10│ │ Items 11-15│
└───────────┘ └───────────┘ └───────────┘

Build-Up - 7 Steps

1

Foundation

Understanding basic pagination concept

Concept: Learn what pagination means and why it is used in APIs.

Pagination means splitting data into smaller parts called pages. Each page shows a limited number of items. For example, if you have 100 items and want 10 per page, you get 10 pages. This helps users get data faster and easier.

Result

You understand that pagination breaks big data into smaller, easier-to-handle chunks.

Understanding pagination as data chunking helps you see why it improves speed and usability.

2

Foundation

FastAPI basics for pagination

Concept: Learn how FastAPI handles query parameters to control pagination.

FastAPI lets you get query parameters from URLs, like ?page=2&size=10. These tell the API which page and how many items per page to return. You use function parameters with default values to read these.

Result

You can accept page number and page size from API requests in FastAPI.

Knowing how to read query parameters is key to making pagination work in FastAPI.

3

Intermediate

Implementing offset-limit pagination

🤔Before reading on: do you think offset-limit pagination uses page numbers or item positions? Commit to your answer.

Concept: Offset-limit pagination uses an offset (start position) and limit (number of items) to fetch data slices.

Offset-limit pagination means you skip a number of items (offset) and then take a limited number (limit). For example, offset=10 and limit=5 means skip first 10 items, then take next 5. In FastAPI, you get offset and limit from query parameters and slice your data accordingly.

Result

You can return any slice of data based on offset and limit, enabling flexible pagination.

Understanding offset-limit helps you control exactly which data slice to return, useful for many APIs.

4

Intermediate

Using page-number and size pagination

🤔Before reading on: does page-number pagination calculate offset internally or require manual offset? Commit to your answer.

Concept: Page-number pagination uses page number and page size to calculate which items to return.

Instead of offset, you use page number and size. For example, page=3 and size=10 means items 21 to 30. You calculate offset as (page-1)*size. FastAPI reads page and size from query parameters, then slices data using this offset and size.

Result

You can implement user-friendly pagination with page numbers, common in web apps.

Knowing how to convert page numbers to offsets bridges user input and data slicing.

5

Intermediate

Adding metadata to pagination responses

🤔Before reading on: do you think pagination responses should include total item count? Commit to your answer.

Concept: Pagination responses often include extra info like total items, current page, and total pages.

Besides data items, APIs send metadata so clients know how many pages exist and where they are. For example, response can have total_items, total_pages, current_page, and items list. This helps clients build navigation controls.

Result

Your API responses become more informative and easier to use for clients.

Including metadata improves user experience by enabling clear navigation and progress tracking.

6

Advanced

Cursor-based pagination for large datasets

🤔Before reading on: do you think cursor pagination uses page numbers or unique item markers? Commit to your answer.

Concept: Cursor pagination uses a unique marker (cursor) to fetch the next set of items, avoiding offset performance issues.

Instead of offset, cursor pagination uses a unique value from the last item (like an ID or timestamp) as a cursor. The API returns items after this cursor. This is faster for large or changing datasets because it avoids skipping items. FastAPI can accept a cursor parameter and query data accordingly.

Result

You can handle large or real-time data efficiently with cursor pagination.

Understanding cursor pagination helps you build scalable APIs that perform well with big data.

7

Expert

Handling pagination edge cases and performance

🤔Before reading on: do you think offset pagination always performs well on large tables? Commit to your answer.

Concept: Learn about performance pitfalls and edge cases like data changes during pagination and large offsets.

Offset pagination can slow down with large offsets because the database must skip many rows. Also, if data changes between requests, users may see duplicates or miss items. Cursor pagination solves some issues but requires stable sorting keys. FastAPI developers must choose the right pattern and handle errors like invalid cursors or page numbers gracefully.

Result

You can design robust pagination that handles real-world data and performance challenges.

Knowing limitations and tradeoffs prevents common bugs and poor user experiences in production.

Under the Hood

Pagination works by limiting the data query to a subset of the full dataset. Offset-limit pagination uses SQL OFFSET and LIMIT clauses to skip and fetch rows. Cursor pagination uses WHERE clauses with comparison operators on unique keys to fetch items after a cursor. FastAPI reads query parameters and passes them to database queries or data slices. The database engine executes these queries efficiently using indexes.

Why designed this way?

Pagination was designed to avoid sending huge data at once, which slows networks and clients. Offset-limit is simple and widely supported but can be slow on large datasets. Cursor pagination was introduced to improve performance and consistency in dynamic data. FastAPI supports flexible query parameter parsing to let developers implement these patterns easily.

┌───────────────┐
│ Client Request│
│ ?page=2&size=5│
└──────┬────────┘
       │
       ▼
┌───────────────┐
│ FastAPI Server│
│ Parses params │
│ Calculates    │
│ offset = (2-1)*5=5│
└──────┬────────┘
       │
       ▼
┌───────────────┐
│ Database Query│
│ SELECT * FROM │
│ table LIMIT 5 │
│ OFFSET 5      │
└──────┬────────┘
       │
       ▼
┌───────────────┐
│ Data Returned │
│ Items 6 to 10 │
└───────────────┘

Myth Busters - 4 Common Misconceptions

Quick: Does offset pagination always perform well on very large datasets? Commit to yes or no.

Common Belief: Offset pagination is always fast and simple for any dataset size.

Reality: Offset pagination slows down on large datasets because the database must scan and skip many rows before returning results.

Why it matters: Using offset pagination on big tables can cause slow API responses and poor user experience.

Quick: Is cursor pagination just a fancy name for offset pagination? Commit to yes or no.

Common Belief: Cursor pagination is the same as offset pagination but with a different name.

Reality: Cursor pagination uses unique item markers to fetch data after a point, not numeric offsets, improving performance and consistency.

Why it matters: Confusing these leads to choosing wrong pagination, causing bugs or slow APIs.

Quick: Does including total item count in pagination responses always cost a lot of performance? Commit to yes or no.

Common Belief: Calculating total items for pagination metadata is always expensive and should be avoided.

Reality: While counting can be costly on huge datasets, many databases optimize counts, and caching can help. Omitting counts reduces usability.

Why it matters: Skipping total counts can confuse users about how many pages exist, hurting navigation.

Quick: Can pagination alone guarantee consistent data views if data changes during navigation? Commit to yes or no.

Common Belief: Pagination ensures users always see consistent data even if the dataset changes.

Reality: Pagination does not guarantee consistency if data is added, removed, or updated between requests; additional strategies are needed.

Why it matters: Ignoring this can cause users to see duplicate or missing items, leading to confusion.

Expert Zone

1

Cursor pagination requires stable and unique sorting keys to avoid missing or repeating items during navigation.

2

Offset pagination can be combined with caching layers to reduce database load for popular pages.

3

Including pagination metadata like next and previous links improves API usability and aligns with REST best practices.

When NOT to use

Avoid offset pagination on very large or frequently changing datasets; use cursor pagination instead. For simple or small datasets, basic page-number pagination is sufficient. If you need random access to any page quickly, offset pagination may be better despite performance costs.

Production Patterns

In production, APIs often combine pagination with filtering and sorting. Cursor pagination is common in social media feeds for smooth scrolling. Offset pagination is used in admin dashboards with stable datasets. Metadata and links are included for client navigation. Error handling for invalid pages or cursors is standard.

Connections

Database indexing

Pagination performance depends on how well the database indexes support fast data retrieval.

Understanding indexing helps optimize pagination queries and avoid slow responses.

User interface design

Pagination patterns influence how users navigate data in apps and websites.

Knowing pagination helps UI designers create better navigation controls and improve user experience.

Memory management in operating systems

Both pagination in APIs and memory pagination divide large resources into smaller chunks for efficient handling.

Recognizing this shared pattern shows how breaking big tasks into smaller parts is a universal solution in computing.

Common Pitfalls

#1 Using offset pagination without limits, causing huge data loads.

Wrong approach:

def get_items(offset: int = 0):
return database.get_all_items()[offset:]

Correct approach:

def get_items(offset: int = 0, limit: int = 10):
return database.get_all_items()[offset:offset+limit]

Root cause:

Not limiting the number of items returned causes the API to send too much data, slowing down responses.

#2 Calculating total item count on every request without caching.

Wrong approach:

total = database.count_items()
items = database.get_items(offset, limit)
return {"total": total, "items": items}

Correct approach:

# Cache total count or update periodically
cached_total = cache.get('total_count')
if cached_total is None:
cached_total = database.count_items()
cache.set('total_count', cached_total, ttl=60)
items = database.get_items(offset, limit)
return {"total": cached_total, "items": items}

Root cause:

Counting all items on every request can be expensive; caching reduces load and improves speed.

#3 Using cursor pagination without stable sorting keys.

Wrong approach:

def get_items(cursor: str = None):
if cursor:
return database.query('WHERE name > ?', cursor)
else:
return database.query('')

Correct approach:

def get_items(cursor: str = None):
# Use unique, indexed key like created_at or id
if cursor:
return database.query('WHERE id > ?', cursor)
else:
return database.query('')

Root cause:

Using non-unique or unstable keys causes missing or duplicated items during pagination.

Key Takeaways

Pagination breaks large data into smaller pages to improve API speed and usability.

FastAPI handles pagination by reading query parameters like page, size, offset, or cursor.

Offset-limit and page-number are common pagination patterns; cursor pagination is better for large or changing data.

Including metadata like total items and current page helps clients navigate data easily.

Understanding pagination limitations and performance tradeoffs is key to building robust APIs.