Keyset pagination for performance in Rest API

Introduction

Keyset pagination helps load data faster by using a fixed point to continue fetching results instead of counting all items each time.

When you have a large list of items and want to show them page by page without slowing down.

When users scroll through a feed or timeline and you want to load new items smoothly.

When counting total items is expensive or not needed for the user experience.

When you want to avoid skipping or repeating items when data changes during pagination.

Syntax

Rest API

GET /items?limit=10&after_id=123

limit sets how many items to fetch per page.

after_id tells the server to return items with IDs greater than this value.

Examples

Fetch 20 products with IDs greater than 50.

Rest API

GET /products?limit=20&after_id=50

Get 15 messages after message ID 200.

Rest API

GET /messages?limit=15&after_id=200

Get the first 10 posts (no cursor yet).

Rest API

GET /posts?limit=10

Sample Program

This simple REST API returns items in pages using keyset pagination. It uses the 'after_id' query to fetch items with IDs greater than the given value. If no 'after_id' is given, it returns the first page.

Rest API

from flask import Flask, request, jsonify

app = Flask(__name__)

# Sample data: list of items with id and name
items = [{'id': i, 'name': f'Item {i}'} for i in range(1, 101)]

@app.route('/items')
def get_items():
    limit = int(request.args.get('limit', 10))
    after_id = request.args.get('after_id')

    if after_id is not None:
        after_id = int(after_id)
        filtered = [item for item in items if item['id'] > after_id]
    else:
        filtered = items

    result = filtered[:limit]
    return jsonify(result)

if __name__ == '__main__':
    app.run(debug=True)

Important Notes

Keyset pagination is faster than offset pagination because it avoids counting or skipping rows.

Use a unique and indexed column (like an ID) for the cursor to keep results consistent.

Remember to handle the case when there are no more items to fetch.

Summary

Keyset pagination uses a fixed point (cursor) to fetch the next page of results efficiently.

It is great for large datasets and real-time feeds where counting all items is slow.

Use query parameters like limit and after_id to control pagination.