Cursor-based pagination in Rest API

Introduction

Cursor-based pagination helps you get data in small parts without missing or repeating items. It uses a marker (cursor) to remember where you left off.

When you want to show a long list of items in small pages on a website or app.

When data changes often and you want to avoid showing the same item twice or skipping items.

When you want faster and more reliable navigation through large datasets.

When you want to load more items as the user scrolls down (infinite scroll).

Syntax

Rest API

GET /items?limit=10&cursor=abc123

limit sets how many items to get per page.

cursor is a token that points to where to start the next page.

Examples

Get the first 5 products from the list.

Rest API

GET /products?limit=5

Get the next 5 products starting after the product with ID 100.

Rest API

GET /products?limit=5&cursor=eyJpZCI6MTAwfQ==

Sample Program

This is a simple web server using Flask. It returns items in pages. You ask for a page with a limit and an optional cursor. The server finds where to start, sends that many items, and gives a cursor for the next page.

Rest API

from flask import Flask, request, jsonify

app = Flask(__name__)

# Sample data: list of items with IDs
items = [{'id': i, 'name': f'Item {i}'} for i in range(1, 101)]

@app.route('/items')
def get_items():
    limit = int(request.args.get('limit', 10))
    cursor = request.args.get('cursor')

    # Find start index based on cursor
    start_index = 0
    if cursor:
        try:
            start_id = int(cursor)
            # Find the index of the item with id == start_id
            for idx, item in enumerate(items):
                if item['id'] == start_id:
                    start_index = idx + 1
                    break
        except ValueError:
            pass

    # Get the slice of items
    page_items = items[start_index:start_index + limit]

    # Prepare next cursor
    if len(page_items) == limit and (start_index + limit) < len(items):
        next_cursor = str(page_items[-1]['id'])
    else:
        next_cursor = None

    return jsonify({
        'items': page_items,
        'next_cursor': next_cursor
    })

if __name__ == '__main__':
    app.run(debug=True)

Important Notes

Cursor-based pagination is more reliable than page-number pagination when data changes often.

The cursor is usually an encoded value like an ID or timestamp.

Always return the next cursor in the response so the client can request the next page.

Summary

Cursor-based pagination uses a marker to get the next set of data.

It helps avoid missing or repeating items when data changes.

Clients send the cursor to get the next page, and servers return a new cursor if more data exists.

Practice

1.

What is the main purpose of cursor-based pagination in REST APIs?

easy

A. To sort data alphabetically before sending

B. To efficiently fetch the next set of data using a marker

C. To cache all data on the client side

D. To send all data in a single response

Solution

1. Step 1: Understand cursor-based pagination concept

   Cursor-based pagination uses a marker (cursor) to fetch the next set of data instead of page numbers.

2. Step 2: Identify the main purpose

   This method helps efficiently retrieve data in chunks and avoid missing or repeating items when data changes.

3. Final Answer:

   To efficiently fetch the next set of data using a marker -> Option B

4. Quick Check:

   Cursor-based pagination = fetch next data with marker [OK]

Hint: Cursor marks position to get next data chunk [OK]

Common Mistakes:

• Confusing cursor with page number
• Thinking it sends all data at once
• Assuming it sorts data alphabetically

2.

Which of the following is the correct way to include a cursor in a REST API request URL?

GET /items?____=abc123

easy

A. cursor

B. limit

C. page

D. offset

Solution

1. Step 1: Identify the query parameter for cursor

   Cursor-based pagination uses a parameter named cursor to mark the position for the next data fetch.

2. Step 2: Match the parameter in the URL

   The URL should include ?cursor=abc123 to pass the cursor value.

3. Final Answer:

   cursor -> Option A

4. Quick Check:

   Cursor parameter in URL = cursor [OK]

Hint: Cursor parameter is usually named 'cursor' [OK]

Common Mistakes:

• Using 'page' or 'offset' which are for offset pagination
• Confusing 'limit' with cursor
• Leaving out the cursor parameter

3.

Given this API response for cursor-based pagination:

{
  "data": ["item4", "item5"],
  "next_cursor": "xyz789"
}

What should the client do to get the next page of data?

medium

A. Send a request with ?cursor=xyz789

B. Send a request with ?page=2

C. Send a request with ?offset=2

D. Send a request with ?limit=2

Solution

1. Step 1: Read the response for next cursor

   The response includes a next_cursor value "xyz789" which marks the next data position.

2. Step 2: Use the cursor in the next request

   The client should send a request with ?cursor=xyz789 to get the next page.

3. Final Answer:

   Send a request with ?cursor=xyz789 -> Option A

4. Quick Check:

   Use next_cursor value as cursor parameter [OK]

Hint: Use next_cursor value as cursor in next request [OK]

Common Mistakes:

• Using page or offset parameters instead of cursor
• Ignoring next_cursor and repeating last request
• Sending limit without cursor

4.

Consider this code snippet for fetching paginated data using cursor-based pagination:

cursor = None
while True:
    response = api.get_items(cursor=cursor)
    process(response.data)
    cursor = response.next_cursor
    if not cursor:
        break

What is the likely bug in this code?

medium

A. It processes data before fetching

B. It does not check if next_cursor exists before assigning

C. It never updates the cursor value

D. It may cause an infinite loop if next_cursor is empty string

Solution

1. Step 1: Analyze the code structure

   The code fetches data, processes it, then assigns cursor = response.next_cursor without checking if next_cursor is empty string.

2. Step 2: Potential infinite loop

   If next_cursor is an empty string, the condition if not cursor will be true and break the loop, so no infinite loop occurs. But if the check is incorrect or cursor is None, it breaks. However, the main issue is if next_cursor is missing, it raises an error.

3. Step 3: Proper handling

   Should check if next_cursor exists and is truthy before assigning and breaking.

4. Final Answer:

   It may cause an infinite loop if next_cursor is empty string -> Option D

5. Quick Check:

   Check for empty or missing cursor carefully [OK]

Hint: Check for empty or missing cursor carefully [OK]

Common Mistakes:

• Assuming next_cursor always exists
• Not breaking loop on missing cursor
• Updating cursor after break

5.

You have an API that returns data with cursor-based pagination. The API returns the following responses in sequence:

1. {"data": ["a", "b"], "next_cursor": "c1"}
2. {"data": ["c", "d"], "next_cursor": "c2"}
3. {"data": ["e"], "next_cursor": null}

You want to collect all items without duplicates even if the API sometimes returns overlapping data due to data changes. Which approach is best?

hard

A. Collect all items in a list and remove duplicates after fetching all pages

B. Fetch only the first page to avoid duplicates

C. Use a set to store items as you fetch each page to avoid duplicates immediately

D. Ignore duplicates and trust API never repeats data

Solution

1. Step 1: Understand overlapping data issue

   Cursor-based pagination can return overlapping items if data changes during pagination, causing duplicates.

2. Step 2: Choose a method to avoid duplicates

   Using a set to store items as they are fetched avoids duplicates immediately and efficiently.

3. Step 3: Evaluate other options

   Removing duplicates after fetching all pages (list) is less efficient. Ignoring duplicates or fetching only first page is incorrect.

4. Final Answer:

   Use a set to store items as you fetch each page to avoid duplicates immediately -> Option C

5. Quick Check:

   Use set to avoid duplicates during fetch [OK]

Hint: Use set to track unique items during pagination [OK]

Common Mistakes:

• Assuming API never returns duplicates
• Removing duplicates only after all data fetched
• Fetching only first page to avoid duplicates