Why Scroll API for deep pagination in Elasticsearch? - Purpose & Use Cases

The Big Idea

What if you could flip through millions of records without ever losing your spot or missing a single one?

The Scenario

Imagine you have a huge library catalog with millions of books, and you want to look through every single one page by page. Trying to manually keep track of where you left off each time is like flipping through endless pages without a bookmark.

The Problem

Manually paginating large data sets is slow and error-prone because you have to remember the last position, and each new page request can be inconsistent if the data changes. This leads to missing or repeated results, making the process frustrating and unreliable.

The Solution

The Scroll API acts like a smart bookmark that remembers exactly where you stopped. It keeps a stable view of the data and lets you fetch large amounts of results efficiently, without losing your place or missing any entries.

Before vs After

✗ Before

GET /index/_search?from=10000&size=10
{
  "query": { "match_all": {} }
}

✓ After

POST /_search/scroll
{
  "scroll": "1m",
  "scroll_id": "DXF1ZXJ5QW5kRmV0Y2gBAAAAAAA"
}

What It Enables

It enables reliable and efficient retrieval of deep pages of data, even when dealing with millions of records.

Real Life Example

A company wants to export all customer records from their database for analysis. Using the Scroll API, they can fetch all records in batches without missing or duplicating any, even if the database is huge.

Key Takeaways

Manual pagination struggles with large data and changing content.

Scroll API keeps a consistent snapshot and remembers your place.

It makes deep data retrieval fast, reliable, and easy.