Search after for efficient pagination in Elasticsearch - Deep Dive

Overview - Search after for efficient pagination

What is it?

Search after is a method in Elasticsearch that helps you get results page by page without slowing down. Instead of jumping to a page by counting all previous results, it uses the last result from the previous page to continue. This makes it faster and more efficient for large lists of data. It is especially useful when you want to scroll through many results smoothly.

Why it matters

Without search after, paginating large data sets can become very slow and costly because Elasticsearch has to count and skip many results for each page. This can make websites or apps lag or even crash when users browse deep into results. Search after solves this by remembering where you left off, so the system only fetches what you need next. This improves user experience and saves computing resources.

Where it fits

Before learning search after, you should understand basic Elasticsearch queries and how pagination works with from and size parameters. After mastering search after, you can explore scroll API for deep scrolling and learn about search contexts and performance tuning in Elasticsearch.

Mental Model

Core Idea

Search after uses the last seen result as a bookmark to fetch the next page efficiently without counting all previous results.

Think of it like...

Imagine reading a book with a bookmark. Instead of flipping through all pages to find where you left off, you just open the book at the bookmark and continue reading. Search after works the same way for search results.

┌───────────────┐     ┌───────────────┐     ┌───────────────┐
│ Page 1       │ --> │ Page 2       │ --> │ Page 3       │
│ Results 1-10 │     │ Results 11-20│     │ Results 21-30│
└───────────────┘     └───────────────┘     └───────────────┘
       ▲                    ▲                    ▲
       │                    │                    │
   Bookmark             Bookmark             Bookmark
(last result sort)   (last result sort)   (last result sort)

Build-Up - 6 Steps

1

Foundation

Basic pagination with from and size

Concept: Learn how Elasticsearch paginates results using from and size parameters.

In Elasticsearch, you can get a page of results by telling it how many results to skip (from) and how many to return (size). For example, from=0 and size=10 returns the first 10 results. From=10 and size=10 returns the next 10 results (results 11-20). This is simple but can be slow for large from values.

Result

You get the requested page of results, but performance slows down as from increases.

Understanding basic pagination shows why skipping many results is inefficient and sets the stage for better methods.

2

Foundation

Sorting results for consistent order

Concept: Sorting results is necessary to use search after correctly.

To use search after, you must sort your results by one or more fields. Sorting ensures the order of results is stable and predictable. For example, sorting by timestamp and then by ID guarantees a unique order. Without sorting, search after cannot know where to continue.

Result

Results are returned in a fixed order, which is essential for pagination.

Knowing that sorting creates a stable order is key to understanding how search after bookmarks work.

3

Intermediate

Introducing search after parameter

🤔Before reading on: do you think search after skips results like from, or does it use a different method? Commit to your answer.

Concept: Search after uses the last result's sort values to fetch the next page, not by skipping results.

Instead of telling Elasticsearch to skip a number of results, search after sends the sort values of the last result from the previous page. Elasticsearch then returns results after that point. This avoids counting or skipping results internally, making it faster for deep pages.

Result

You get the next page of results efficiently, even for large data sets.

Understanding that search after uses a bookmark instead of skipping results explains why it is more efficient.

4

Intermediate

Using search after with multi-field sorting

🤔Before reading on: do you think search after works with just one sort field or multiple? Commit to your answer.

Concept: Search after requires all sort fields to be included in the search after values to continue correctly.

When sorting by multiple fields, you must provide search after values for each field in order. For example, if sorting by timestamp and ID, the search after array must include the last timestamp and last ID from the previous page. This ensures the next page starts exactly after the last result.

Result

Pagination continues smoothly and correctly with complex sorting.

Knowing that search after needs all sort values prevents errors and ensures accurate pagination.

5

Advanced

Limitations and performance considerations

🤔Before reading on: do you think search after can jump to any page instantly like from? Commit to your answer.

Concept: Search after cannot jump to arbitrary pages; it only moves forward from the last position, but it is more efficient for deep pagination.

Unlike from, search after does not allow jumping directly to page 10 without fetching pages 1 to 9 first. This is because it needs the last result's sort values to continue. However, it avoids the costly skipping of results, making it faster for scrolling through many pages. It is best for sequential access.

Result

You get fast, stable pagination for deep scrolling but lose random page access.

Understanding search after's tradeoff between speed and random access helps choose the right pagination method.

6

Expert

Handling changes in data during pagination

🤔Before reading on: do you think search after results stay consistent if data changes during pagination? Commit to your answer.

Concept: Search after pagination can be affected by data changes, so strategies are needed to maintain consistency.

If data is added, deleted, or updated while paginating with search after, results may shift or be missed. To handle this, you can use a point-in-time (PIT) context in Elasticsearch, which freezes the index state for consistent pagination. Without PIT, results might be inconsistent or duplicated.

Result

Pagination remains stable and consistent even if data changes during browsing.

Knowing how to maintain consistency with PIT prevents confusing user experiences and data errors.

Under the Hood

Elasticsearch sorts documents by specified fields and stores the sort values. When search after is used, it sends the last document's sort values to the server. The server then uses these values as a cursor to efficiently find the next set of documents greater than those values. This avoids scanning or skipping all previous documents, reducing resource use and latency.

Why designed this way?

Traditional pagination with from and size became inefficient as data grew because skipping many results requires Elasticsearch to collect and discard them internally. Search after was designed to solve this by using a cursor-like approach, inspired by database cursor pagination, to improve performance and scalability for deep pagination.

┌───────────────┐       ┌───────────────┐       ┌───────────────┐
│ Client       │       │ Elasticsearch │       │ Data Storage  │
│ sends query  │──────▶│ receives query│──────▶│ sorted docs   │
│ with search  │       │ with search   │       │               │
│ after values │       │ after values  │       │               │
└───────────────┘       └───────────────┘       └───────────────┘
        ▲                       │                       │
        │                       │ finds docs after     │
        │                       │ given sort values    │
        │                       ▼                       ▼
┌─────────────────────────────────────────────────────────────┐
│ Elasticsearch returns next page of results starting after   │
│ the given sort values without scanning all previous results │
└─────────────────────────────────────────────────────────────┘

Myth Busters - 4 Common Misconceptions

Quick: Does search after allow jumping directly to any page like from? Commit to yes or no.

Common Belief: Search after lets you jump to any page instantly, just like from and size.

Reality: Search after only allows moving forward from the last seen result; it cannot jump to arbitrary pages without sequentially fetching previous pages.

Why it matters: Expecting random page access with search after leads to design mistakes and user frustration when pages cannot be skipped.

Quick: Do you think search after works without sorting? Commit to yes or no.

Common Belief: Search after can be used without sorting the results.

Reality: Search after requires a consistent sort order to work because it uses the last result's sort values as a cursor.

Why it matters: Using search after without sorting causes errors or unpredictable results, breaking pagination.

Quick: Does search after guarantee stable results if data changes during pagination? Commit to yes or no.

Common Belief: Search after always returns consistent results even if data changes during pagination.

Reality: Without using point-in-time contexts, search after results can shift or duplicate if data changes during pagination.

Why it matters: Ignoring data changes can cause confusing user experiences and incorrect data display.

Quick: Is search after always faster than from and size? Commit to yes or no.

Common Belief: Search after is always faster than from and size for any pagination depth.

Reality: Search after is faster for deep pagination but may not be faster for shallow pages or small result sets.

Why it matters: Misusing search after for shallow pagination can add unnecessary complexity without performance benefits.

Expert Zone

1

Search after requires all sort fields to be unique or combined with a unique field like _id to avoid missing or duplicate results.

2

Using point-in-time (PIT) contexts with search after is essential for consistent pagination in dynamic data environments.

3

Search after queries can be combined with filters and aggregations, but care must be taken to maintain sort order and performance.

When NOT to use

Do not use search after when you need random access to arbitrary pages or when the result set is small and shallow pagination is sufficient. In those cases, from and size or the scroll API might be better alternatives.

Production Patterns

In production, search after is commonly used for user-facing infinite scroll interfaces, log browsing tools, and APIs that return large ordered lists. It is paired with PIT contexts to ensure consistency and combined with multi-field sorting including unique IDs to guarantee stable pagination.

Connections

Database cursor pagination

Search after is a form of cursor pagination used in databases.

Understanding cursor pagination in databases helps grasp how search after avoids costly offset scans by using a bookmark to continue fetching results.

Infinite scrolling UI design

Search after enables efficient infinite scrolling by fetching next results smoothly.

Knowing how search after works helps frontend developers design smooth user experiences without lag or repeated data.

Version control systems

Both use a form of bookmarking to track progress through data changes.

Recognizing that search after bookmarks a position like a commit pointer in version control reveals a shared pattern of efficient state tracking.

Common Pitfalls

#1 Using search after without sorting the results.

Wrong approach:

{
"query": { "match_all": {} },
"search_after": [123]
}

Correct approach:

{
"query": { "match_all": {} },
"sort": [ {"timestamp": "asc"}, {"_id": "asc"} ],
"search_after": ["2023-01-01T00:00:00", "123"]
}

Root cause:

Search after requires a consistent sort order to know where to continue; without sorting, it cannot work.

#2 Trying to jump directly to page 10 using search after without fetching pages 1-9.

Wrong approach:

{
"query": { "match_all": {} },
"search_after": ["some_value_for_page_10"]
}

Correct approach:

Fetch pages sequentially using search after from page 1 to 9 to get the correct search_after values for page 10.

Root cause:

Search after is a cursor-based method that only moves forward from the last position; it cannot jump arbitrarily.

#3 Not using point-in-time context when data changes during pagination.

Wrong approach:

{
"query": { "match_all": {} },
"sort": ["timestamp"],
"search_after": ["2023-01-01T00:00:00"]
// no PIT context
}

Correct approach:

{
"pit": { "id": "PIT_ID", "keep_alive": "1m" },
"query": { "match_all": {} },
"sort": ["timestamp"],
"search_after": ["2023-01-01T00:00:00"]
}

Root cause:

Without PIT, data changes cause inconsistent pagination results.

Key Takeaways

Search after is a pagination method that uses the last result's sort values as a bookmark to fetch the next page efficiently.

It requires sorting results by one or more fields to create a stable and unique order for pagination.

Unlike from and size, search after does not skip results internally, making it faster for deep pagination but unsuitable for random page jumps.

To maintain consistent results when data changes, use search after together with point-in-time contexts.

Understanding search after helps build scalable, smooth user experiences for browsing large datasets.

Practice

1. What is the main purpose of using search_after in Elasticsearch pagination?

easy

A. To filter documents based on a query

B. To sort documents alphabetically by default

C. To efficiently paginate through large result sets without performance loss

D. To update documents in bulk

Solution

1. Step 1: Understand pagination challenges

   Deep pagination with large result sets can be slow and inefficient using traditional methods like from and size.

2. Step 2: Role of search_after

   search_after uses the last sort values from the previous page to fetch the next page efficiently, avoiding performance issues.

3. Final Answer:

   To efficiently paginate through large result sets without performance loss -> Option C

4. Quick Check:

   Purpose of search_after = Efficient pagination [OK]

Hint: Remember: search_after uses last sort values for fast paging [OK]

Common Mistakes:

• Confusing search_after with filtering
• Thinking search_after sorts results automatically
• Using search_after without sorting

2. Which of the following is the correct syntax snippet to use search_after in an Elasticsearch query?

easy

A. "search_after": ["last_sort_value"]

B. "search_after": "last_sort_value"

C. "search_after": {"value": "last_sort_value"}

D. "search_after": true

Solution

1. Step 1: Check the expected data type for search_after

   The search_after parameter expects an array of sort values, not a single string or object.

2. Step 2: Match syntax with correct format

   "search_after": ["last_sort_value"] correctly shows search_after as an array with the last sort value inside.

3. Final Answer:

   "search_after": ["last_sort_value"] -> Option A

4. Quick Check:

   search_after syntax = array of values [OK]

Hint: search_after always takes an array of sort values [OK]

Common Mistakes:

• Passing a single string instead of an array
• Using an object instead of an array
• Setting search_after to a boolean

3. Given this Elasticsearch query snippet, what will be the effect of adding "search_after": [1627891234567]?

{
  "size": 5,
  "sort": [{"timestamp": "asc"}],
  "search_after": [1627891234567]
}

medium

A. It causes a syntax error because search_after is not allowed here

B. It returns the first 5 documents sorted by timestamp ascending

C. It returns 5 documents with timestamp less than or equal to 1627891234567

D. It returns 5 documents with timestamp strictly greater than 1627891234567

Solution

1. Step 1: Understand sorting and search_after usage

   The query sorts documents by timestamp ascending and uses search_after with a timestamp value.

2. Step 2: Effect of search_after value

   search_after tells Elasticsearch to return documents after the given sort value, so only documents with timestamp greater than 1627891234567 are returned.

3. Final Answer:

   It returns 5 documents with timestamp strictly greater than 1627891234567 -> Option D

4. Quick Check:

   search_after filters results after given sort value [OK]

Hint: search_after returns results after the given sort values [OK]

Common Mistakes:

• Thinking it returns documents before the value
• Assuming it returns the first page always
• Confusing search_after with from/size pagination

4. You wrote this Elasticsearch query to paginate results:

{
  "size": 10,
  "sort": [{"date": "desc"}],
  "search_after": "2023-01-01T00:00:00"
}

But it returns an error. What is the likely cause?

medium

A. size cannot be 10 with search_after

B. search_after value must be an array, not a string

C. sort order must be ascending for search_after

D. date field cannot be used in sort

Solution

1. Step 1: Check the type of search_after value

   The search_after parameter requires an array of values, but here it is a string.

2. Step 2: Identify the error cause

   Passing a string instead of an array causes a syntax error in the query.

3. Final Answer:

   search_after value must be an array, not a string -> Option B

4. Quick Check:

   search_after requires array input [OK]

Hint: Always wrap search_after values in an array [OK]

Common Mistakes:

• Passing single value without array brackets
• Using unsupported sort order
• Misunderstanding size limits with search_after

5. You want to paginate through a large dataset sorted by user_id (ascending) and then timestamp (descending). Which search_after value correctly fetches the next page after user_id=42 and timestamp=1680000000?

hard

A. [42, 1680000000]

B. [42, -1680000000]

C. [1680000000, 42]

D. ["42", "1680000000"]

Solution

1. Step 1: Understand sort order and search_after values

   The sort is by user_id ascending, then timestamp descending. The search_after array must match this order.

2. Step 2: Match values to sort order

   The correct search_after is an array with user_id first, then timestamp. Since timestamp is descending, the value is used as is (no negation).

3. Final Answer:

   [42, 1680000000] -> Option A

4. Quick Check:

   search_after array matches sort fields order [OK]

Hint: search_after array order matches sort fields order exactly [OK]

Common Mistakes:

• Reversing order of values in search_after
• Negating timestamp for descending sort
• Using strings instead of numbers without need