Search after for efficient pagination in Elasticsearch - Step-by-Step Execution

Concept Flow - Search after for efficient pagination

Initial Search Request

↓

Get Sorted Results

↓

Extract Last Sort Values

↓

Use search_after with Last Sort Values

↓

Get Next Page Results

↓

Repeat for More Pages or Stop

The flow shows how to get the first page, then use the last item's sort values to fetch the next page efficiently.

Execution Sample

Elasticsearch

GET /my_index/_search
{
  "size": 3,
  "sort": [{"date": "asc"}, {"_id": "asc"}],
  "search_after": ["2023-01-01T00:00:00", "doc_3"]
}

This query fetches 3 documents after the document with date '2023-01-01T00:00:00' and id 'doc_3', sorted by date and id.

Execution Table

Step	Action	Sort Values Used	Result	Next Step
1	Initial search without search_after	None	Returns first 3 docs sorted by date asc, id asc	Extract last doc's sort values
2	Extract last doc's sort values	Last doc sort: ["2023-01-01T00:00:00", "doc_3"]	Ready for next page query	Use search_after with these values
3	Search with search_after	["2023-01-01T00:00:00", "doc_3"]	Returns next 3 docs after last doc	Repeat or stop
4	Repeat steps 2-3	Updated last doc's sort values	Fetch next pages efficiently	Stop when no more docs

💡 No more documents after last search_after values, pagination ends

Variable Tracker

Variable	Start	After Step 1	After Step 2	After Step 3	Final
search_after	None	None	["2023-01-01T00:00:00", "doc_3"]	[Updated last doc's sort values]	None (end)
results	Empty	[doc_1, doc_2, doc_3]	[doc_4, doc_5, doc_6]	[Next page docs]	Empty (end)

Key Moments - 3 Insights

Why do we need to use the last document's sort values for search_after?

Because search_after requires the exact sort values of the last document from the previous page to fetch the next page correctly, as shown in step 2 of the execution_table.

Can we use search_after without sorting?

No, search_after only works with sorted results because it uses the sort values to know where to continue, as seen in the code sample and step 3.

What happens if we use search_after with wrong or missing sort values?

The query will not return the correct next page and may skip or repeat documents, breaking pagination, as implied by the importance of step 2.

Visual Quiz - 3 Questions

Look at the execution_table, what are the sort values used in step 3?

A ["2023-01-02T00:00:00", "doc_6"]

B None

C ["2023-01-01T00:00:00", "doc_3"]

D ["doc_3", "2023-01-01T00:00:00"]

💡 Hint

Check the 'Sort Values Used' column in step 3 of the execution_table.

At which step does the system extract the last document's sort values?

A Step 2

B Step 1

C Step 3

D Step 4

💡 Hint

Look for the step where 'Extract last doc's sort values' is described in the execution_table.

If the initial search size changes from 3 to 5, how does the variable 'results' change after step 1?

A It will contain 3 documents

B It will contain 5 documents

C It will be empty

D It will contain 1 document

💡 Hint

Refer to variable_tracker 'results' after step 1 and consider the 'size' parameter in the code sample.

Concept Snapshot

Search after pagination uses the last document's sort values to fetch the next page.
Always sort your results to use search_after.
Extract last doc's sort values from previous page.
Use search_after with those values in next query.
Efficient for deep pagination without performance loss.

Full Transcript

This visual execution shows how Elasticsearch's search_after works for efficient pagination. First, an initial search fetches a page of sorted results. Then, the last document's sort values are extracted. These values are used in the next search_after query to get the next page. This process repeats until no more documents remain. The key is that search_after requires sorted results and the exact last sort values to continue pagination correctly. The execution table traces each step, showing actions, sort values, and results. Variable tracking shows how search_after and results change over steps. Common confusions include why sort values are needed and that search_after cannot work without sorting. The quiz tests understanding of these steps and values. This method avoids the performance issues of deep pagination with from/size.

Practice

1. What is the main purpose of using search_after in Elasticsearch pagination?

easy

A. To filter documents based on a query

B. To sort documents alphabetically by default

C. To efficiently paginate through large result sets without performance loss

D. To update documents in bulk

Solution

1. Step 1: Understand pagination challenges

   Deep pagination with large result sets can be slow and inefficient using traditional methods like from and size.

2. Step 2: Role of search_after

   search_after uses the last sort values from the previous page to fetch the next page efficiently, avoiding performance issues.

3. Final Answer:

   To efficiently paginate through large result sets without performance loss -> Option C

4. Quick Check:

   Purpose of search_after = Efficient pagination [OK]

Hint: Remember: search_after uses last sort values for fast paging [OK]

Common Mistakes:

• Confusing search_after with filtering
• Thinking search_after sorts results automatically
• Using search_after without sorting

2. Which of the following is the correct syntax snippet to use search_after in an Elasticsearch query?

easy

A. "search_after": ["last_sort_value"]

B. "search_after": "last_sort_value"

C. "search_after": {"value": "last_sort_value"}

D. "search_after": true

Solution

1. Step 1: Check the expected data type for search_after

   The search_after parameter expects an array of sort values, not a single string or object.

2. Step 2: Match syntax with correct format

   "search_after": ["last_sort_value"] correctly shows search_after as an array with the last sort value inside.

3. Final Answer:

   "search_after": ["last_sort_value"] -> Option A

4. Quick Check:

   search_after syntax = array of values [OK]

Hint: search_after always takes an array of sort values [OK]

Common Mistakes:

• Passing a single string instead of an array
• Using an object instead of an array
• Setting search_after to a boolean

3. Given this Elasticsearch query snippet, what will be the effect of adding "search_after": [1627891234567]?

{
  "size": 5,
  "sort": [{"timestamp": "asc"}],
  "search_after": [1627891234567]
}

medium

A. It causes a syntax error because search_after is not allowed here

B. It returns the first 5 documents sorted by timestamp ascending

C. It returns 5 documents with timestamp less than or equal to 1627891234567

D. It returns 5 documents with timestamp strictly greater than 1627891234567

Solution

1. Step 1: Understand sorting and search_after usage

   The query sorts documents by timestamp ascending and uses search_after with a timestamp value.

2. Step 2: Effect of search_after value

   search_after tells Elasticsearch to return documents after the given sort value, so only documents with timestamp greater than 1627891234567 are returned.

3. Final Answer:

   It returns 5 documents with timestamp strictly greater than 1627891234567 -> Option D

4. Quick Check:

   search_after filters results after given sort value [OK]

Hint: search_after returns results after the given sort values [OK]

Common Mistakes:

• Thinking it returns documents before the value
• Assuming it returns the first page always
• Confusing search_after with from/size pagination

4. You wrote this Elasticsearch query to paginate results:

{
  "size": 10,
  "sort": [{"date": "desc"}],
  "search_after": "2023-01-01T00:00:00"
}

But it returns an error. What is the likely cause?

medium

A. size cannot be 10 with search_after

B. search_after value must be an array, not a string

C. sort order must be ascending for search_after

D. date field cannot be used in sort

Solution

1. Step 1: Check the type of search_after value

   The search_after parameter requires an array of values, but here it is a string.

2. Step 2: Identify the error cause

   Passing a string instead of an array causes a syntax error in the query.

3. Final Answer:

   search_after value must be an array, not a string -> Option B

4. Quick Check:

   search_after requires array input [OK]

Hint: Always wrap search_after values in an array [OK]

Common Mistakes:

• Passing single value without array brackets
• Using unsupported sort order
• Misunderstanding size limits with search_after

5. You want to paginate through a large dataset sorted by user_id (ascending) and then timestamp (descending). Which search_after value correctly fetches the next page after user_id=42 and timestamp=1680000000?

hard

A. [42, 1680000000]

B. [42, -1680000000]

C. [1680000000, 42]

D. ["42", "1680000000"]

Solution

1. Step 1: Understand sort order and search_after values

   The sort is by user_id ascending, then timestamp descending. The search_after array must match this order.

2. Step 2: Match values to sort order

   The correct search_after is an array with user_id first, then timestamp. Since timestamp is descending, the value is used as is (no negation).

3. Final Answer:

   [42, 1680000000] -> Option A

4. Quick Check:

   search_after array matches sort fields order [OK]

Hint: search_after array order matches sort fields order exactly [OK]

Common Mistakes:

• Reversing order of values in search_after
• Negating timestamp for descending sort
• Using strings instead of numbers without need