Search after for efficient pagination in Elasticsearch - Step-by-Step Execution

Concept Flow - Search after for efficient pagination

Initial Search Request

↓

Get Sorted Results

↓

Extract Last Sort Values

↓

Use search_after with Last Sort Values

↓

Get Next Page Results

↓

Repeat for More Pages or Stop

The flow shows how to get the first page, then use the last item's sort values to fetch the next page efficiently.

Execution Sample

Elasticsearch

GET /my_index/_search
{
  "size": 3,
  "sort": [{"date": "asc"}, {"_id": "asc"}],
  "search_after": ["2023-01-01T00:00:00", "doc_3"]
}

This query fetches 3 documents after the document with date '2023-01-01T00:00:00' and id 'doc_3', sorted by date and id.

Execution Table

Step	Action	Sort Values Used	Result	Next Step
1	Initial search without search_after	None	Returns first 3 docs sorted by date asc, id asc	Extract last doc's sort values
2	Extract last doc's sort values	Last doc sort: ["2023-01-01T00:00:00", "doc_3"]	Ready for next page query	Use search_after with these values
3	Search with search_after	["2023-01-01T00:00:00", "doc_3"]	Returns next 3 docs after last doc	Repeat or stop
4	Repeat steps 2-3	Updated last doc's sort values	Fetch next pages efficiently	Stop when no more docs

💡 No more documents after last search_after values, pagination ends

Variable Tracker

Variable	Start	After Step 1	After Step 2	After Step 3	Final
search_after	None	None	["2023-01-01T00:00:00", "doc_3"]	[Updated last doc's sort values]	None (end)
results	Empty	[doc_1, doc_2, doc_3]	[doc_4, doc_5, doc_6]	[Next page docs]	Empty (end)

Key Moments - 3 Insights

Why do we need to use the last document's sort values for search_after?

Because search_after requires the exact sort values of the last document from the previous page to fetch the next page correctly, as shown in step 2 of the execution_table.

Can we use search_after without sorting?

No, search_after only works with sorted results because it uses the sort values to know where to continue, as seen in the code sample and step 3.

What happens if we use search_after with wrong or missing sort values?

The query will not return the correct next page and may skip or repeat documents, breaking pagination, as implied by the importance of step 2.

Visual Quiz - 3 Questions

Look at the execution_table, what are the sort values used in step 3?

A ["2023-01-02T00:00:00", "doc_6"]

B None

C ["2023-01-01T00:00:00", "doc_3"]

D ["doc_3", "2023-01-01T00:00:00"]

💡 Hint

Check the 'Sort Values Used' column in step 3 of the execution_table.

At which step does the system extract the last document's sort values?

A Step 2

B Step 1

C Step 3

D Step 4

💡 Hint

Look for the step where 'Extract last doc's sort values' is described in the execution_table.

If the initial search size changes from 3 to 5, how does the variable 'results' change after step 1?

A It will contain 3 documents

B It will contain 5 documents

C It will be empty

D It will contain 1 document

💡 Hint

Refer to variable_tracker 'results' after step 1 and consider the 'size' parameter in the code sample.

Concept Snapshot

Search after pagination uses the last document's sort values to fetch the next page.
Always sort your results to use search_after.
Extract last doc's sort values from previous page.
Use search_after with those values in next query.
Efficient for deep pagination without performance loss.

Full Transcript

This visual execution shows how Elasticsearch's search_after works for efficient pagination. First, an initial search fetches a page of sorted results. Then, the last document's sort values are extracted. These values are used in the next search_after query to get the next page. This process repeats until no more documents remain. The key is that search_after requires sorted results and the exact last sort values to continue pagination correctly. The execution table traces each step, showing actions, sort values, and results. Variable tracking shows how search_after and results change over steps. Common confusions include why sort values are needed and that search_after cannot work without sorting. The quiz tests understanding of these steps and values. This method avoids the performance issues of deep pagination with from/size.