Search after for efficient pagination in Elasticsearch - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

Output of search_after with sort keys

What will be the output of the following Elasticsearch query snippet when using search_after for pagination?

Assume the index contains documents sorted by timestamp ascending and id ascending.

{
  "size": 2,
  "query": { "match_all": {} },
  "sort": [
    { "timestamp": "asc" },
    { "id": "asc" }
  ],
  "search_after": ["2023-01-01T00:00:00", "100"]
}

Given the documents:

• {"timestamp": "2023-01-01T00:00:00", "id": "100"}
• {"timestamp": "2023-01-01T00:00:01", "id": "101"}
• {"timestamp": "2023-01-01T00:00:02", "id": "102"}

Which documents will be returned?

Elasticsearch

{
  "size": 2,
  "query": { "match_all": {} },
  "sort": [
    { "timestamp": "asc" },
    { "id": "asc" }
  ],
  "search_after": ["2023-01-01T00:00:00", "100"]
}

A Documents with timestamp "2023-01-01T00:00:01" and id "101", and timestamp "2023-01-01T00:00:02" and id "102"

B Documents with timestamp "2023-01-01T00:00:00" and id "100" only

C Documents with timestamp "2023-01-01T00:00:00" and id "100", and timestamp "2023-01-01T00:00:01" and id "101"

D No documents returned

💡 Hint

Remember that search_after returns documents after the given sort values, excluding the document with those exact values.

The search_after parameter returns documents strictly after the given sort values. Since the sort is by timestamp ascending and then id ascending, documents with timestamp "2023-01-01T00:00:00" and id "100" are excluded. The next documents are those with timestamp "2023-01-01T00:00:01" and id "101", and then "2023-01-01T00:00:02" and id "102".

🧠 Conceptual

intermediate

Why use search_after instead of from/size for deep pagination?

Why is search_after recommended over from and size for deep pagination in Elasticsearch?

A Because from/size does not support sorting, while search_after does.

B Because search_after automatically caches all previous pages for faster access.

C Because search_after uses the last sort values to efficiently continue from the previous page, avoiding performance issues.

D Because search_after returns documents in random order, which is faster.

💡 Hint

Think about how Elasticsearch handles skipping documents with from/size.

Using from/size for deep pagination requires Elasticsearch to skip many documents, which is slow and resource-intensive. search_after uses the last document's sort values to continue efficiently without skipping.

🔧 Debug

advanced

Identify the error in search_after usage

What error will this Elasticsearch query produce?

{
  "size": 3,
  "query": { "match_all": {} },
  "sort": [
    { "date": "desc" }
  ],
  "search_after": ["2023-05-01"]
}

Assuming the date field is of type date and the sort expects a date and a tie-breaker field.

Elasticsearch

{
  "size": 3,
  "query": { "match_all": {} },
  "sort": [
    { "date": "desc" }
  ],
  "search_after": ["2023-05-01"]
}

A The query will run successfully and return documents after the given date.

B Elasticsearch will raise a SearchPhaseExecutionException due to missing tie-breaker sort field.

C Elasticsearch will raise a ParsingException due to invalid date format in search_after.

D The query will return documents sorted ascending instead of descending.

💡 Hint

search_after requires the sort fields and search_after values to match exactly in number and order.

When using search_after, the sort array must include a tie-breaker field (like _id) to ensure unique sorting. The search_after array must have the same number of values as the sort fields. Missing the tie-breaker causes an error.

📝 Syntax

advanced

Correct search_after syntax for multi-field sort

Which of the following is the correct search_after syntax for this sort?

"sort": [
  { "price": "asc" },
  { "rating": "desc" },
  { "_id": "asc" }
]

A [100, 4.5, "abc123"]

B [100, "abc123", 4.5]

C ["abc123", 100, 4.5]

D [4.5, 100, "abc123"]

💡 Hint

The order of values in search_after must match the order of fields in sort.

The search_after array must have values in the same order as the sort fields: price (asc), rating (desc), then _id (asc). So the values are [price_value, rating_value, _id_value].

🚀 Application

expert

Implementing efficient deep pagination with search_after

You have an Elasticsearch index with millions of documents sorted by created_at ascending and doc_id ascending.

You want to fetch page 1000 with 10 documents per page efficiently.

Which approach correctly uses search_after to get page 1000?

A Use a scroll API instead of search_after for page 1000.

B Use from: 9990 and size: 10 to jump directly to page 1000.

C Use search_after with the sort values from page 1 to jump directly to page 1000.

D Iteratively query pages 1 to 999 using search_after, storing the last sort values each time, then query page 1000 with those values.

💡 Hint

search_after requires the last document's sort values from the previous page to continue.

search_after requires the last document's sort values from the previous page to fetch the next page. To get page 1000, you must iterate through pages 1 to 999, saving the last sort values each time, then use those values to query page 1000.

Practice

1. What is the main purpose of using search_after in Elasticsearch pagination?

easy

A. To filter documents based on a query

B. To sort documents alphabetically by default

C. To efficiently paginate through large result sets without performance loss

D. To update documents in bulk

Solution

1. Step 1: Understand pagination challenges

   Deep pagination with large result sets can be slow and inefficient using traditional methods like from and size.

2. Step 2: Role of search_after

   search_after uses the last sort values from the previous page to fetch the next page efficiently, avoiding performance issues.

3. Final Answer:

   To efficiently paginate through large result sets without performance loss -> Option C

4. Quick Check:

   Purpose of search_after = Efficient pagination [OK]

Hint: Remember: search_after uses last sort values for fast paging [OK]

Common Mistakes:

• Confusing search_after with filtering
• Thinking search_after sorts results automatically
• Using search_after without sorting

2. Which of the following is the correct syntax snippet to use search_after in an Elasticsearch query?

easy

A. "search_after": ["last_sort_value"]

B. "search_after": "last_sort_value"

C. "search_after": {"value": "last_sort_value"}

D. "search_after": true

Solution

1. Step 1: Check the expected data type for search_after

   The search_after parameter expects an array of sort values, not a single string or object.

2. Step 2: Match syntax with correct format

   "search_after": ["last_sort_value"] correctly shows search_after as an array with the last sort value inside.

3. Final Answer:

   "search_after": ["last_sort_value"] -> Option A

4. Quick Check:

   search_after syntax = array of values [OK]

Hint: search_after always takes an array of sort values [OK]

Common Mistakes:

• Passing a single string instead of an array
• Using an object instead of an array
• Setting search_after to a boolean

3. Given this Elasticsearch query snippet, what will be the effect of adding "search_after": [1627891234567]?

{
  "size": 5,
  "sort": [{"timestamp": "asc"}],
  "search_after": [1627891234567]
}

medium

A. It causes a syntax error because search_after is not allowed here

B. It returns the first 5 documents sorted by timestamp ascending

C. It returns 5 documents with timestamp less than or equal to 1627891234567

D. It returns 5 documents with timestamp strictly greater than 1627891234567

Solution

1. Step 1: Understand sorting and search_after usage

   The query sorts documents by timestamp ascending and uses search_after with a timestamp value.

2. Step 2: Effect of search_after value

   search_after tells Elasticsearch to return documents after the given sort value, so only documents with timestamp greater than 1627891234567 are returned.

3. Final Answer:

   It returns 5 documents with timestamp strictly greater than 1627891234567 -> Option D

4. Quick Check:

   search_after filters results after given sort value [OK]

Hint: search_after returns results after the given sort values [OK]

Common Mistakes:

• Thinking it returns documents before the value
• Assuming it returns the first page always
• Confusing search_after with from/size pagination

4. You wrote this Elasticsearch query to paginate results:

{
  "size": 10,
  "sort": [{"date": "desc"}],
  "search_after": "2023-01-01T00:00:00"
}

But it returns an error. What is the likely cause?

medium

A. size cannot be 10 with search_after

B. search_after value must be an array, not a string

C. sort order must be ascending for search_after

D. date field cannot be used in sort

Solution

1. Step 1: Check the type of search_after value

   The search_after parameter requires an array of values, but here it is a string.

2. Step 2: Identify the error cause

   Passing a string instead of an array causes a syntax error in the query.

3. Final Answer:

   search_after value must be an array, not a string -> Option B

4. Quick Check:

   search_after requires array input [OK]

Hint: Always wrap search_after values in an array [OK]

Common Mistakes:

• Passing single value without array brackets
• Using unsupported sort order
• Misunderstanding size limits with search_after

5. You want to paginate through a large dataset sorted by user_id (ascending) and then timestamp (descending). Which search_after value correctly fetches the next page after user_id=42 and timestamp=1680000000?

hard

A. [42, 1680000000]

B. [42, -1680000000]

C. [1680000000, 42]

D. ["42", "1680000000"]

Solution

1. Step 1: Understand sort order and search_after values

   The sort is by user_id ascending, then timestamp descending. The search_after array must match this order.

2. Step 2: Match values to sort order

   The correct search_after is an array with user_id first, then timestamp. Since timestamp is descending, the value is used as is (no negation).

3. Final Answer:

   [42, 1680000000] -> Option A

4. Quick Check:

   search_after array matches sort fields order [OK]

Hint: search_after array order matches sort fields order exactly [OK]

Common Mistakes:

• Reversing order of values in search_after
• Negating timestamp for descending sort
• Using strings instead of numbers without need