How to Paginate Aggregation Results in Elasticsearch

To paginate aggregation results in Elasticsearch, use the composite aggregation which supports the after parameter to fetch the next page of buckets. This method allows you to scroll through large aggregation results efficiently without missing or duplicating data.

Syntax

The composite aggregation syntax includes a sources array defining the fields to aggregate on, and an optional after key to paginate results.

Key parts:

• sources: List of fields to group by.
• size: Number of buckets to return per page.
• after: The last bucket key from the previous page to continue pagination.

json

{
  "aggs": {
    "my_buckets": {
      "composite": {
        "size": 10,
        "sources": [
          { "field_name": { "terms": { "field": "field_name.keyword" } } }
        ],
        "after": { "field_name": "last_value_from_previous_page" }
      }
    }
  }
}

Example

This example shows how to paginate aggregation results on a field called category.keyword. The first query fetches the first 3 buckets. The second query uses the after key from the first response to get the next 3 buckets.

json

{
  "size": 0,
  "aggs": {
    "categories": {
      "composite": {
        "size": 3,
        "sources": [
          { "category": { "terms": { "field": "category.keyword" } } }
        ]
      }
    }
  }
}

--- Response snippet ---
"aggregations": {
  "categories": {
    "buckets": [
      { "key": { "category": "Books" }, "doc_count": 15 },
      { "key": { "category": "Clothing" }, "doc_count": 10 },
      { "key": { "category": "Electronics" }, "doc_count": 8 }
    ],
    "after_key": { "category": "Electronics" }
  }
}

--- Next query to get next page ---
{
  "size": 0,
  "aggs": {
    "categories": {
      "composite": {
        "size": 3,
        "sources": [
          { "category": { "terms": { "field": "category.keyword" } } }
        ],
        "after": { "category": "Electronics" }
      }
    }
  }
}

Output

"buckets": [
  { "key": { "category": "Furniture" }, "doc_count": 5 },
  { "key": { "category": "Toys" }, "doc_count": 3 }
]

Common Pitfalls

1. Using terms aggregation for pagination: The terms aggregation does not support pagination with an after key, so it cannot reliably paginate large result sets.

2. Forgetting to use after_key from the response: Always use the after_key from the previous response to fetch the next page; otherwise, you will get duplicate or missing buckets.

3. Changing the sources between requests: The sources array must remain the same for all paginated requests to maintain consistency.

json

{
  "aggs": {
    "wrong_pagination": {
      "terms": {
        "field": "category.keyword",
        "size": 10
      }
    }
  }
}

-- This will NOT paginate properly --

{
  "aggs": {
    "correct_pagination": {
      "composite": {
        "size": 10,
        "sources": [
          { "category": { "terms": { "field": "category.keyword" } } }
        ],
        "after": { "category": "last_value" }
      }
    }
  }
}

Quick Reference

• Use composite aggregation for paginating buckets.
• Set size to control buckets per page.
• Use after key from previous response to get next page.
• Keep sources consistent across requests.

Key Takeaways

Use composite aggregation with the after parameter to paginate aggregation buckets.

Always use the after_key from the previous response to fetch the next page.

Do not use terms aggregation for pagination as it does not support reliable scrolling.

Keep the sources array unchanged between paginated requests for consistent results.