Scroll API for deep pagination in Elasticsearch - Mini Project: Build & Apply

Scroll API for deep pagination

📖 Scenario: You work with a large collection of documents in Elasticsearch. You want to retrieve all documents matching a query, but the number of results is too big to get in one request. Elasticsearch's Scroll API helps you fetch results in batches, like flipping pages in a book, so you can see all data without missing any.

🎯 Goal: Build a program that uses Elasticsearch's Scroll API to fetch all documents matching a query in batches, handling deep pagination efficiently.

📋 What You'll Learn

Create an initial search request with a scroll parameter

Store the scroll ID returned by Elasticsearch

Use the scroll ID to fetch the next batch of results

Repeat fetching until no more results remain

Print the total number of documents retrieved

💡 Why This Matters

🌍 Real World

When working with very large datasets in Elasticsearch, normal pagination can be inefficient or limited. The Scroll API lets you retrieve all matching documents in manageable batches, like reading pages of a book, without missing any data.

💼 Career

Many data engineer and backend developer roles require handling large search results efficiently. Knowing how to use Elasticsearch's Scroll API is important for building scalable search and analytics applications.

1

Create initial search request with scroll

Write code to send an initial search request to Elasticsearch index products with a match_all query and a scroll time of 1m. Store the response in a variable called response.

Elasticsearch

# Your code here

Need a hint?

Use client.search with scroll='1m' and a match_all query inside body.

2

Store scroll ID and prepare to fetch batches

Extract the _scroll_id from response and store it in a variable called scroll_id. Also, create a list called all_hits and add the hits from response to it.

Elasticsearch

response = client.search(
    index='products',
    scroll='1m',
    body={
        'query': {
            'match_all': {}
        }
    }
)
# Your code here

Need a hint?

Access _scroll_id from response and assign it to scroll_id. Then get hits from response['hits']['hits'] and assign to all_hits.

3

Fetch all batches using scroll ID

Use a while loop to keep fetching batches using client.scroll with scroll_id and scroll='1m'. Update scroll_id with the new scroll ID from each response. Add the hits from each batch to all_hits. Stop when the batch has no hits.

Elasticsearch

response = client.search(
    index='products',
    scroll='1m',
    body={
        'query': {
            'match_all': {}
        }
    }
)
scroll_id = response['_scroll_id']
all_hits = response['hits']['hits']

# Your code here

Need a hint?

Use a while True loop. Inside, call client.scroll with current scroll_id and scroll='1m'. Update scroll_id. Get hits and break if empty. Otherwise, add hits to all_hits.

4

Print total number of documents retrieved

Write a print statement to display the total number of documents retrieved by printing the length of all_hits.

Elasticsearch

response = client.search(
    index='products',
    scroll='1m',
    body={
        'query': {
            'match_all': {}
        }
    }
)
scroll_id = response['_scroll_id']
all_hits = response['hits']['hits']

while True:
    response = client.scroll(scroll_id=scroll_id, scroll='1m')
    scroll_id = response['_scroll_id']
    hits = response['hits']['hits']
    if not hits:
        break
    all_hits.extend(hits)

# Your code here

Need a hint?

Use print(len(all_hits)) to show how many documents were fetched in total.