Bulk indexing optimization in Elasticsearch - Mini Project: Build & Apply

Bulk indexing optimization

📖 Scenario: You work with Elasticsearch to store many documents quickly. Instead of adding one document at a time, you want to use bulk indexing to save time and resources.

🎯 Goal: Build a simple bulk indexing process using Elasticsearch's bulk API to add multiple documents efficiently.

📋 What You'll Learn

Create a list of documents with exact fields and values

Set a bulk size limit variable

Write a loop to prepare bulk request body with action and data lines

Print the final bulk request body as a string

💡 Why This Matters

🌍 Real World

Bulk indexing is used in real projects to add many records to Elasticsearch quickly, saving time and reducing server load.

💼 Career

Knowing how to optimize bulk indexing is important for roles like backend developer, data engineer, and search engineer working with Elasticsearch.

1

Create the documents list

Create a list called documents with these exact dictionaries: {"id": 1, "name": "Alice", "age": 30}, {"id": 2, "name": "Bob", "age": 25}, and {"id": 3, "name": "Charlie", "age": 35}.

Elasticsearch

# Create the list of documents
# Your code here

Need a hint?

Use a Python list with dictionaries inside. Each dictionary has keys id, name, and age.

2

Set the bulk size limit

Create a variable called bulk_size and set it to 2 to limit how many documents to send in one bulk request.

Elasticsearch

documents = [
    {"id": 1, "name": "Alice", "age": 30},
    {"id": 2, "name": "Bob", "age": 25},
    {"id": 3, "name": "Charlie", "age": 35}
]
# Set the bulk size limit
# Your code here

Need a hint?

Just create a variable bulk_size and assign the number 2.

3

Prepare the bulk request body

Create a list called bulk_body. Use a for loop with variable doc to go through documents. For each doc, add two dictionaries to bulk_body: one with {"index": {"_id": doc["id"]}} and one with the doc itself.

Elasticsearch

documents = [
    {"id": 1, "name": "Alice", "age": 30},
    {"id": 2, "name": "Bob", "age": 25},
    {"id": 3, "name": "Charlie", "age": 35}
]
bulk_size = 2
# Prepare the bulk request body
# Your code here

Need a hint?

Remember, the bulk API needs an action line and a data line for each document.

4

Print the bulk request body

Print the string version of bulk_body using print(str(bulk_body)) to see the final bulk request content.

Elasticsearch

documents = [
    {"id": 1, "name": "Alice", "age": 30},
    {"id": 2, "name": "Bob", "age": 25},
    {"id": 3, "name": "Charlie", "age": 35}
]
bulk_size = 2
bulk_body = []
for doc in documents:
    bulk_body.append({"index": {"_id": doc["id"]}})
    bulk_body.append(doc)
# Print the bulk request body
# Your code here

Need a hint?

Use print(str(bulk_body)) to show the list as a string.