Bulk indexing optimization in Elasticsearch - Cheat Sheet & Quick Revision

Recall & Review

beginner

What is bulk indexing in Elasticsearch?

Bulk indexing is a method to send multiple indexing or update requests in a single API call to Elasticsearch, improving speed and reducing overhead.

beginner

Why is it important to optimize bulk indexing?

Optimizing bulk indexing reduces network overhead, improves throughput, and prevents cluster overload, leading to faster and more reliable data ingestion.

intermediate

What is a good practice for choosing the bulk request size?

Choose a bulk size that balances memory use and speed, typically between 5MB to 15MB or 1000 to 5000 documents per bulk request, depending on your cluster capacity.

intermediate

How can you handle failures during bulk indexing?

Check the bulk API response for errors, retry failed items selectively, and implement exponential backoff to avoid overwhelming the cluster.

intermediate

What role does refresh interval play in bulk indexing optimization?

Temporarily increasing the refresh interval or disabling automatic refresh during bulk indexing reduces overhead and improves indexing speed.

What is the main benefit of using bulk indexing in Elasticsearch?

A Increasing the number of shards

B Reducing the number of network calls

C Improving query speed

D Decreasing disk space usage

Bulk indexing reduces network overhead by sending many documents in one request.

Which bulk request size is generally recommended for optimal performance?

A 1 document per request

B Less than 100 bytes

C More than 100MB

D 5MB to 15MB or 1000 to 5000 documents

A bulk size of 5MB to 15MB or 1000 to 5000 documents balances speed and memory use.

How should you handle errors returned by the bulk API?

A Retry only failed documents with backoff

B Retry all documents regardless

C Ignore them and continue

D Stop indexing immediately

Retrying only failed documents with exponential backoff prevents overload and ensures data consistency.

What happens if you disable automatic refresh during bulk indexing?

A Indexing speed improves

B Search results update immediately

C Cluster memory usage decreases

D Documents are lost

Disabling refresh reduces overhead, allowing faster indexing.

Which of the following is NOT a bulk indexing optimization technique?

A Using bulk API instead of single requests

B Choosing an appropriate bulk size

C Indexing documents one by one

D Increasing refresh interval during indexing

Indexing documents one by one is inefficient and not an optimization.

Explain how to optimize bulk indexing in Elasticsearch for better performance.

Think about request size, error handling, and refresh settings.

Describe the steps to handle failures during bulk indexing in Elasticsearch.

Focus on error detection and retry strategy.

Practice

1. What is the main benefit of using the _bulk API in Elasticsearch for indexing documents?

easy

A. It reduces the number of network requests by sending many documents at once.

B. It automatically fixes errors in documents before indexing.

C. It compresses documents to save disk space.

D. It indexes documents one by one to ensure accuracy.

Solution

1. Step 1: Understand the purpose of bulk API

   The bulk API is designed to send multiple documents in a single request to Elasticsearch.

2. Step 2: Identify the main advantage

   Sending many documents at once reduces network overhead and speeds up indexing.

3. Final Answer:

   It reduces the number of network requests by sending many documents at once. -> Option A

4. Quick Check:

   Bulk API = fewer requests = faster indexing [OK]

Hint: Bulk API batches documents to reduce network calls [OK]

Common Mistakes:

• Thinking bulk API fixes document errors automatically
• Believing bulk API compresses data for storage
• Assuming bulk API indexes documents one by one

2. Which of the following is the correct JSON structure for a single bulk action in Elasticsearch?

easy

A. { "index": { "_index": "myindex", "_id": "1" } }\n{ "field": "value" }

B. A, C, and D are all valid bulk actions

C. { "update": { "_index": "myindex", "_id": "1" } }\n{ "doc": { "field": "value" } }

D. { "create": { "_index": "myindex" } }\n{ "field": "value" }

Solution

1. Step 1: Review bulk action types

   Elasticsearch bulk API supports multiple actions: index, create, update.

2. Step 2: Check each option

   A shows an index action, C an update action, D a create action. All are valid formats.

3. Final Answer:

   A, C, and D are all valid bulk actions -> Option B

4. Quick Check:

   Bulk supports index, create, update actions [OK]

Hint: Bulk API supports index, create, update actions [OK]

Common Mistakes:

• Thinking only index action is allowed
• Confusing create and update JSON formats
• Missing newline between action and data lines

3. Given this Python snippet using Elasticsearch bulk API, what will be the output if one document has a mapping error?

from elasticsearch import Elasticsearch, helpers
es = Elasticsearch()
docs = [
  {"_index": "test", "_id": "1", "field": "value1"},
  {"_index": "test", "_id": "2", "field": 123}  # mapping error if field expects string
]
response = helpers.bulk(es, docs)
print(response)

medium

A. (2, []) # all documents indexed successfully

B. (0, [{"index": {"_id": "1", "error": "mapper_parsing_exception"}}, {"index": {"_id": "2", "error": "mapper_parsing_exception"}}])

C. Raises a Python exception and stops

D. (1, [{"index": {"_id": "2", "error": "mapper_parsing_exception"}}])

Solution

1. Step 1: Understand helpers.bulk behavior

   helpers.bulk returns a tuple: (success_count, errors_list). It continues indexing even if some docs fail.

2. Step 2: Analyze the documents

   First doc is valid, second has a mapping error (wrong type). So one success, one error.

3. Final Answer:

   (1, [{"index": {"_id": "2", "error": "mapper_parsing_exception"}}]) -> Option D

4. Quick Check:

   One success, one mapping error = (1, [{"index": {"_id": "2", "error": "mapper_parsing_exception"}}]) [OK]

Hint: helpers.bulk returns (success_count, errors) tuple [OK]

Common Mistakes:

• Assuming bulk stops on first error
• Expecting a Python exception instead of error info
• Misreading success count as total docs

4. You wrote this bulk request but it fails with a parsing error. What is the mistake?

{ "index": { "_index": "myindex", "_id": "1" }
{ "field": "value" }

medium

A. Incorrect _id field type

B. Missing comma between JSON objects

C. Missing newline between action and data lines

D. Using index instead of create action

Solution

1. Step 1: Check bulk request format

   Bulk API requires each action line and data line to be separated by a newline character.

2. Step 2: Identify the error

   The given request misses a newline between the two JSON objects, causing parsing failure.

3. Final Answer:

   Missing newline between action and data lines -> Option C

4. Quick Check:

   Bulk lines must be separated by newlines [OK]

Hint: Each bulk action and data must be on separate lines [OK]

Common Mistakes:

• Forgetting newline between JSON objects
• Adding commas between bulk lines
• Confusing index and create actions

5. You want to optimize bulk indexing for 10,000 documents. Which approach best balances speed and reliability?

hard

A. Split documents into batches of 500, send each batch, and check for errors after each batch.

B. Send all 10,000 documents in a single bulk request without checking errors.

C. Index documents one by one to catch errors immediately.

D. Send batches of 10 documents to avoid any errors.

Solution

1. Step 1: Consider bulk request size

   Very large bulk requests (like 10,000 docs) can cause memory or timeout issues.

2. Step 2: Choose batch size and error handling

   Splitting into moderate batches (e.g., 500) balances speed and resource use. Checking errors after each batch ensures reliability.

3. Final Answer:

   Split documents into batches of 500, send each batch, and check for errors after each batch. -> Option A

4. Quick Check:

   Batching + error check = optimal bulk indexing [OK]

Hint: Use moderate batch sizes and check errors after each [OK]

Common Mistakes:

• Sending too large batches causing failures
• Ignoring errors during bulk indexing
• Sending very small batches losing speed benefits