Bulk API for batch operations in Elasticsearch - Step-by-Step Execution

Concept Flow - Bulk API for batch operations

Prepare batch of operations

↓

Send batch request to Bulk API

↓

Bulk API processes each operation

↓

Receive response with success/failure per operation

↓

Handle errors or confirm success

↓

End

The Bulk API takes many operations in one request, processes them all, then returns individual results for each.

Execution Sample

Elasticsearch

{ "index": { "_index": "test", "_id": "1" } }
{ "field1": "value1" }
{ "delete": { "_index": "test", "_id": "2" } }
{ "update": { "_index": "test", "_id": "3" } }
{ "doc": { "field2": "value2" } }

This batch request indexes a document, deletes another, and updates a third in one call.

Execution Table

Step	Operation	Action	Target Document	Result
1	index	Add document	id=1 in index 'test'	Success
2	delete	Remove document	id=2 in index 'test'	Success
3	update	Modify document	id=3 in index 'test'	Success
4	response	Return results	All operations	All succeeded

💡 All batch operations processed; Bulk API returns success for each.

Variable Tracker

Variable	Start	After 1	After 2	After 3	Final
batch_operations	empty	index op added	delete op added	update op added	full batch ready
response	none	none	none	none	results for all ops

Key Moments - 3 Insights

Why do we send multiple operations in one Bulk API request?

Sending many operations together reduces network overhead and speeds up processing, as shown in execution_table rows 1-3.

How do we know if each operation succeeded?

The Bulk API response includes individual results for each operation, as seen in execution_table row 4.

What happens if one operation in the batch fails?

The Bulk API still processes all operations and reports success or failure per operation, so you can handle errors individually.

Visual Quiz - 3 Questions

Look at the execution_table, what operation is performed at step 2?

A Index a document

B Update a document

C Delete a document

D Return response

💡 Hint

Check the 'Operation' column at step 2 in the execution_table.

At which step does the Bulk API return the results for all operations?

A Step 2

B Step 4

C Step 3

D Step 1

💡 Hint

Look for the 'response' action in the execution_table.

If the delete operation failed, how would the execution_table change?

A Step 2 result would show failure

B Step 1 would show failure

C Step 4 would be missing

D All steps would show failure

💡 Hint

Errors are reported per operation as in the 'Result' column of execution_table.

Concept Snapshot

Bulk API lets you send many index, update, or delete operations in one request.
Each operation is listed as a JSON action line followed by optional data.
The API processes all operations and returns individual success or failure.
This reduces network calls and improves speed for batch changes.
Always check the response to handle errors per operation.

Full Transcript

The Bulk API in Elasticsearch allows sending multiple operations like index, update, and delete in a single request. This batch approach reduces network overhead and speeds up processing. Each operation is specified as a JSON action line, optionally followed by data. The Bulk API processes all operations and returns a response with individual results for each. This way, you can see which operations succeeded or failed and handle errors accordingly. The execution table shows indexing a document, deleting another, updating a third, and then receiving the combined response. This method is efficient for batch operations.

Practice

1. What is the main purpose of the Elasticsearch Bulk API?

easy

A. To perform multiple document operations in a single request

B. To create a new Elasticsearch index

C. To search documents with complex queries

D. To delete an entire Elasticsearch cluster

Solution

1. Step 1: Understand Bulk API functionality

   The Bulk API is designed to handle many document operations like add, update, or delete in one request to improve efficiency.

2. Step 2: Compare options with Bulk API purpose

   Creating indexes, searching, or deleting clusters are not the Bulk API's main tasks.

3. Final Answer:

   To perform multiple document operations in a single request -> Option A

4. Quick Check:

   Bulk API = batch operations [OK]

Hint: Bulk API batches many actions in one request [OK]

Common Mistakes:

• Confusing Bulk API with search API
• Thinking Bulk API creates indexes
• Assuming Bulk API deletes clusters

2. Which of the following is the correct format for a single action in the Elasticsearch Bulk API?

easy

A. {"index": {"_index": "test", "_id": "1", "field": "value"}}

B. {"index": {"_index": "test", "_id": "1"}}
{"field": "value"}

C. [{"index": {"_index": "test", "_id": "1"}}, {"field": "value"}]

D. {"field": "value"}
{"index": {"_index": "test", "_id": "1"}}

Solution

1. Step 1: Recall Bulk API action format

   Each action in Bulk API consists of two JSON lines: first metadata (like index and id), then the document data.

2. Step 2: Check options for correct order and format

   Combining metadata and data in one JSON object is invalid. The reversed order with data line before metadata line is wrong. Using a JSON array is invalid. {"index": {"_index": "test", "_id": "1"}} {"field": "value"} correctly shows metadata line followed by data line.

3. Final Answer:

   {"index": {"_index": "test", "_id": "1"}} {"field": "value"} -> Option B

4. Quick Check:

   Metadata line then data line = correct Bulk format [OK]

Hint: Metadata line first, then data line in Bulk API [OK]

Common Mistakes:

• Swapping metadata and data lines
• Using JSON arrays instead of lines
• Combining metadata and data in one JSON object

3. Given the following Bulk API request body, what will happen?

{"index": {"_index": "products", "_id": "10"}}
{"name": "Laptop", "price": 1200}
{"delete": {"_index": "products", "_id": "5"}}
{"index": {"_index": "products", "_id": "11"}}
{"name": "Mouse", "price": 25}

medium

A. Adds two documents and deletes one document

B. Adds three documents

C. Deletes two documents

D. Syntax error due to missing data line for delete

Solution

1. Step 1: Analyze each Bulk API action

   First action: index document with id 10 (add Laptop). Second action: delete document with id 5 (no data line needed). Third action: index document with id 11 (add Mouse).

2. Step 2: Confirm correct syntax and operations

   Delete action does not require a data line, so no syntax error. Two documents are added, one deleted.

3. Final Answer:

   Adds two documents and deletes one document -> Option A

4. Quick Check:

   Index + delete + index = add 2, delete 1 [OK]

Hint: Delete action needs no data line in Bulk API [OK]

Common Mistakes:

• Expecting data line after delete action
• Counting delete as add
• Confusing action order

4. Identify the error in this Bulk API request snippet:

{"update": {"_index": "users", "_id": "3"}}
{"name": "Alice"}

medium

A. Update action cannot be used with _id

B. Incorrect index name format

C. Missing "doc" field inside the data line for update

D. Data line should be empty for update

Solution

1. Step 1: Understand update action format

   Update action requires the data line to contain a "doc" field wrapping the fields to update.

2. Step 2: Check the given data line

   The data line only has {"name": "Alice"} without "doc", so it is invalid syntax for update.

3. Final Answer:

   Missing "doc" field inside the data line for update -> Option C

4. Quick Check:

   Update data needs "doc" wrapper [OK]

Hint: Update data must be inside "doc" field [OK]

Common Mistakes:

• Omitting "doc" in update data
• Thinking update needs empty data line
• Confusing update with index action

5. You want to update multiple documents but only if their "status" field is "pending". How can you use the Bulk API to do this efficiently?

hard

A. Use Bulk API with delete actions for all documents and re-index updated ones

B. Use Bulk API with index actions to overwrite all documents regardless of status

C. Use Bulk API with search queries to filter documents before updating

D. Use Bulk API with update actions including a script that checks "status" before updating

Solution

1. Step 1: Understand conditional updates in Bulk API

   Bulk API update actions can include scripts that check document fields before applying changes.

2. Step 2: Evaluate options for conditional update

   Use Bulk API with update actions including a script that checks "status" before updating uses update with scripts to check "status" = "pending" before updating, which is efficient. Use Bulk API with index actions to overwrite all documents regardless of status overwrites all documents ignoring condition. Use Bulk API with delete actions for all documents and re-index updated ones deletes unnecessarily. Use Bulk API with search queries to filter documents before updating is invalid because Bulk API does not support search queries.

3. Final Answer:

   Use Bulk API with update actions including a script that checks "status" before updating -> Option D

4. Quick Check:

   Conditional update = update + script in Bulk API [OK]

Hint: Use update with scripts for conditional Bulk updates [OK]

Common Mistakes:

• Using index to overwrite without condition
• Deleting documents unnecessarily
• Trying to filter with search inside Bulk API