Document ID strategies (auto vs manual) in Elasticsearch

Introduction

Document IDs help find and update data quickly in Elasticsearch. You can let Elasticsearch create them automatically or set your own IDs.

When you want Elasticsearch to handle unique IDs for you automatically.

When you have a unique key from your data and want to use it as the document ID.

When you need to update or delete documents by a known ID.

When you want to avoid duplicate documents by controlling the ID.

When you want to link documents with meaningful IDs for easier searching.

Syntax

Elasticsearch

POST /index_name/_doc
{
  "field": "value"
}

POST /index_name/_doc/document_id
{
  "field": "value"
}

Without specifying an ID, Elasticsearch creates one automatically.

To set your own ID, add it after the index and _doc in the URL.

Examples

Elasticsearch creates a unique ID automatically for this new product.

Elasticsearch

POST /products/_doc
{
  "name": "Coffee Mug",
  "price": 12.99
}

We set the document ID to '12345' manually for easy reference later.

Elasticsearch

POST /products/_doc/12345
{
  "name": "Tea Cup",
  "price": 9.99
}

Retrieve the document with the manual ID '12345'.

Elasticsearch

GET /products/_doc/12345

Sample Program

This example adds two books to the 'library' index. The first book gets an automatic ID. The second book uses a manual ID 'book-001'. Then it retrieves the second book by its manual ID.

Elasticsearch

POST /library/_doc
{
  "title": "Elasticsearch Basics",
  "author": "Jane Doe"
}

POST /library/_doc/book-001
{
  "title": "Advanced Elasticsearch",
  "author": "John Smith"
}

GET /library/_doc/book-001

Important Notes

Manual IDs let you update or delete documents easily by ID.

Automatic IDs are good when you don't have a unique key and want Elasticsearch to handle uniqueness.

Using manual IDs incorrectly can cause overwriting documents if IDs repeat.

Summary

Document IDs identify each document uniquely in Elasticsearch.

You can let Elasticsearch create IDs automatically or set your own manual IDs.

Manual IDs help with updates and avoiding duplicates but require careful management.

Practice

1. What is the main difference between automatic and manual document IDs in Elasticsearch?

easy

A. Automatic IDs are generated by Elasticsearch; manual IDs are set by the user.

B. Automatic IDs require user input; manual IDs are generated by Elasticsearch.

C. Automatic IDs are always numeric; manual IDs are always strings.

D. Automatic IDs cannot be used for updates; manual IDs cannot.

Solution

1. Step 1: Understand automatic ID generation

   Elasticsearch creates unique IDs automatically when you don't specify one.

2. Step 2: Understand manual ID assignment

   Users can assign their own IDs to documents for control and easier updates.

3. Final Answer:

   Automatic IDs are generated by Elasticsearch; manual IDs are set by the user. -> Option A

4. Quick Check:

   Auto = Elasticsearch generated, Manual = user set [OK]

Hint: Auto = system creates ID, manual = you create ID [OK]

Common Mistakes:

• Confusing which side creates the ID
• Thinking automatic IDs are numeric only
• Believing manual IDs cannot be updated

2. Which of the following is the correct syntax to index a document with a manual ID in Elasticsearch using JSON?

easy

A. { "index": "my_index", "_id": "123", "body": { "name": "Alice" } }

B. { "index": "my_index", "id": "123", "body": { "name": "Alice" } }

C. { "index": "my_index", "id": 123, "doc": { "name": "Alice" } }

D. { "index": "my_index", "_id": 123, "doc": { "name": "Alice" } }

Solution

1. Step 1: Identify the correct field for manual ID

   Elasticsearch uses "_id" (with underscore) to specify manual document IDs.

2. Step 2: Check the correct structure for indexing

   The "body" key holds the document content; "doc" is used for updates, not indexing.

3. Final Answer:

   { "index": "my_index", "_id": "123", "body": { "name": "Alice" } } -> Option A

4. Quick Check:

   Manual ID uses "_id" and "body" for document [OK]

Hint: Use "_id" for manual IDs and "body" for document content [OK]

Common Mistakes:

• Using "id" instead of "_id"
• Confusing "doc" with "body" for indexing
• Passing numeric ID without quotes

3. Given this indexing request with manual ID:

{ "index": "products", "_id": "p100", "body": { "name": "Pen", "price": 1.5 } }

What happens if you send the same request again with updated price 1.8?

medium

A. A new document is created with a different ID.

B. The existing document with ID "p100" is updated with the new price.

C. An error occurs because duplicate IDs are not allowed.

D. The document is ignored and no changes happen.

Solution

1. Step 1: Understand manual ID behavior on indexing

   Using the same manual ID overwrites the existing document with that ID.

2. Step 2: Check the effect of sending updated data

   The document with ID "p100" will be updated to have price 1.8.

3. Final Answer:

   The existing document with ID "p100" is updated with the new price. -> Option B

4. Quick Check:

   Same manual ID means update existing document [OK]

Hint: Same manual ID updates document, no new doc created [OK]

Common Mistakes:

• Thinking a new document is created with same ID
• Expecting an error on duplicate ID
• Assuming no change happens on reindex

4. You tried to index a document with this JSON:

{ "index": "users", "id": "u123", "body": { "name": "Bob" } }

But the document was not saved with ID "u123". What is the likely problem?

medium

A. The "body" field is missing required fields.

B. The index name "users" is invalid.

C. The field "id" should be "_id" to set manual ID.

D. Manual IDs are not allowed in Elasticsearch.

Solution

1. Step 1: Check the ID field name

   Elasticsearch requires "_id" (with underscore) to specify manual IDs, not "id".

2. Step 2: Confirm impact of wrong field

   Using "id" means Elasticsearch ignores it and generates an automatic ID instead.

3. Final Answer:

   The field "id" should be "_id" to set manual ID. -> Option C

4. Quick Check:

   Manual ID needs "_id", not "id" [OK]

Hint: Use "_id" to set manual ID, not "id" [OK]

Common Mistakes:

• Using "id" instead of "_id"
• Assuming missing fields cause ID issues
• Thinking manual IDs are disallowed

5. You want to avoid duplicate documents in Elasticsearch. Which strategy can help you achieve this?

hard

A. Use a random string as manual ID without relation to document content.

B. Always use automatic IDs and ignore duplicates.

C. Index documents without IDs and rely on timestamps to find duplicates.

D. Use a manual ID based on a unique field like email to prevent duplicates.

Solution

1. Step 1: Understand duplicate prevention

   Duplicates can be avoided by assigning manual IDs based on unique fields like email.

2. Step 2: Compare with automatic ID approach

   Automatic IDs are random and do not prevent duplicates because each document gets a new ID.

3. Final Answer:

   Use a manual ID based on a unique field like email to prevent duplicates. -> Option D

4. Quick Check:

   Manual unique IDs prevent duplicates, auto IDs do not [OK]

Hint: Manual unique IDs prevent duplicates better than auto IDs [OK]

Common Mistakes:

• Thinking auto IDs prevent duplicates
• Using random manual IDs unrelated to content
• Relying on timestamps instead of IDs