Document ID strategies (auto vs manual) in Elasticsearch - When to Use Which

The Big Idea

What if your data could organize itself without you worrying about unique IDs?

The Scenario

Imagine you have a huge stack of papers and you want to organize them by writing a unique number on each one by hand.

Every time you add a new paper, you must check all previous numbers to avoid duplicates.

The Problem

Writing IDs manually is slow and tiring.

You might accidentally reuse a number, causing confusion or lost papers.

It's hard to keep track when the stack grows big.

The Solution

Using automatic ID generation, the system assigns a unique number for each paper instantly.

You don't worry about duplicates or tracking numbers.

This makes adding new papers fast and error-free.

Before vs After

✗ Before

POST /index/_doc/123
{
  "name": "Alice"
}

✓ After

POST /index/_doc
{
  "name": "Alice"
}

What It Enables

Automatic IDs let you focus on your data, not on managing unique identifiers.

Real Life Example

When you upload photos to a cloud album, each photo gets a unique ID automatically, so you never have to name or number them yourself.

Key Takeaways

Manual IDs require careful tracking and can cause errors.

Automatic IDs save time and prevent duplicates.

Choosing the right strategy depends on your control needs and data uniqueness.

Practice

1. What is the main difference between automatic and manual document IDs in Elasticsearch?

easy

A. Automatic IDs are generated by Elasticsearch; manual IDs are set by the user.

B. Automatic IDs require user input; manual IDs are generated by Elasticsearch.

C. Automatic IDs are always numeric; manual IDs are always strings.

D. Automatic IDs cannot be used for updates; manual IDs cannot.

Solution

1. Step 1: Understand automatic ID generation

   Elasticsearch creates unique IDs automatically when you don't specify one.

2. Step 2: Understand manual ID assignment

   Users can assign their own IDs to documents for control and easier updates.

3. Final Answer:

   Automatic IDs are generated by Elasticsearch; manual IDs are set by the user. -> Option A

4. Quick Check:

   Auto = Elasticsearch generated, Manual = user set [OK]

Hint: Auto = system creates ID, manual = you create ID [OK]

Common Mistakes:

• Confusing which side creates the ID
• Thinking automatic IDs are numeric only
• Believing manual IDs cannot be updated

2. Which of the following is the correct syntax to index a document with a manual ID in Elasticsearch using JSON?

easy

A. { "index": "my_index", "_id": "123", "body": { "name": "Alice" } }

B. { "index": "my_index", "id": "123", "body": { "name": "Alice" } }

C. { "index": "my_index", "id": 123, "doc": { "name": "Alice" } }

D. { "index": "my_index", "_id": 123, "doc": { "name": "Alice" } }

Solution

1. Step 1: Identify the correct field for manual ID

   Elasticsearch uses "_id" (with underscore) to specify manual document IDs.

2. Step 2: Check the correct structure for indexing

   The "body" key holds the document content; "doc" is used for updates, not indexing.

3. Final Answer:

   { "index": "my_index", "_id": "123", "body": { "name": "Alice" } } -> Option A

4. Quick Check:

   Manual ID uses "_id" and "body" for document [OK]

Hint: Use "_id" for manual IDs and "body" for document content [OK]

Common Mistakes:

• Using "id" instead of "_id"
• Confusing "doc" with "body" for indexing
• Passing numeric ID without quotes

3. Given this indexing request with manual ID:

{ "index": "products", "_id": "p100", "body": { "name": "Pen", "price": 1.5 } }

What happens if you send the same request again with updated price 1.8?

medium

A. A new document is created with a different ID.

B. The existing document with ID "p100" is updated with the new price.

C. An error occurs because duplicate IDs are not allowed.

D. The document is ignored and no changes happen.

Solution

1. Step 1: Understand manual ID behavior on indexing

   Using the same manual ID overwrites the existing document with that ID.

2. Step 2: Check the effect of sending updated data

   The document with ID "p100" will be updated to have price 1.8.

3. Final Answer:

   The existing document with ID "p100" is updated with the new price. -> Option B

4. Quick Check:

   Same manual ID means update existing document [OK]

Hint: Same manual ID updates document, no new doc created [OK]

Common Mistakes:

• Thinking a new document is created with same ID
• Expecting an error on duplicate ID
• Assuming no change happens on reindex

4. You tried to index a document with this JSON:

{ "index": "users", "id": "u123", "body": { "name": "Bob" } }

But the document was not saved with ID "u123". What is the likely problem?

medium

A. The "body" field is missing required fields.

B. The index name "users" is invalid.

C. The field "id" should be "_id" to set manual ID.

D. Manual IDs are not allowed in Elasticsearch.

Solution

1. Step 1: Check the ID field name

   Elasticsearch requires "_id" (with underscore) to specify manual IDs, not "id".

2. Step 2: Confirm impact of wrong field

   Using "id" means Elasticsearch ignores it and generates an automatic ID instead.

3. Final Answer:

   The field "id" should be "_id" to set manual ID. -> Option C

4. Quick Check:

   Manual ID needs "_id", not "id" [OK]

Hint: Use "_id" to set manual ID, not "id" [OK]

Common Mistakes:

• Using "id" instead of "_id"
• Assuming missing fields cause ID issues
• Thinking manual IDs are disallowed

5. You want to avoid duplicate documents in Elasticsearch. Which strategy can help you achieve this?

hard

A. Use a random string as manual ID without relation to document content.

B. Always use automatic IDs and ignore duplicates.

C. Index documents without IDs and rely on timestamps to find duplicates.

D. Use a manual ID based on a unique field like email to prevent duplicates.

Solution

1. Step 1: Understand duplicate prevention

   Duplicates can be avoided by assigning manual IDs based on unique fields like email.

2. Step 2: Compare with automatic ID approach

   Automatic IDs are random and do not prevent duplicates because each document gets a new ID.

3. Final Answer:

   Use a manual ID based on a unique field like email to prevent duplicates. -> Option D

4. Quick Check:

   Manual unique IDs prevent duplicates, auto IDs do not [OK]

Hint: Manual unique IDs prevent duplicates better than auto IDs [OK]

Common Mistakes:

• Thinking auto IDs prevent duplicates
• Using random manual IDs unrelated to content
• Relying on timestamps instead of IDs