Document ID strategies (auto vs manual) in Elasticsearch - Hands-On Comparison

Document ID strategies (auto vs manual)

📖 Scenario: You are managing a small online store's product catalog using Elasticsearch. You want to add products to the catalog and decide how to assign document IDs. Sometimes you want Elasticsearch to create IDs automatically, and other times you want to assign your own IDs to keep track of products easily.

🎯 Goal: Learn how to add documents to Elasticsearch with both automatic and manual document IDs, and see the difference in how documents are stored and retrieved.

📋 What You'll Learn

Create an index called products

Add a product document with an automatic ID

Add a product document with a manual ID

Retrieve and print the documents to see their IDs

💡 Why This Matters

🌍 Real World

In real online stores or apps, you often need to store product data in Elasticsearch. Choosing between automatic and manual document IDs helps you manage and find products easily.

💼 Career

Knowing how to manage document IDs in Elasticsearch is important for backend developers and data engineers working with search and data storage systems.

1

Create the products index

Write a command to create an Elasticsearch index called products.

Elasticsearch

# Write the command to create the 'products' index
# Your code here

Hint

Use the PUT method with the index name products to create the index.

2

Add a product with an automatic document ID

Write a command to add a product document with name as "Coffee Mug" and price as 12.99 to the products index. Let Elasticsearch create the document ID automatically by not specifying it.

Elasticsearch

PUT /products
{}

# Add a product document with automatic ID below
# Your code here

Hint

Use POST /products/_doc to add a document with an automatic ID.

3

Add a product with a manual document ID

Write a command to add a product document with name as "Tea Cup" and price as 9.99 to the products index. Use the manual document ID tea-cup-001.

Elasticsearch

PUT /products
{}

POST /products/_doc
{
  "name": "Coffee Mug",
  "price": 12.99
}

# Add a product document with manual ID 'tea-cup-001' below
# Your code here

Hint

Use PUT /products/_doc/tea-cup-001 to add a document with a manual ID.

4

Retrieve and print the documents to see their IDs

Write commands to retrieve and print the documents with the manual ID tea-cup-001 and the automatically created document ID for the "Coffee Mug" product. Use the GET method for both.

Elasticsearch

PUT /products
{}

POST /products/_doc
{
  "name": "Coffee Mug",
  "price": 12.99
}

PUT /products/_doc/tea-cup-001
{
  "name": "Tea Cup",
  "price": 9.99
}

# Retrieve and print the document with manual ID 'tea-cup-001'
# Retrieve and print the document with automatic ID (replace  with actual ID)
# Your code here

Hint

Use GET /products/_doc/tea-cup-001 to get the manual ID document. For the automatic ID, you need to know the ID from the response or list documents.

Practice

1. What is the main difference between automatic and manual document IDs in Elasticsearch?

easy

A. Automatic IDs are generated by Elasticsearch; manual IDs are set by the user.

B. Automatic IDs require user input; manual IDs are generated by Elasticsearch.

C. Automatic IDs are always numeric; manual IDs are always strings.

D. Automatic IDs cannot be used for updates; manual IDs cannot.

Solution

1. Step 1: Understand automatic ID generation

   Elasticsearch creates unique IDs automatically when you don't specify one.

2. Step 2: Understand manual ID assignment

   Users can assign their own IDs to documents for control and easier updates.

3. Final Answer:

   Automatic IDs are generated by Elasticsearch; manual IDs are set by the user. -> Option A

4. Quick Check:

   Auto = Elasticsearch generated, Manual = user set [OK]

Hint: Auto = system creates ID, manual = you create ID [OK]

Common Mistakes:

• Confusing which side creates the ID
• Thinking automatic IDs are numeric only
• Believing manual IDs cannot be updated

2. Which of the following is the correct syntax to index a document with a manual ID in Elasticsearch using JSON?

easy

A. { "index": "my_index", "_id": "123", "body": { "name": "Alice" } }

B. { "index": "my_index", "id": "123", "body": { "name": "Alice" } }

C. { "index": "my_index", "id": 123, "doc": { "name": "Alice" } }

D. { "index": "my_index", "_id": 123, "doc": { "name": "Alice" } }

Solution

1. Step 1: Identify the correct field for manual ID

   Elasticsearch uses "_id" (with underscore) to specify manual document IDs.

2. Step 2: Check the correct structure for indexing

   The "body" key holds the document content; "doc" is used for updates, not indexing.

3. Final Answer:

   { "index": "my_index", "_id": "123", "body": { "name": "Alice" } } -> Option A

4. Quick Check:

   Manual ID uses "_id" and "body" for document [OK]

Hint: Use "_id" for manual IDs and "body" for document content [OK]

Common Mistakes:

• Using "id" instead of "_id"
• Confusing "doc" with "body" for indexing
• Passing numeric ID without quotes

3. Given this indexing request with manual ID:

{ "index": "products", "_id": "p100", "body": { "name": "Pen", "price": 1.5 } }

What happens if you send the same request again with updated price 1.8?

medium

A. A new document is created with a different ID.

B. The existing document with ID "p100" is updated with the new price.

C. An error occurs because duplicate IDs are not allowed.

D. The document is ignored and no changes happen.

Solution

1. Step 1: Understand manual ID behavior on indexing

   Using the same manual ID overwrites the existing document with that ID.

2. Step 2: Check the effect of sending updated data

   The document with ID "p100" will be updated to have price 1.8.

3. Final Answer:

   The existing document with ID "p100" is updated with the new price. -> Option B

4. Quick Check:

   Same manual ID means update existing document [OK]

Hint: Same manual ID updates document, no new doc created [OK]

Common Mistakes:

• Thinking a new document is created with same ID
• Expecting an error on duplicate ID
• Assuming no change happens on reindex

4. You tried to index a document with this JSON:

{ "index": "users", "id": "u123", "body": { "name": "Bob" } }

But the document was not saved with ID "u123". What is the likely problem?

medium

A. The "body" field is missing required fields.

B. The index name "users" is invalid.

C. The field "id" should be "_id" to set manual ID.

D. Manual IDs are not allowed in Elasticsearch.

Solution

1. Step 1: Check the ID field name

   Elasticsearch requires "_id" (with underscore) to specify manual IDs, not "id".

2. Step 2: Confirm impact of wrong field

   Using "id" means Elasticsearch ignores it and generates an automatic ID instead.

3. Final Answer:

   The field "id" should be "_id" to set manual ID. -> Option C

4. Quick Check:

   Manual ID needs "_id", not "id" [OK]

Hint: Use "_id" to set manual ID, not "id" [OK]

Common Mistakes:

• Using "id" instead of "_id"
• Assuming missing fields cause ID issues
• Thinking manual IDs are disallowed

5. You want to avoid duplicate documents in Elasticsearch. Which strategy can help you achieve this?

hard

A. Use a random string as manual ID without relation to document content.

B. Always use automatic IDs and ignore duplicates.

C. Index documents without IDs and rely on timestamps to find duplicates.

D. Use a manual ID based on a unique field like email to prevent duplicates.

Solution

1. Step 1: Understand duplicate prevention

   Duplicates can be avoided by assigning manual IDs based on unique fields like email.

2. Step 2: Compare with automatic ID approach

   Automatic IDs are random and do not prevent duplicates because each document gets a new ID.

3. Final Answer:

   Use a manual ID based on a unique field like email to prevent duplicates. -> Option D

4. Quick Check:

   Manual unique IDs prevent duplicates, auto IDs do not [OK]

Hint: Manual unique IDs prevent duplicates better than auto IDs [OK]

Common Mistakes:

• Thinking auto IDs prevent duplicates
• Using random manual IDs unrelated to content
• Relying on timestamps instead of IDs