Updating documents in Elasticsearch - Deep Dive

Overview - Updating documents

What is it?

Updating documents in Elasticsearch means changing the data stored in an existing document without replacing the entire document. Instead of deleting and re-adding, you modify only the parts you want to change. This helps keep your data fresh and accurate while saving time and resources.

Why it matters

Without the ability to update documents, you would have to delete and re-index entire documents every time you want to change something. This would be slow, inefficient, and could cause data loss or inconsistency. Updating documents allows real-time data correction and smooth user experiences in search applications.

Where it fits

Before learning document updates, you should understand how Elasticsearch stores and indexes documents. After mastering updates, you can explore advanced features like scripting updates, partial updates, and version control to handle complex data changes safely.

Mental Model

Core Idea

Updating a document in Elasticsearch means changing only the parts you want without replacing the whole document, keeping data fresh efficiently.

Think of it like...

Imagine a library card catalog where you can erase and rewrite just the phone number on a card instead of replacing the entire card every time the number changes.

┌───────────────┐
│ Document ID   │
├───────────────┤
│ Field A: old  │
│ Field B: old  │
└───────────────┘
       ↓ update Field B
┌───────────────┐
│ Document ID   │
├───────────────┤
│ Field A: old  │
│ Field B: new  │
└───────────────┘

Build-Up - 7 Steps

1

Foundation

What is a document update

Concept: Understanding that updating means changing parts of a stored document without full replacement.

In Elasticsearch, documents are JSON objects stored in an index. Updating a document means changing some fields inside it. You do this by specifying the document's ID and the new values for the fields you want to change.

Result

You can change data inside a document without deleting and re-adding it.

Knowing that updates modify only parts of a document helps you avoid unnecessary work and keeps your data consistent.

2

Foundation

How to update documents simply

Concept: Using the Update API to change document fields by ID.

You use the Elasticsearch Update API with the document's ID. You send a JSON body with the fields to change. For example, to update a user's name, you send {"doc": {"name": "new name"}}. Elasticsearch merges this with the existing document.

Result

The document's specified fields are changed, others stay the same.

Understanding the Update API basics lets you efficiently fix or add data without full reindexing.

3

Intermediate

Partial updates with scripting

🤔Before reading on: do you think you can update a field by adding a number to it directly, or must you send the full new value? Commit to your answer.

Concept: Using scripts to perform complex updates like incrementing values or conditional changes.

Sometimes you want to change a field based on its current value, like increasing a counter. Elasticsearch lets you write painless scripts in the update request. For example, {"script": {"source": "ctx._source.counter += params.count", "params": {"count": 1}}} increments the counter by 1.

Result

Fields can be updated dynamically based on current data.

Knowing scripting updates unlocks powerful, flexible data changes without fetching and resubmitting documents.

4

Intermediate

Handling update conflicts

🤔Before reading on: do you think Elasticsearch automatically handles two users updating the same document at the same time, or can conflicts happen? Commit to your answer.

Concept: Understanding version conflicts and how to manage them during updates.

When multiple updates happen simultaneously, Elasticsearch uses optimistic concurrency control. Each document has a version number. If you try to update a document with an old version, Elasticsearch rejects it to prevent overwriting newer changes. You can handle this by retrying or using the 'retry_on_conflict' parameter.

Result

Updates are safe from accidental overwrites but may require retries.

Knowing about conflicts helps you build reliable update logic in concurrent environments.

5

Advanced

Upsert: update or insert

🤔Before reading on: do you think an update request can create a document if it doesn't exist, or only update existing ones? Commit to your answer.

Concept: Using upsert to update a document if it exists or create it if it doesn't.

Elasticsearch lets you combine update and insert with 'upsert'. You provide a 'doc' to update and an 'upsert' document to create if missing. This avoids separate checks and makes your code simpler.

Result

Documents are created or updated in one atomic operation.

Understanding upsert simplifies workflows where you want to ensure data presence without extra queries.

6

Advanced

Performance considerations for updates

🤔Before reading on: do you think updates in Elasticsearch modify data in place on disk, or do they work differently? Commit to your answer.

Concept: Learning how Elasticsearch handles updates internally and their impact on performance.

Elasticsearch does not update documents in place. Instead, it marks the old document as deleted and adds a new version. This means frequent updates can increase disk usage and slow searches until segments are merged. Planning update frequency and using bulk updates helps performance.

Result

You understand update costs and how to optimize them.

Knowing the internal update mechanism helps you design efficient, scalable Elasticsearch systems.

7

Expert

Advanced scripting and painless nuances

🤔Before reading on: do you think Elasticsearch scripts can access all document fields freely, or are there restrictions? Commit to your answer.

Concept: Deep dive into painless scripting language features and limitations for updates.

Painless scripts run safely inside Elasticsearch with limited access to prevent security risks. Scripts can read and modify fields but cannot perform network calls or access external resources. Understanding script context, parameters, and error handling is key to writing robust update scripts.

Result

You can write complex, safe update scripts that handle edge cases.

Mastering painless scripting nuances prevents bugs and security issues in production updates.

Under the Hood

When you update a document, Elasticsearch does not change the stored data directly. Instead, it marks the old document as deleted and writes a new document with the updated data. This is because Elasticsearch uses immutable segments for storage. The update process involves fetching the current document, applying changes (via doc or script), and indexing the new version. Later, background merges clean up deleted documents to optimize storage.

Why designed this way?

This design allows Elasticsearch to be fast and scalable for search queries. Immutable segments simplify concurrency and reduce locking. Although updates are more costly than simple writes, this tradeoff ensures high read performance and data integrity. Alternatives like in-place updates would complicate concurrency and slow down searches.

┌───────────────┐       ┌───────────────┐
│ Old Document  │──────▶│ Marked Deleted│
└───────────────┘       └───────────────┘
                             │
                             ▼
                      ┌───────────────┐
                      │ New Document  │
                      └───────────────┘
                             │
                             ▼
                    ┌───────────────────┐
                    │ Background Merge  │
                    │ Cleans Deleted    │
                    └───────────────────┘

Myth Busters - 4 Common Misconceptions

Quick: Does updating a document replace the entire document or only the specified fields? Commit to your answer.

Common Belief: Updating a document replaces the whole document with the new data.

Reality: Updating modifies only the specified fields, leaving others unchanged.

Why it matters: Believing updates replace whole documents can lead to accidental data loss if you send partial data thinking it will merge.

Quick: Can you update a document without knowing its current version safely? Commit to your answer.

Common Belief: You can update documents anytime without worrying about version conflicts.

Reality: Elasticsearch uses versioning to prevent conflicting updates; ignoring this can cause update failures.

Why it matters: Ignoring version conflicts can cause lost updates or errors in concurrent environments.

Quick: Does Elasticsearch update documents in place on disk? Commit to your answer.

Common Belief: Updates modify the document directly on disk to save space and time.

Reality: Updates create new document versions and mark old ones deleted; no in-place modification occurs.

Why it matters: Misunderstanding this can cause surprise at disk usage growth and slower performance with many updates.

Quick: Can you use any programming language for update scripts in Elasticsearch? Commit to your answer.

Common Belief: You can write update scripts in any language you want.

Reality: Elasticsearch supports only the painless scripting language for updates to ensure safety and performance.

Why it matters: Trying unsupported languages wastes time and can cause security risks if unsafe scripts were allowed.

Expert Zone

1

Update requests can include 'detect_noop' to skip indexing if the update does not change the document, saving resources.

2

Using 'retry_on_conflict' with scripting updates is essential in high-concurrency environments to avoid lost updates.

3

Bulk update operations can improve throughput but require careful error handling to avoid partial failures.

When NOT to use

Avoid frequent small updates on very large documents or high-update-rate indices; instead, consider denormalizing data or using append-only logs. For real-time updates with minimal overhead, consider using external caches or databases designed for fast writes.

Production Patterns

In production, updates are often batched using the Bulk API for efficiency. Upserts are common for user profiles or counters. Scripts handle complex logic like conditional increments or timestamp updates. Monitoring update conflicts and segment merges helps maintain performance.

Connections

Version Control Systems

Both use versioning to manage changes and prevent conflicts.

Understanding how Elasticsearch uses version numbers to avoid update conflicts is similar to how Git manages concurrent changes in code.

Immutable Data Structures

Elasticsearch stores data immutably, creating new versions instead of changing existing ones.

Knowing immutable data principles helps explain why Elasticsearch marks old documents deleted and writes new ones on update.

Transactional Systems in Banking

Both require safe, consistent updates with conflict handling to avoid data corruption.

Learning how Elasticsearch handles update conflicts and retries parallels how banking systems ensure transaction integrity under concurrent access.

Common Pitfalls

#1 Trying to update a document by sending only the changed fields without using the 'doc' keyword.

Wrong approach:

POST /index/_update/1
{
"name": "new name"
}

Correct approach:

POST /index/_update/1
{
"doc": {"name": "new name"}
}

Root cause:

Misunderstanding the Update API format causes Elasticsearch to reject or ignore the update.

#2 Ignoring version conflicts and not handling retries in concurrent updates.

Wrong approach:

POST /index/_update/1
{
"doc": {"counter": 5}
}

Correct approach:

POST /index/_update/1?retry_on_conflict=3
{
"script": {"source": "ctx._source.counter += params.count", "params": {"count": 1}}
}

Root cause:

Not using 'retry_on_conflict' or scripting leads to lost updates when multiple clients write simultaneously.

#3 Using update requests for very frequent changes on large documents without considering performance.

Wrong approach:

Updating a large document field-by-field many times per second without batching.

Correct approach:

Batch updates using the Bulk API and minimize update frequency or redesign data model.

Root cause:

Not understanding Elasticsearch's immutable segment storage causes performance degradation and disk bloat.

Key Takeaways

Updating documents in Elasticsearch changes only specified fields without replacing the whole document.

Elasticsearch uses versioning and optimistic concurrency to prevent conflicting updates and data loss.

Updates create new document versions and mark old ones deleted, which affects performance and storage.

Scripting enables dynamic and conditional updates, unlocking powerful data manipulation.

Using upsert combines update and insert in one operation, simplifying data workflows.