Deleting documents in Elasticsearch - Deep Dive

Overview - Deleting documents

What is it?

Deleting documents in Elasticsearch means removing specific pieces of data from an index. Each document is like a record or entry stored in Elasticsearch. When you delete a document, it no longer appears in search results or queries. This helps keep your data up-to-date and relevant.

Why it matters

Without the ability to delete documents, your Elasticsearch index would keep growing with outdated or incorrect data. This would make searches slower and less accurate, causing confusion and wasted resources. Deleting documents ensures your data stays clean and your searches return the right information quickly.

Where it fits

Before learning to delete documents, you should understand how Elasticsearch stores and indexes documents. After mastering deletion, you can learn about updating documents and managing index lifecycle for efficient data handling.

Mental Model

Core Idea

Deleting a document in Elasticsearch marks it as removed so it no longer appears in searches, but the actual removal happens later during index cleanup.

Think of it like...

Imagine a library where you cross out a book's entry in the catalog to show it's no longer available, but the book stays on the shelf until the librarian removes it during cleaning.

┌─────────────┐       ┌───────────────┐       ┌───────────────┐
│ Document in │──────▶│ Marked as     │──────▶│ Physically     │
│ Elasticsearch│       │ Deleted (flag)│       │ Removed later │
└─────────────┘       └───────────────┘       └───────────────┘

Build-Up - 7 Steps

1

Foundation

What is a document in Elasticsearch

Concept: Understanding the basic unit of data storage in Elasticsearch.

A document is a JSON object that holds data in Elasticsearch. It is stored inside an index and has a unique ID. For example, a document could represent a user profile or a product listing.

Result

You know that documents are the pieces of data you can add, search, update, or delete in Elasticsearch.

Knowing what a document is helps you understand what you are deleting when you remove data.

2

Foundation

How deletion works conceptually

Concept: Deletion in Elasticsearch does not immediately remove data but marks it for removal.

When you delete a document, Elasticsearch marks it as deleted internally. The document still exists physically until a process called segment merging cleans it up. This means deletion is fast and does not block other operations.

Result

You understand that deletion is a two-step process: marking and later physical removal.

Understanding this prevents confusion about why deleted documents might still appear briefly or why disk space is not freed immediately.

3

Intermediate

Deleting a document by ID

🤔Before reading on: do you think deleting a document requires searching for it first or can you delete directly by ID? Commit to your answer.

Concept: You can delete a document directly if you know its unique ID.

Elasticsearch provides a Delete API where you specify the index and the document's ID to remove it. For example, a DELETE request to /index/_doc/document_id removes that document.

Result

The document with the given ID is marked as deleted and will no longer appear in search results.

Knowing you can delete by ID makes deletion efficient and precise without extra searching.

4

Intermediate

Deleting documents by query

🤔Before reading on: do you think deleting by query removes documents immediately or marks them first? Commit to your answer.

Concept: You can delete multiple documents matching a search query at once.

Elasticsearch's Delete By Query API lets you specify a query to find documents to delete. All matching documents are marked as deleted. For example, deleting all documents where status is 'inactive'.

Result

All documents matching the query are marked deleted and excluded from future searches.

Deleting by query allows bulk removal based on conditions, saving time over deleting individually.

5

Intermediate

Handling version conflicts during deletion

🤔Before reading on: do you think deleting a document can fail if the document changed since you last saw it? Commit to your answer.

Concept: Elasticsearch uses versioning to prevent deleting a document that has changed unexpectedly.

When deleting, you can specify a version number. If the document's current version doesn't match, the delete fails to avoid removing updated data by mistake.

Result

You avoid accidental deletion of documents that have been updated concurrently.

Understanding version conflicts helps maintain data integrity in concurrent environments.

6

Advanced

Impact of deletion on index performance

🤔Before reading on: do you think frequent deletions improve or degrade Elasticsearch performance? Commit to your answer.

Concept: Frequent deletions can cause index fragmentation and affect performance until cleanup happens.

Deleted documents remain in segments until merged. Many deletions increase segment count and slow searches. Elasticsearch periodically merges segments to reclaim space and optimize performance.

Result

You see that deletion affects index health and that maintenance is needed for optimal speed.

Knowing this helps you plan deletion frequency and index maintenance to keep Elasticsearch fast.

7

Expert

Internals of segment merging after deletion

🤔Before reading on: do you think segment merging happens immediately after deletion or is scheduled? Commit to your answer.

Concept: Segment merging is a background process that physically removes deleted documents from storage.

Elasticsearch stores data in immutable segments. When documents are deleted, they are flagged but not removed. The merge process combines segments, skipping deleted documents, freeing disk space and improving search speed.

Result

Deleted documents are fully removed only after merges, which happen asynchronously.

Understanding segment merging explains why deletions don't free space instantly and why index tuning matters.

Under the Hood

Elasticsearch stores documents in immutable segments on disk. When a document is deleted, it is not immediately removed but marked with a tombstone flag. This marking is fast and does not rewrite segments. Later, a background process called segment merging combines smaller segments into larger ones, skipping deleted documents. This process physically removes deleted data and reclaims disk space. Until merging, deleted documents still occupy space but are invisible to searches.

Why designed this way?

This design balances speed and consistency. Immediate physical deletion would require rewriting large files, slowing down operations. Marking deletions allows fast writes and searches without blocking. Segment merging runs in the background to optimize storage and performance. Alternatives like immediate deletion were rejected because they hurt responsiveness and scalability.

┌───────────────┐       ┌───────────────┐       ┌───────────────┐
│ Immutable     │       │ Document      │       │ Segment       │
│ Segments on   │──────▶│ Marked as     │──────▶│ Merging       │
│ Disk          │       │ Deleted       │       │ Process       │
└───────────────┘       └───────────────┘       └───────────────┘
        │                      │                       │
        ▼                      ▼                       ▼
  Search reads          Deleted docs hidden       Deleted docs
  skip deleted docs     but still occupy space   physically removed
                        until merge completes

Myth Busters - 4 Common Misconceptions

Quick: Does deleting a document immediately free disk space? Commit yes or no.

Common Belief: Deleting a document instantly removes it and frees disk space.

Reality: Deletion only marks the document as deleted; disk space is freed later during segment merging.

Why it matters: Expecting immediate space recovery can lead to confusion and mismanagement of storage resources.

Quick: Can you delete documents without knowing their IDs? Commit yes or no.

Common Belief: You must know the exact document ID to delete it.

Reality: You can delete documents by query, removing all that match certain conditions without knowing IDs.

Why it matters: Believing you need IDs limits your ability to efficiently clean up data in bulk.

Quick: Does deleting a document guarantee it won't appear in search results immediately? Commit yes or no.

Common Belief: Deleted documents disappear from search results instantly.

Reality: Deleted documents are excluded from searches immediately, but due to eventual consistency and refresh intervals, they might briefly appear.

Why it matters: Misunderstanding this can cause confusion when deleted data still shows up shortly after deletion.

Quick: Is it safe to delete documents without considering version conflicts? Commit yes or no.

Common Belief: You can delete any document anytime without checking if it changed.

Reality: Deleting without handling version conflicts risks removing updated data accidentally.

Why it matters: Ignoring version conflicts can cause data loss in concurrent environments.

Expert Zone

1

Deleted documents still consume disk space until segment merges, so frequent deletions can degrade performance if merges lag.

2

Delete By Query operations are internally implemented as a combination of search and bulk delete, which can be costly on large datasets.

3

Versioning during deletion is crucial in distributed clusters to avoid race conditions and ensure data consistency.

When NOT to use

Avoid frequent individual document deletions in high-throughput systems; instead, consider using time-based indices and deleting entire indices or using index lifecycle management. For real-time deletion needs, consider external data stores optimized for fast deletes.

Production Patterns

In production, deletions are often batched or done via Delete By Query during off-peak hours. Time-series data is managed by deleting whole indices by date. Version conflicts are handled carefully to prevent data loss. Monitoring segment merges and disk usage is standard practice.

Connections

Garbage Collection in Programming

Both mark unused data first and clean it up later.

Understanding how Elasticsearch delays physical deletion like garbage collection helps grasp why deletions are fast but space is reclaimed asynchronously.

Database Transactions

Deletion operations must consider consistency and concurrency like transactions.

Knowing about version conflicts in deletion connects to transaction isolation concepts ensuring data integrity.

Library Catalog Management

Deleting documents is like marking books as removed in a catalog before physically removing them.

This connection shows how real-world inventory systems handle removal in stages, similar to Elasticsearch.

Common Pitfalls

#1 Trying to delete a document without specifying the correct index or ID.

Wrong approach:

DELETE /_doc/12345

Correct approach:

DELETE /my_index/_doc/12345

Root cause:

Not understanding that Elasticsearch requires both index and document ID to locate the document.

#2 Assuming Delete By Query deletes documents instantly and frees disk space immediately.

Wrong approach:

POST /my_index/_delete_by_query { "query": { "match_all": {} } }

Correct approach:

Use Delete By Query as above but monitor segment merges and disk usage to confirm cleanup.

Root cause:

Misunderstanding that deletion marks documents and that physical removal is asynchronous.

#3 Ignoring version conflicts and deleting documents blindly in concurrent environments.

Wrong approach:

DELETE /my_index/_doc/12345 without version parameter

Correct approach:

DELETE /my_index/_doc/12345?version=5

Root cause:

Not handling concurrent updates leads to accidental deletion of newer data.

Key Takeaways

Deleting documents in Elasticsearch marks them as removed but does not immediately erase them from disk.

You can delete documents by their unique ID or by matching queries to remove many at once.

Deleted documents remain invisible to searches but still occupy space until segment merging cleans them up.

Handling version conflicts during deletion prevents accidental data loss in concurrent environments.

Understanding the delayed physical removal and its impact on performance helps maintain healthy Elasticsearch clusters.