Reindexing data in Elasticsearch - Deep Dive

Overview - Reindexing data

What is it?

Reindexing data in Elasticsearch means copying data from one index to another. This process allows you to change the structure or settings of your data without losing it. It is like making a fresh copy of your data with improvements or fixes. This helps keep your search system fast and accurate.

Why it matters

Without reindexing, you cannot easily update the way your data is stored or searched. If your data structure is outdated or incorrect, your searches may be slow or wrong. Reindexing solves this by letting you create a new, improved version of your data without downtime or data loss. This keeps your system reliable and efficient.

Where it fits

Before learning reindexing, you should understand basic Elasticsearch concepts like indexes, documents, and mappings. After mastering reindexing, you can explore advanced topics like index templates, aliases, and performance tuning. Reindexing is a key skill for managing data lifecycle in Elasticsearch.

Mental Model

Core Idea

Reindexing is the process of copying and transforming data from one Elasticsearch index to another to update or improve its structure without losing data.

Think of it like...

Imagine you have a photo album with old pictures glued in. Reindexing is like carefully taking each photo out, fixing or enhancing it, and placing it into a new album that looks better and is easier to browse.

┌───────────────┐       ┌───────────────┐
│ Source Index  │──────▶│ Reindexing    │
│ (old data)    │       │ Process       │
└───────────────┘       └───────────────┘
                              │
                              ▼
                      ┌───────────────┐
                      │ Target Index  │
                      │ (new data)    │
                      └───────────────┘

Build-Up - 7 Steps

1

Foundation

Understanding Elasticsearch Index Basics

Concept: Learn what an index is and how data is stored in Elasticsearch.

An Elasticsearch index is like a folder that holds many documents. Each document is a piece of data with fields, like a record in a spreadsheet. Indexes organize data so you can search it quickly. Knowing this helps you understand why you might want to copy or change an index.

Result

You know that an index holds data and that documents inside have fields you can search.

Understanding the role of an index is essential because reindexing moves data between these containers to improve or fix them.

2

Foundation

What is Reindexing in Elasticsearch

Concept: Introduce the idea of copying data from one index to another to update or fix it.

Reindexing means taking all documents from one index and copying them into a new index. This lets you change how data is stored or searched without deleting anything. For example, you can fix mistakes in field types or add new fields during reindexing.

Result

You understand that reindexing creates a new index with data copied from the old one, possibly changed.

Knowing that reindexing is a safe way to update data structure helps you manage data without losing it.

3

Intermediate

Using the Reindex API

🤔Before reading on: do you think reindexing changes data in place or creates a new copy? Commit to your answer.

Concept: Learn how to use Elasticsearch's Reindex API to perform the copying process.

Elasticsearch provides a Reindex API that lets you specify the source index and the target index. When you run it, Elasticsearch copies documents from source to target. You can also add scripts to modify data during this process. For example: POST _reindex { "source": { "index": "old_index" }, "dest": { "index": "new_index" } } This command copies all data from 'old_index' to 'new_index'.

Result

A new index 'new_index' is created with all documents from 'old_index'.

Understanding the Reindex API shows how Elasticsearch safely copies data and allows changes during the process.

4

Intermediate

Changing Mappings During Reindexing

🤔Before reading on: can you change the data structure while reindexing, or only copy as-is? Commit to your answer.

Concept: Learn how to update field types or add new fields by creating a new index with desired mappings before reindexing.

You cannot change mappings directly during reindexing. Instead, you create a new index with the desired mappings first. Then you reindex data into it. For example, if you want a field to be a keyword instead of text, define that in the new index mapping. After reindexing, the data fits the new structure.

Result

Data is copied into a new index with updated field types or settings.

Knowing that reindexing works with a pre-created target index helps you plan schema changes safely.

5

Intermediate

Reindexing with Data Transformation Scripts

🤔Before reading on: do you think you can modify data values during reindexing? Commit to your answer.

Concept: Learn how to use painless scripts in the Reindex API to change data as it copies.

Elasticsearch lets you add a script to the Reindex API to modify documents during copying. For example, you can rename fields, change values, or add new fields. Example: POST _reindex { "source": { "index": "old_index" }, "dest": { "index": "new_index" }, "script": { "source": "ctx._source.new_field = ctx._source.old_field + '_updated'" } } This adds a new field based on an old one.

Result

The new index contains documents with modified data as specified by the script.

Understanding scripting during reindexing unlocks powerful data transformation without extra steps.

6

Advanced

Handling Large Data and Performance

🤔Before reading on: do you think reindexing large indexes happens instantly or takes time? Commit to your answer.

Concept: Learn strategies to reindex large datasets efficiently without impacting cluster performance.

Reindexing large indexes can take time and use resources. You can control this by: - Using 'slice' to split reindexing into parallel tasks. - Limiting requests per second to reduce load. - Running reindex during low traffic times. Example with slicing: POST _reindex { "source": { "index": "old_index", "slice": { "id": 0, "max": 2 } }, "dest": { "index": "new_index" } } Run slices with id 0 and 1 in parallel.

Result

Reindexing completes faster and with less impact on the cluster.

Knowing how to manage resources during reindexing prevents downtime and keeps your system healthy.

7

Expert

Reindexing and Index Aliases for Zero Downtime

🤔Before reading on: can you update an index without stopping searches? Commit to your answer.

Concept: Learn how to use index aliases to switch from old to new index seamlessly after reindexing.

Index aliases are like nicknames for indexes. You can point an alias to one index at a time. To update data without downtime: 1. Create a new index with updated mappings. 2. Reindex data into the new index. 3. Switch the alias from old to new index. 4. Delete the old index if desired. This way, your applications always query the alias and never notice the switch.

Result

Searches continue without interruption while data structure updates happen behind the scenes.

Understanding aliases with reindexing enables smooth upgrades and high availability in production.

Under the Hood

Reindexing works by reading documents from the source index using a scroll search, which efficiently fetches batches of documents. Each document is then indexed into the target index as a new document. If a script is provided, it modifies the document before indexing. This process happens in the cluster and can be parallelized. The source index remains unchanged during this operation.

Why designed this way?

Elasticsearch separates reading and writing to avoid locking or downtime. Using scroll search ensures stable snapshots of data during reindexing. The design allows flexible data transformation and safe schema changes. Alternatives like in-place mapping changes were limited or risky, so reindexing provides a controlled, reliable method.

┌───────────────┐       ┌───────────────┐       ┌───────────────┐
│ Source Index  │──────▶│ Scroll Search │──────▶│ Document      │
│ (read only)   │       │ (batch fetch) │       │ Transformation│
└───────────────┘       └───────────────┘       └───────────────┘
                                                      │
                                                      ▼
                                              ┌───────────────┐
                                              │ Target Index  │
                                              │ (write new)   │
                                              └───────────────┘

Myth Busters - 4 Common Misconceptions

Quick: Does reindexing modify the original index data? Commit yes or no.

Common Belief: Reindexing changes data inside the original index directly.

Reality: Reindexing copies data to a new index; the original index remains unchanged.

Why it matters: Believing this causes fear of data loss or confusion about how to update mappings safely.

Quick: Can you change field types on the fly during reindexing? Commit yes or no.

Common Belief: You can change field types directly during the reindexing process without creating a new index first.

Reality: You must create the target index with the desired mappings before reindexing; reindexing itself does not change mappings.

Why it matters: Misunderstanding this leads to failed reindexing or incorrect data structure.

Quick: Does reindexing always happen instantly? Commit yes or no.

Common Belief: Reindexing is a quick operation regardless of data size.

Reality: Reindexing large datasets can take significant time and resources, requiring careful management.

Why it matters: Ignoring this can cause unexpected downtime or cluster overload.

Quick: Is it safe to delete the old index immediately after starting reindexing? Commit yes or no.

Common Belief: Once reindexing starts, the old index is no longer needed and can be deleted immediately.

Reality: You should keep the old index until reindexing completes and the new index is verified to avoid data loss.

Why it matters: Deleting too early risks losing data if reindexing fails or is incomplete.

Expert Zone

1

Reindexing does not preserve internal document versioning or routing unless explicitly handled, which can affect update conflicts.

2

Using slices for parallel reindexing improves speed but requires careful coordination to avoid duplicate or missed documents.

3

Reindexing can be combined with index aliases and write blocks to achieve zero downtime upgrades in production environments.

When NOT to use

Reindexing is not suitable for real-time data updates or small fixes; use update APIs for minor changes. Also, avoid reindexing if the cluster is under heavy load; consider offline maintenance windows or incremental reindexing instead.

Production Patterns

In production, teams often create new indexes with updated mappings, reindex data during off-peak hours using slices, then atomically switch aliases to the new index. They monitor reindex progress and validate data before deleting old indexes to ensure reliability.

Connections

Database Migration

Reindexing is a form of data migration within Elasticsearch, similar to moving data between database schemas.

Understanding reindexing helps grasp how data migrations work in other systems, emphasizing safe data transformation and minimal downtime.

Version Control Systems

Reindexing with aliases is like branching and merging in version control, allowing smooth transitions between data versions.

This connection shows how managing data versions and updates can follow similar principles across software and data systems.

Supply Chain Management

Reindexing resembles repackaging products in a supply chain to improve quality or presentation before delivery.

Seeing reindexing as repackaging clarifies why data needs transformation and careful handling before being 'delivered' to users.

Common Pitfalls

#1 Deleting the old index before confirming reindex success.

Wrong approach:

DELETE /old_index

POST _reindex
{
"source": { "index": "old_index" },
"dest": { "index": "new_index" }
}

Correct approach:

POST _reindex
{
"source": { "index": "old_index" },
"dest": { "index": "new_index" }
}

# After verifying new_index
DELETE /old_index

Root cause:

Misunderstanding that reindexing is asynchronous and that the old data is still needed until the new index is fully ready.

#2 Trying to change field types by reindexing without creating the new index first.

Wrong approach:

POST _reindex
{
"source": { "index": "old_index" },
"dest": { "index": "old_index" }
}

Correct approach:

PUT /new_index
{
"mappings": { "properties": { "field": { "type": "keyword" } } }
}

POST _reindex
{
"source": { "index": "old_index" },
"dest": { "index": "new_index" }
}

Root cause:

Believing reindexing can modify mappings on the fly instead of requiring a new index with desired mappings.

#3 Running reindex without controlling resource usage on large datasets.

Wrong approach:

POST _reindex
{
"source": { "index": "big_index" },
"dest": { "index": "new_index" }
}

Correct approach:

POST _reindex
{
"source": { "index": "big_index", "slice": { "id": 0, "max": 4 } },
"dest": { "index": "new_index" },
"max_docs": 10000
}

Root cause:

Not accounting for cluster load and the time needed to process large volumes of data safely.

Key Takeaways

Reindexing copies data from one Elasticsearch index to another, enabling safe updates to data structure and settings.

You must create the target index with desired mappings before reindexing; the process itself copies documents without changing mappings.

Using the Reindex API with scripts allows data transformation during copying, making reindexing a powerful tool for data fixes and improvements.

Managing large data reindexing requires techniques like slicing and rate limiting to avoid performance issues.

Combining reindexing with index aliases enables zero downtime upgrades, keeping search services available during data updates.