Reindexing data in Elasticsearch - Step-by-Step Execution

Concept Flow - Reindexing data

Start Reindex Request

↓

Read Documents from Source Index

↓

Apply Optional Transformations

↓

Write Documents to Destination Index

↓

Check for Errors

↓

Complete or Retry

Reindexing reads documents from one index, optionally changes them, then writes to another index.

Execution Sample

Elasticsearch

{
  "source": { "index": "old_index" },
  "dest": { "index": "new_index" }
}

This JSON tells Elasticsearch to copy all documents from 'old_index' to 'new_index'.

Execution Table

Step	Action	Source Index Docs Read	Docs Written to Destination	Status
1	Start reindex	0	0	Request sent
2	Read batch of 100 docs	100	0	Reading documents
3	Write batch of 100 docs	100	100	Documents written
4	Read next batch of 100 docs	200	100	Reading documents
5	Write next batch of 100 docs	200	200	Documents written
6	Read final batch of 50 docs	250	200	Reading documents
7	Write final batch of 50 docs	250	250	Documents written
8	Reindex complete	250	250	Success

💡 All 250 documents read from source and written to destination index successfully.

Variable Tracker

Variable	Start	After Step 2	After Step 4	After Step 6	Final
docs_read	0	100	200	250	250
docs_written	0	0	100	200	250
status	idle	reading	reading	reading	success

Key Moments - 3 Insights

Why do docs_written lag behind docs_read during the process?

Because Elasticsearch reads a batch of documents first (docs_read increases), then writes them to the destination (docs_written increases). See steps 2 and 3 in execution_table.

What happens if an error occurs during writing?

The reindex process pauses or retries depending on error handling. This is not shown in the current successful trace but would appear as a different status in the execution_table.

Can we transform documents during reindexing?

Yes, you can add scripts or queries to modify documents between reading and writing. This example shows a simple copy without changes.

Visual Quiz - 3 Questions

Look at the execution_table, how many documents have been written after step 5?

A 100

B 150

C 200

D 250

💡 Hint

Check the 'Docs Written to Destination' column at step 5.

At which step does the reindexing process complete successfully?

A Step 8

B Step 7

C Step 6

D Step 5

💡 Hint

Look for the row with status 'Success' in the execution_table.

If the source index had 500 documents instead of 250, how would the docs_read value change after step 4?

A 400

B 200

C 250

D 500

💡 Hint

Step 4 shows docs_read after reading two batches of 100 documents each.

Concept Snapshot

Reindexing copies documents from one Elasticsearch index to another.
Use a JSON body with 'source' and 'dest' indexes.
Elasticsearch reads documents in batches, optionally transforms them, then writes to the new index.
Monitor progress by tracking docs read and written.
Errors can pause or retry the process.
Useful for changing mappings or reorganizing data.

Full Transcript

Reindexing data in Elasticsearch means copying documents from one index to another. The process starts by sending a reindex request. Elasticsearch reads documents from the source index in batches. After reading each batch, it writes those documents to the destination index. This continues until all documents are copied. The execution table shows each step: starting, reading batches, writing batches, and completing successfully. Variables like docs_read and docs_written track progress. Beginners often wonder why docs_written lags behind docs_read; this is because reading happens before writing each batch. Transformations can be added between reading and writing but are not shown here. The process ends when all documents are copied without errors.

Practice

1.

What is the main purpose of _reindex API in Elasticsearch?

easy

A. To delete an index permanently

B. To copy data from one index to another

C. To update the Elasticsearch cluster settings

D. To create a new index with default settings

Solution

1. Step 1: Understand the function of _reindex

   The _reindex API is designed to copy documents from a source index to a destination index.

2. Step 2: Differentiate from other operations

   Deleting an index, updating cluster settings, or creating new indexes are different APIs and not the purpose of _reindex.

3. Final Answer:

   To copy data from one index to another -> Option B

4. Quick Check:

   _reindex copies data [OK]

Hint: Remember: reindex means copying data between indexes [OK]

Common Mistakes:

• Confusing reindex with deleting indexes
• Thinking reindex updates cluster settings
• Assuming reindex creates new indexes without data

2.

Which of the following is the correct minimal syntax to reindex data from old_index to new_index using Elasticsearch's REST API?

{
  "source": {"index": "old_index"},
  "dest": {"index": "new_index"}
}

easy

A. { "source": { "index": "old_index" }, "dest": { "index": "new_index" } }

B. { "source": { "index": "new_index" }, "dest": { "index": "old_index" } }

C. { "source": { "index": "old_index" }, "destination": { "index": "new_index" } }

D. { "src": { "index": "old_index" }, "dest": { "index": "new_index" } }

Solution

1. Step 1: Check the required keys for reindex

   The correct keys are source and dest with nested index names.

2. Step 2: Validate the index names and key spelling

   { "source": { "index": "old_index" }, "dest": { "index": "new_index" } } uses correct keys and index names. { "source": { "index": "new_index" }, "dest": { "index": "old_index" } } swaps source and destination incorrectly. { "source": { "index": "old_index" }, "destination": { "index": "new_index" } } uses wrong key destination. { "src": { "index": "old_index" }, "dest": { "index": "new_index" } } uses wrong key src.

3. Final Answer:

   { "source": { "index": "old_index" }, "dest": { "index": "new_index" } } -> Option A

4. Quick Check:

   Use source and dest keys exactly [OK]

Hint: Use exact keys: source and dest for reindex JSON [OK]

Common Mistakes:

• Swapping source and destination indexes
• Using wrong keys like 'src' or 'destination'
• Misspelling 'dest' or 'source'

3.

Given the following reindex request, what will be the result?

{
  "source": { "index": "products" },
  "dest": { "index": "products_new" },
  "script": {
    "source": "ctx._source.price = ctx._source.price * 1.1"
  }
}

Assuming products has documents with a price field.

medium

A. Only documents with price > 1.1 are copied

B. Documents copied without any price change

C. Reindex fails due to script syntax error

D. All documents copied with price increased by 10%

Solution

1. Step 1: Understand the script in reindex

   The script modifies each document's price by multiplying it by 1.1 during reindexing.

2. Step 2: Effect on documents

   All documents from products are copied to products_new with updated price values increased by 10%.

3. Final Answer:

   All documents copied with price increased by 10% -> Option D

4. Quick Check:

   Script modifies price during reindex [OK]

Hint: Scripts can modify fields during reindexing [OK]

Common Mistakes:

• Assuming script filters documents instead of modifying
• Thinking script causes syntax error
• Believing price remains unchanged

4.

Identify the error in this reindex request:

{
  "source": { "index": "old_index" },
  "dest": { "index": "new_index" },
  "query": {
    "match": { "status": "active" }
  }
}

medium

A. The query should be inside the "source" object

B. "query" is not allowed inside reindex requests

C. The "match" syntax is incorrect

D. The "dest" index name cannot be "new_index"

Solution

1. Step 1: Check placement of query in reindex

   The query must be inside the source object to filter documents.

2. Step 2: Validate other parts

   The match syntax is correct, and dest index name is valid.

3. Final Answer:

   The query should be inside the "source" object -> Option A

4. Quick Check:

   Put query inside source to filter during reindex [OK]

Hint: Put filters inside source, not at top level [OK]

Common Mistakes:

• Placing query outside source object
• Misunderstanding match query syntax
• Thinking destination index name is invalid

5.

You want to reindex data from logs_2023 to logs_archive but only include documents where level is "error" and add a new field archived set to true. Which reindex request achieves this?

hard

A. { "source": { "index": "logs_2023", "query": { "match": { "level": "error" } } }, "dest": { "index": "logs_archive" }, "script": { "source": "ctx._source.archived = false" } }

B. { "source": { "index": "logs_2023" }, "dest": { "index": "logs_archive" }, "query": { "term": { "level": "error" } }, "script": { "source": "ctx._source.archived = true" } }

C. { "source": { "index": "logs_2023", "query": { "term": { "level": "error" } } }, "dest": { "index": "logs_archive" }, "script": { "source": "ctx._source.archived = true" } }

D. { "source": { "index": "logs_2023", "query": { "term": { "level": "error" } } }, "dest": { "index": "logs_archive" }, "script": { "source": "ctx._source.archived = false" } }

Solution

1. Step 1: Filter documents by level "error" inside source

   The query with term filter must be inside the source object to select only error level logs.

2. Step 2: Add new field "archived" with value true via script

   The script sets ctx._source.archived = true to add the new field during reindex.

3. Step 3: Validate requests

   The request that places query inside source with term filter for level: "error" and script setting ctx._source.archived = true is correct. Placing query at top level is invalid. Using match instead of term or setting archived to false is incorrect.

4. Final Answer:

   { "source": { "index": "logs_2023", "query": { "term": { "level": "error" } } }, "dest": { "index": "logs_archive" }, "script": { "source": "ctx._source.archived = true" } } -> Option C

5. Quick Check:

   Filter in source + script sets field true [OK]

Hint: Filter inside source; script adds fields during reindex [OK]

Common Mistakes:

• Placing query outside source object
• Setting wrong value for new field
• Using match instead of term for exact filter