Ingest pipelines in Elasticsearch - Step-by-Step Execution

Concept Flow - Ingest pipelines

Document arrives

↓

Ingest Pipeline starts

↓

Processor 1: Modify or enrich document

↓

Processor 2: Modify or enrich document

↓

... more processors ...

↓

Pipeline ends

↓

Document indexed or error handled

The document flows through a series of processors in the ingest pipeline, each modifying or enriching it before final indexing.

Execution Sample

Elasticsearch

PUT _ingest/pipeline/my_pipeline
{
  "processors": [
    {"set": {"field": "field1", "value": "value1"}},
    {"rename": {"field": "field1", "target_field": "field2"}}
  ]
}

Defines an ingest pipeline that sets a field and then renames it.

Execution Table

Step	Processor	Document State Before	Processor Action	Document State After
1	set	{}	Set field1 to 'value1'	{"field1": "value1"}
2	rename	{"field1": "value1"}	Rename field1 to field2	{"field2": "value1"}
3	end	{"field2": "value1"}	Pipeline ends, document ready for indexing	{"field2": "value1"}

💡 All processors applied; document is ready for indexing.

Variable Tracker

Variable	Start	After Step 1	After Step 2	Final
document	{}	{"field1": "value1"}	{"field2": "value1"}	{"field2": "value1"}

Key Moments - 3 Insights

Why does the document state change after each processor?

Each processor modifies the document by adding, changing, or removing fields as shown in the execution_table rows 1 and 2.

What happens if a processor fails?

If a processor fails, the pipeline stops and the document is not indexed; this is why the exit_note confirms all processors succeeded.

Can processors be reordered?

Yes, the order matters because each processor works on the document state left by the previous one, as seen in the step-by-step changes.

Visual Quiz - 3 Questions

Look at the execution_table, what is the document state after Step 1?

A {}

B {"field1": "value1"}

C {"field2": "value1"}

D {"field1": "value2"}

💡 Hint

Check the 'Document State After' column for Step 1 in the execution_table.

At which step does the field get renamed?

A Step 1

B Step 3

C Step 2

D No renaming happens

💡 Hint

Look at the 'Processor' and 'Processor Action' columns in the execution_table.

If the 'set' processor was removed, what would be the document state after Step 2?

A Error because field1 does not exist

B {}

C {"field2": "value1"}

D {"field1": "value1"}

💡 Hint

Consider that the rename processor needs 'field1' to exist before renaming, as shown in the variable_tracker.

Concept Snapshot

Ingest pipelines process documents step-by-step before indexing.
Each processor modifies or enriches the document.
Processors run in order; output of one is input to next.
If any processor fails, indexing stops.
Use pipelines to clean or transform data automatically.

Full Transcript

An ingest pipeline in Elasticsearch takes a document and passes it through a series of processors. Each processor changes the document by adding, renaming, or modifying fields. For example, a 'set' processor adds a field, and a 'rename' processor changes a field's name. The document state updates after each processor. If all processors succeed, the document is indexed. If any processor fails, the pipeline stops and the document is not indexed. The order of processors matters because each depends on the previous state. This process helps prepare data automatically before storing it.

Practice

1. What is the main purpose of an ingest pipeline in Elasticsearch?

easy

A. To create visualizations from data

B. To query documents faster

C. To backup Elasticsearch data

D. To modify documents before they are indexed

Solution

1. Step 1: Understand ingest pipeline role

   An ingest pipeline processes documents before they are stored in Elasticsearch.

2. Step 2: Compare options to pipeline purpose

   Only To modify documents before they are indexed matches this role; others describe unrelated tasks.

3. Final Answer:

   To modify documents before they are indexed -> Option D

4. Quick Check:

   Ingest pipeline = modify before indexing [OK]

Hint: Ingest pipelines change data before saving [OK]

Common Mistakes:

• Confusing pipelines with query optimization
• Thinking pipelines create backups
• Assuming pipelines generate reports

2. Which of the following is the correct way to define a processor that adds a field "source": "web" in an ingest pipeline?

easy

A. "processors": [{"add": {"field": "source", "value": "web"}}]

B. "processors": [{"remove": {"field": "source", "value": "web"}}]

C. "processors": [{"set": {"field": "source", "value": "web"}}]

D. "processors": [{"append": {"field": "source", "value": "web"}}]

Solution

1. Step 1: Identify processor to add or set field

   The set processor assigns or adds a field with a value.

2. Step 2: Check syntax correctness

   "processors": [{"set": {"field": "source", "value": "web"}}] uses set with correct JSON structure; add is not a valid processor name.

3. Final Answer:

   "processors": [{"set": {"field": "source", "value": "web"}}] -> Option C

4. Quick Check:

   Use set to add fields [OK]

Hint: Use 'set' processor to add or update fields [OK]

Common Mistakes:

• Using 'add' instead of 'set' processor
• Confusing 'append' with adding new fields
• Trying to remove field when adding

3. Given this ingest pipeline snippet:

{"processors": [{"rename": {"field": "old_name", "target_field": "new_name"}}]}

What happens to a document {"old_name": "value"} when processed?

medium

A. {"old_name": "value", "new_name": "value"}

B. {"new_name": "value"}

C. {"old_name": null}

D. {"old_name": "value"}

Solution

1. Step 1: Understand rename processor effect

   The rename processor moves the value from old_name to new_name, removing the original field.

2. Step 2: Apply to example document

   Input {"old_name": "value"} becomes {"new_name": "value"} after processing.

3. Final Answer:

   {"new_name": "value"} -> Option B

4. Quick Check:

   Rename moves field, removes old [OK]

Hint: Rename moves field value, deletes old field [OK]

Common Mistakes:

• Thinking rename duplicates field
• Assuming old field remains
• Confusing rename with copy

4. You wrote this pipeline processor:

{"processors": [{"convert": {"field": "age", "type": "integer"}}]}

But when simulating, you get an error. What is the likely cause?

medium

A. The field "age" does not exist in the document

B. The "convert" processor does not exist

C. The "type" value "integer" is invalid

D. The pipeline must have a "description" field

Solution

1. Step 1: Check processor and type validity

   The "convert" processor and type "integer" are valid in Elasticsearch ingest pipelines.

2. Step 2: Identify common error cause

   If the field "age" is missing in the document, conversion fails causing error during simulation.

3. Final Answer:

   The field "age" does not exist in the document -> Option A

4. Quick Check:

   Missing field causes convert error [OK]

Hint: Convert errors if field missing in document [OK]

Common Mistakes:

• Assuming convert processor is invalid
• Thinking 'integer' is wrong type
• Believing description is mandatory

5. You want to create an ingest pipeline that adds a field "status": "new" only if the field "priority" exists. Which processor setup achieves this?

hard

A. "processors": [{"set": {"field": "status", "value": "new", "if": "ctx.priority != null"}}]

B. "processors": [{"set": {"field": "status", "value": "new"}}, {"exists": {"field": "priority"}}]

C. "processors": [{"conditional": {"field": "priority", "processor": {"set": {"field": "status", "value": "new"}}}}]

D. "processors": [{"set": {"field": "status", "value": "new"}, "condition": "priority"}]

Solution

1. Step 1: Understand conditional execution in pipelines

   Processors support an "if" condition using painless scripts to run only when true.

2. Step 2: Identify correct syntax for conditional set

   "processors": [{"set": {"field": "status", "value": "new", "if": "ctx.priority != null"}}] uses "if": "ctx.priority != null" which runs set only if "priority" exists.

3. Final Answer:

   "processors": [{"set": {"field": "status", "value": "new", "if": "ctx.priority != null"}}] -> Option A

4. Quick Check:

   Use "if" with painless script for conditions [OK]

Hint: Use 'if' with painless script to conditionally run processor [OK]

Common Mistakes:

• Using non-existent 'conditional' processor
• Placing condition outside processor block
• Misusing 'exists' as processor