Enrich processor in Elasticsearch - Step-by-Step Execution

Concept Flow - Enrich processor

Start ingest pipeline

↓

Receive document

↓

Extract key field

↓

Lookup enrich index with key

↓

Match found?

No→Skip enrich

Yes↓

Merge enrich data into document

↓

Pass enriched document to next processor

↓

End ingest pipeline

The enrich processor takes a field from the incoming document, looks it up in an enrich index, and merges matching data into the document before continuing.

Execution Sample

Elasticsearch

PUT _ingest/pipeline/enrich-pipeline
{
  "processors": [
    {
      "enrich": {
        "policy_name": "user-policy",
        "field": "user_id",
        "target_field": "user_info"
      }
    }
  ]
}

Defines an ingest pipeline that enriches documents by looking up user info based on user_id and adds it under user_info field.

Execution Table

Step	Action	Input Document	Lookup Key	Enrich Index Result	Output Document
1	Receive document	{"user_id": "123", "event": "login"}	123	-	{"user_id": "123", "event": "login"}
2	Lookup enrich index	-	123	{"name": "Alice", "role": "admin"}	-
3	Merge enrich data	-	-	-	{"user_id": "123", "event": "login", "user_info": {"name": "Alice", "role": "admin"}}
4	Pass to next processor	-	-	-	{"user_id": "123", "event": "login", "user_info": {"name": "Alice", "role": "admin"}}
5	End pipeline	-	-	-	Pipeline finished with enriched document

💡 Pipeline ends after enriching document or skipping if no match found.

Variable Tracker

Variable	Start	After Step 1	After Step 2	After Step 3	Final
document	{"user_id": "123", "event": "login"}	{"user_id": "123", "event": "login"}	{"user_id": "123", "event": "login"}	{"user_id": "123", "event": "login", "user_info": {"name": "Alice", "role": "admin"}}	{"user_id": "123", "event": "login", "user_info": {"name": "Alice", "role": "admin"}}
lookup_key	-	123	123	123	123
enrich_result	-	-	{"name": "Alice", "role": "admin"}	{"name": "Alice", "role": "admin"}	{"name": "Alice", "role": "admin"}

Key Moments - 3 Insights

What happens if the enrich index has no matching entry for the lookup key?

If no match is found (see execution_table step 2), the enrich processor skips merging and passes the original document unchanged to the next processor.

How does the enrich processor know which field to use for lookup?

The processor uses the 'field' parameter from the pipeline definition (shown in execution_sample) to extract the lookup key from the incoming document.

Where does the enriched data get added in the document?

The enriched data is merged under the 'target_field' specified in the pipeline (here 'user_info'), as shown in execution_table step 3.

Visual Quiz - 3 Questions

Look at the execution_table, what is the value of 'lookup_key' after Step 1?

A 123

B -

C Alice

D {"name": "Alice", "role": "admin"}

💡 Hint

Check variable_tracker column 'After Step 1' for 'lookup_key'

At which step does the document get enriched with user info?

A Step 1

B Step 3

C Step 2

D Step 4

💡 Hint

Look at execution_table 'Action' and 'Output Document' columns

If the enrich index had no matching entry, what would happen to the output document?

A The pipeline would fail

B It would be enriched with empty data

C It would remain unchanged

D The document would be deleted

💡 Hint

Refer to key_moments explanation about no match scenario

Concept Snapshot

Enrich processor syntax:
{
  "enrich": {
    "policy_name": "policy",
    "field": "lookup_field",
    "target_field": "field_to_add"
  }
}

Behavior: Looks up a value from an enrich index using 'field' from the document, merges found data under 'target_field'.
If no match, document passes unchanged.

Full Transcript

The enrich processor in Elasticsearch ingest pipelines takes a field from the incoming document, uses it as a key to look up data in an enrich index, and merges the found data into the document under a target field. The process starts by receiving the document, extracting the lookup key, querying the enrich index, and if a match is found, merging the data. If no match is found, the document continues unchanged. This allows adding extra information to documents during ingestion based on external data.

Practice

1. What is the main purpose of the enrich processor in Elasticsearch?

easy

A. To perform full-text search on documents

B. To delete documents based on a condition

C. To add extra data to documents by matching fields from an enrich policy

D. To create new indexes automatically

Solution

1. Step 1: Understand the enrich processor role

   The enrich processor is designed to add additional data to documents by matching fields from an external data source called an enrich policy.

2. Step 2: Compare options with enrich processor function

   Full-text search, document deletion, and automatic index creation describe other Elasticsearch features, not enrich processor functionality.

3. Final Answer:

   To add extra data to documents by matching fields from an enrich policy -> Option C

4. Quick Check:

   Enrich processor = add matched data [OK]

Hint: Enrich processor adds data by matching fields [OK]

Common Mistakes:

• Confusing enrich processor with delete or search operations
• Thinking enrich processor creates indexes
• Assuming it modifies original data sources

2. Which of the following is the correct syntax to define an enrich processor in an ingest pipeline?

easy

A. "enrich": {policy_name: "my_policy", field: "user_id", target_field: "user_info"}

B. "enrich": {"policy_name": "my_policy", "field": "user_id", "target_field": "user_info"}

C. "enrich": {"index": "my_index", "field": "user_id"}

D. "enrich": {"policy": "my_policy", "match_field": "user_id"}

Solution

1. Step 1: Recall enrich processor syntax

   The enrich processor requires the keys: policy_name, field, and target_field to specify the enrich policy, the field to match, and where to store enriched data.

2. Step 2: Identify correct option

   "enrich": {"policy_name": "my_policy", "field": "user_id", "target_field": "user_info"} correctly uses policy_name, field, and target_field with proper JSON quoting. Unquoted keys like policy_name cause syntax errors. Using index or match_field are incorrect keys.

3. Final Answer:

   "enrich": {"policy_name": "my_policy", "field": "user_id", "target_field": "user_info"} -> Option B

4. Quick Check:

   Correct keys = policy_name, field, target_field [OK]

Hint: Use policy_name, field, and target_field keys [OK]

Common Mistakes:

• Using wrong keys like 'index' or 'match_field'
• Omitting target_field key
• Incorrect JSON syntax or missing quotes

3. Given the following enrich processor configuration in an ingest pipeline:

{
  "enrich": {
    "policy_name": "user_policy",
    "field": "user_id",
    "target_field": "user_data"
  }
}

If a document {"user_id": "123"} matches the enrich policy with {"user_id": "123", "name": "Alice"}, what will be the resulting document after enrichment?

medium

A. {"user_id": "123", "user_data": {"user_id": "123", "name": "Alice"}}

B. {"user_id": "123", "name": "Alice"}

C. {"user_id": "123", "user_data": "Alice"}

D. {"user_id": "123"}

Solution

1. Step 1: Understand enrich processor output

   The enrich processor adds matched data under the target_field. Here, user_data will contain the matched document from the enrich policy.

2. Step 2: Apply to given document

   The original document has user_id: 123. The enrich policy matches and returns {"user_id": "123", "name": "Alice"}. This is added under user_data.

3. Final Answer:

   {"user_id": "123", "user_data": {"user_id": "123", "name": "Alice"}} -> Option A

4. Quick Check:

   Enrich adds matched doc under target_field [OK]

Hint: Matched data goes inside target_field key [OK]

Common Mistakes:

• Replacing original fields instead of adding under target_field
• Adding only a single field value instead of full matched doc
• Not nesting matched data inside target_field

4. You have this enrich processor configuration:

{
  "enrich": {
    "policy_name": "product_policy",
    "field": "product_id"
  }
}

When running the pipeline, you get an error. What is the likely cause?

medium

A. Missing target_field key in the enrich processor configuration

B. Incorrect policy_name value

C. The field value should be product_name

D. Enrich processor cannot be used in ingest pipelines

Solution

1. Step 1: Check required keys for enrich processor

   The enrich processor requires policy_name, field, and target_field. The configuration is missing target_field.

2. Step 2: Validate other options

   Incorrect policy_name value might cause errors but is not guaranteed. The field value should be product_name is incorrect because field should match the field to join on. Enrich processor cannot be used in ingest pipelines is false; enrich processor is valid in ingest pipelines.

3. Final Answer:

   Missing target_field key in the enrich processor configuration -> Option A

4. Quick Check:

   target_field is required in enrich processor [OK]

Hint: Always include target_field in enrich processor config [OK]

Common Mistakes:

• Forgetting target_field key
• Assuming enrich processor can't be in pipelines
• Changing field name incorrectly

5. You want to enrich incoming documents with user details but only if the user is active. Which approach correctly applies this condition using the enrich processor in an ingest pipeline?

hard

A. Use the enrich processor with a condition key to check user.active == true

B. Add a match condition inside the enrich processor to filter active users

C. Use the enrich processor with policy_name and then add a script processor to filter by user.active == true

D. Create an enrich policy that only indexes active users and use enrich processor normally

Solution

1. Step 1: Understand enrich processor filtering capabilities

   The enrich processor itself does not support filtering conditions inside its config. Filtering must be done before or by the enrich policy data.

2. Step 2: Choose correct approach for conditional enrichment

   Creating an enrich policy that only contains active users ensures only active user data is matched and added. Adding a match condition or condition key inside the enrich processor is unsupported syntax. Using a script processor after does not prevent enrichment for inactive users.

3. Final Answer:

   Create an enrich policy that only indexes active users and use enrich processor normally -> Option D

4. Quick Check:

   Filter data in enrich policy, not in processor [OK]

Hint: Filter data in enrich policy, not processor config [OK]

Common Mistakes:

• Trying to filter inside enrich processor config
• Using unsupported condition keys
• Filtering after enrichment instead of before