Enrich processor in Elasticsearch

Introduction

The enrich processor helps add extra information to your documents automatically. It looks up data from another source and adds it to your data.

You want to add user details to logs based on user ID.

You need to add product info to sales records without changing the original data source.

You want to enrich error logs with server names from a separate list.

You want to add location details to IP addresses in your documents.

Syntax

Elasticsearch

{
  "enrich": {
    "policy_name": "policy_name_here",
    "field": "field_to_match",
    "target_field": "field_to_add_data",
    "max_matches": 1
  }
}

policy_name is the name of the enrich policy you created.

field is the field in your document to match with the enrich index.

Examples

This adds user details to documents by matching the user_id field.

Elasticsearch

{
  "enrich": {
    "policy_name": "user_policy",
    "field": "user_id",
    "target_field": "user_details"
  }
}

This adds product info to documents by matching product_code, limiting to one match.

Elasticsearch

{
  "enrich": {
    "policy_name": "product_policy",
    "field": "product_code",
    "target_field": "product_info",
    "max_matches": 1
  }
}

Sample Program

This example creates an enrich policy to add user name and email from the users index. Then it simulates adding this info to a document with user_id 123.

Elasticsearch

PUT /users/_doc/1
{
  "user_id": "123",
  "name": "John Doe",
  "email": "[e-mail]"
}

PUT /_enrich/policy/user_policy
{
  "match": {
    "indices": ["users"],
    "match_field": "user_id",
    "enrich_fields": ["name", "email"]
  }
}

POST /_enrich/policy/user_policy/_execute

POST /_ingest/pipeline/_simulate
{
  "pipeline": {
    "processors": [
      {
        "enrich": {
          "policy_name": "user_policy",
          "field": "user_id",
          "target_field": "user_info"
        }
      }
    ]
  },
  "docs": [
    {
      "_source": {
        "user_id": "123"
      }
    }
  ]
}

Important Notes

You must create and execute the enrich policy before using the enrich processor.

The enrich processor works during ingest, so it adds data as documents are indexed.

Use max_matches to control how many matches to add; default is 1.

Summary

The enrich processor adds extra data to documents by matching fields.

It uses enrich policies created from other data sources.

This helps keep your data enriched without changing original sources.

Practice

1. What is the main purpose of the enrich processor in Elasticsearch?

easy

A. To perform full-text search on documents

B. To delete documents based on a condition

C. To add extra data to documents by matching fields from an enrich policy

D. To create new indexes automatically

Solution

1. Step 1: Understand the enrich processor role

   The enrich processor is designed to add additional data to documents by matching fields from an external data source called an enrich policy.

2. Step 2: Compare options with enrich processor function

   Full-text search, document deletion, and automatic index creation describe other Elasticsearch features, not enrich processor functionality.

3. Final Answer:

   To add extra data to documents by matching fields from an enrich policy -> Option C

4. Quick Check:

   Enrich processor = add matched data [OK]

Hint: Enrich processor adds data by matching fields [OK]

Common Mistakes:

• Confusing enrich processor with delete or search operations
• Thinking enrich processor creates indexes
• Assuming it modifies original data sources

2. Which of the following is the correct syntax to define an enrich processor in an ingest pipeline?

easy

A. "enrich": {policy_name: "my_policy", field: "user_id", target_field: "user_info"}

B. "enrich": {"policy_name": "my_policy", "field": "user_id", "target_field": "user_info"}

C. "enrich": {"index": "my_index", "field": "user_id"}

D. "enrich": {"policy": "my_policy", "match_field": "user_id"}

Solution

1. Step 1: Recall enrich processor syntax

   The enrich processor requires the keys: policy_name, field, and target_field to specify the enrich policy, the field to match, and where to store enriched data.

2. Step 2: Identify correct option

   "enrich": {"policy_name": "my_policy", "field": "user_id", "target_field": "user_info"} correctly uses policy_name, field, and target_field with proper JSON quoting. Unquoted keys like policy_name cause syntax errors. Using index or match_field are incorrect keys.

3. Final Answer:

   "enrich": {"policy_name": "my_policy", "field": "user_id", "target_field": "user_info"} -> Option B

4. Quick Check:

   Correct keys = policy_name, field, target_field [OK]

Hint: Use policy_name, field, and target_field keys [OK]

Common Mistakes:

• Using wrong keys like 'index' or 'match_field'
• Omitting target_field key
• Incorrect JSON syntax or missing quotes

3. Given the following enrich processor configuration in an ingest pipeline:

{
  "enrich": {
    "policy_name": "user_policy",
    "field": "user_id",
    "target_field": "user_data"
  }
}

If a document {"user_id": "123"} matches the enrich policy with {"user_id": "123", "name": "Alice"}, what will be the resulting document after enrichment?

medium

A. {"user_id": "123", "user_data": {"user_id": "123", "name": "Alice"}}

B. {"user_id": "123", "name": "Alice"}

C. {"user_id": "123", "user_data": "Alice"}

D. {"user_id": "123"}

Solution

1. Step 1: Understand enrich processor output

   The enrich processor adds matched data under the target_field. Here, user_data will contain the matched document from the enrich policy.

2. Step 2: Apply to given document

   The original document has user_id: 123. The enrich policy matches and returns {"user_id": "123", "name": "Alice"}. This is added under user_data.

3. Final Answer:

   {"user_id": "123", "user_data": {"user_id": "123", "name": "Alice"}} -> Option A

4. Quick Check:

   Enrich adds matched doc under target_field [OK]

Hint: Matched data goes inside target_field key [OK]

Common Mistakes:

• Replacing original fields instead of adding under target_field
• Adding only a single field value instead of full matched doc
• Not nesting matched data inside target_field

4. You have this enrich processor configuration:

{
  "enrich": {
    "policy_name": "product_policy",
    "field": "product_id"
  }
}

When running the pipeline, you get an error. What is the likely cause?

medium

A. Missing target_field key in the enrich processor configuration

B. Incorrect policy_name value

C. The field value should be product_name

D. Enrich processor cannot be used in ingest pipelines

Solution

1. Step 1: Check required keys for enrich processor

   The enrich processor requires policy_name, field, and target_field. The configuration is missing target_field.

2. Step 2: Validate other options

   Incorrect policy_name value might cause errors but is not guaranteed. The field value should be product_name is incorrect because field should match the field to join on. Enrich processor cannot be used in ingest pipelines is false; enrich processor is valid in ingest pipelines.

3. Final Answer:

   Missing target_field key in the enrich processor configuration -> Option A

4. Quick Check:

   target_field is required in enrich processor [OK]

Hint: Always include target_field in enrich processor config [OK]

Common Mistakes:

• Forgetting target_field key
• Assuming enrich processor can't be in pipelines
• Changing field name incorrectly

5. You want to enrich incoming documents with user details but only if the user is active. Which approach correctly applies this condition using the enrich processor in an ingest pipeline?

hard

A. Use the enrich processor with a condition key to check user.active == true

B. Add a match condition inside the enrich processor to filter active users

C. Use the enrich processor with policy_name and then add a script processor to filter by user.active == true

D. Create an enrich policy that only indexes active users and use enrich processor normally

Solution

1. Step 1: Understand enrich processor filtering capabilities

   The enrich processor itself does not support filtering conditions inside its config. Filtering must be done before or by the enrich policy data.

2. Step 2: Choose correct approach for conditional enrichment

   Creating an enrich policy that only contains active users ensures only active user data is matched and added. Adding a match condition or condition key inside the enrich processor is unsupported syntax. Using a script processor after does not prevent enrichment for inactive users.

3. Final Answer:

   Create an enrich policy that only indexes active users and use enrich processor normally -> Option D

4. Quick Check:

   Filter data in enrich policy, not in processor [OK]

Hint: Filter data in enrich policy, not processor config [OK]

Common Mistakes:

• Trying to filter inside enrich processor config
• Using unsupported condition keys
• Filtering after enrichment instead of before