Why Enrich processor in Elasticsearch? - Purpose & Use Cases

The Big Idea

What if your data could automatically fill in missing details without you lifting a finger?

The Scenario

Imagine you have a big list of customer orders and another list with detailed customer information. You want to add the customer details to each order manually by looking up each customer one by one.

The Problem

Doing this by hand or with simple scripts is slow and mistakes happen easily. You might miss some customers or add wrong details because the lists are large and not connected automatically.

The Solution

The Enrich processor in Elasticsearch automatically adds extra data from one index to documents in another during indexing. It saves time and avoids errors by doing the lookups and merges for you.

Before vs After

✗ Before

for order in orders:
    customer = find_customer(order.customer_id)
    order.details = customer.details

✓ After

PUT _ingest/pipeline/enrich_pipeline
{
  "processors": [
    {
      "enrich": {
        "policy_name": "customer_policy",
        "field": "customer_id",
        "target_field": "customer_details"
      }
    }
  ]
}

What It Enables

You can enrich your data automatically and reliably during indexing, making searches smarter and faster without extra manual work.

Real Life Example

A company indexes sales orders and uses the Enrich processor to add customer loyalty status from another index, so they can quickly find orders from VIP customers.

Key Takeaways

Manual data merging is slow and error-prone.

Enrich processor automates adding related data during indexing.

This makes your search data richer and more useful instantly.

Practice

1. What is the main purpose of the enrich processor in Elasticsearch?

easy

A. To perform full-text search on documents

B. To delete documents based on a condition

C. To add extra data to documents by matching fields from an enrich policy

D. To create new indexes automatically

Solution

1. Step 1: Understand the enrich processor role

   The enrich processor is designed to add additional data to documents by matching fields from an external data source called an enrich policy.

2. Step 2: Compare options with enrich processor function

   Full-text search, document deletion, and automatic index creation describe other Elasticsearch features, not enrich processor functionality.

3. Final Answer:

   To add extra data to documents by matching fields from an enrich policy -> Option C

4. Quick Check:

   Enrich processor = add matched data [OK]

Hint: Enrich processor adds data by matching fields [OK]

Common Mistakes:

• Confusing enrich processor with delete or search operations
• Thinking enrich processor creates indexes
• Assuming it modifies original data sources

2. Which of the following is the correct syntax to define an enrich processor in an ingest pipeline?

easy

A. "enrich": {policy_name: "my_policy", field: "user_id", target_field: "user_info"}

B. "enrich": {"policy_name": "my_policy", "field": "user_id", "target_field": "user_info"}

C. "enrich": {"index": "my_index", "field": "user_id"}

D. "enrich": {"policy": "my_policy", "match_field": "user_id"}

Solution

1. Step 1: Recall enrich processor syntax

   The enrich processor requires the keys: policy_name, field, and target_field to specify the enrich policy, the field to match, and where to store enriched data.

2. Step 2: Identify correct option

   "enrich": {"policy_name": "my_policy", "field": "user_id", "target_field": "user_info"} correctly uses policy_name, field, and target_field with proper JSON quoting. Unquoted keys like policy_name cause syntax errors. Using index or match_field are incorrect keys.

3. Final Answer:

   "enrich": {"policy_name": "my_policy", "field": "user_id", "target_field": "user_info"} -> Option B

4. Quick Check:

   Correct keys = policy_name, field, target_field [OK]

Hint: Use policy_name, field, and target_field keys [OK]

Common Mistakes:

• Using wrong keys like 'index' or 'match_field'
• Omitting target_field key
• Incorrect JSON syntax or missing quotes

3. Given the following enrich processor configuration in an ingest pipeline:

{
  "enrich": {
    "policy_name": "user_policy",
    "field": "user_id",
    "target_field": "user_data"
  }
}

If a document {"user_id": "123"} matches the enrich policy with {"user_id": "123", "name": "Alice"}, what will be the resulting document after enrichment?

medium

A. {"user_id": "123", "user_data": {"user_id": "123", "name": "Alice"}}

B. {"user_id": "123", "name": "Alice"}

C. {"user_id": "123", "user_data": "Alice"}

D. {"user_id": "123"}

Solution

1. Step 1: Understand enrich processor output

   The enrich processor adds matched data under the target_field. Here, user_data will contain the matched document from the enrich policy.

2. Step 2: Apply to given document

   The original document has user_id: 123. The enrich policy matches and returns {"user_id": "123", "name": "Alice"}. This is added under user_data.

3. Final Answer:

   {"user_id": "123", "user_data": {"user_id": "123", "name": "Alice"}} -> Option A

4. Quick Check:

   Enrich adds matched doc under target_field [OK]

Hint: Matched data goes inside target_field key [OK]

Common Mistakes:

• Replacing original fields instead of adding under target_field
• Adding only a single field value instead of full matched doc
• Not nesting matched data inside target_field

4. You have this enrich processor configuration:

{
  "enrich": {
    "policy_name": "product_policy",
    "field": "product_id"
  }
}

When running the pipeline, you get an error. What is the likely cause?

medium

A. Missing target_field key in the enrich processor configuration

B. Incorrect policy_name value

C. The field value should be product_name

D. Enrich processor cannot be used in ingest pipelines

Solution

1. Step 1: Check required keys for enrich processor

   The enrich processor requires policy_name, field, and target_field. The configuration is missing target_field.

2. Step 2: Validate other options

   Incorrect policy_name value might cause errors but is not guaranteed. The field value should be product_name is incorrect because field should match the field to join on. Enrich processor cannot be used in ingest pipelines is false; enrich processor is valid in ingest pipelines.

3. Final Answer:

   Missing target_field key in the enrich processor configuration -> Option A

4. Quick Check:

   target_field is required in enrich processor [OK]

Hint: Always include target_field in enrich processor config [OK]

Common Mistakes:

• Forgetting target_field key
• Assuming enrich processor can't be in pipelines
• Changing field name incorrectly

5. You want to enrich incoming documents with user details but only if the user is active. Which approach correctly applies this condition using the enrich processor in an ingest pipeline?

hard

A. Use the enrich processor with a condition key to check user.active == true

B. Add a match condition inside the enrich processor to filter active users

C. Use the enrich processor with policy_name and then add a script processor to filter by user.active == true

D. Create an enrich policy that only indexes active users and use enrich processor normally

Solution

1. Step 1: Understand enrich processor filtering capabilities

   The enrich processor itself does not support filtering conditions inside its config. Filtering must be done before or by the enrich policy data.

2. Step 2: Choose correct approach for conditional enrichment

   Creating an enrich policy that only contains active users ensures only active user data is matched and added. Adding a match condition or condition key inside the enrich processor is unsupported syntax. Using a script processor after does not prevent enrichment for inactive users.

3. Final Answer:

   Create an enrich policy that only indexes active users and use enrich processor normally -> Option D

4. Quick Check:

   Filter data in enrich policy, not in processor [OK]

Hint: Filter data in enrich policy, not processor config [OK]

Common Mistakes:

• Trying to filter inside enrich processor config
• Using unsupported condition keys
• Filtering after enrichment instead of before