Enrich processor in Elasticsearch - Mini Project: Build & Apply

Enrich Processor in Elasticsearch

📖 Scenario: You work in a company that stores customer orders in Elasticsearch. You want to add extra customer details to each order automatically using Elasticsearch's enrich processor.

🎯 Goal: Build an Elasticsearch ingest pipeline that enriches order documents with customer information from a separate customer index.

📋 What You'll Learn

Create an enrich policy named customer-policy that matches on customer_id from the customers index.

Create an ingest pipeline named order-enrich-pipeline that uses the enrich processor with the customer-policy.

Test the pipeline by simulating an ingest of an order document with a customer_id.

Print the enriched order document showing added customer details.

💡 Why This Matters

🌍 Real World

Enrich processor helps automatically add related data to documents during ingest, saving time and avoiding manual joins.

💼 Career

Understanding enrich processors is useful for Elasticsearch engineers and developers working on data pipelines and search applications.

1

Create an enrich policy

Write a JSON request to create an enrich policy called customer-policy that uses the match type on the customer_id field from the customers index.

Elasticsearch

{
  "policy": {
    "match": {
      "indices": ["customers"],
      "match_field": "customer_id",
      "enrich_fields": ["name", "email"]
    }
  }
}

# Your code here

Hint

Use the PUT _enrich/policy/customer-policy API with a JSON body defining the match type, indices, match_field, and enrich_fields.

2

Create an ingest pipeline with enrich processor

Write a JSON request to create an ingest pipeline named order-enrich-pipeline that uses the enrich processor with the policy customer-policy and targets the field customer_id. The enriched data should be added under the field customer_info.

Elasticsearch

{
  "policy": {
    "match": {
      "indices": ["customers"],
      "match_field": "customer_id",
      "enrich_fields": ["name", "email"]
    }
  }
}

# Your code here to create the ingest pipeline

Hint

Use the PUT _ingest/pipeline/order-enrich-pipeline API with a processors array containing an enrich processor configured with the policy name, field, and target_field.

3

Simulate ingesting an order document

Write a JSON request to simulate ingesting an order document with customer_id set to 123 through the order-enrich-pipeline. Include an order_id field with value abc-001.

Elasticsearch

{
  "policy": {
    "match": {
      "indices": ["customers"],
      "match_field": "customer_id",
      "enrich_fields": ["name", "email"]
    }
  }
}

PUT _ingest/pipeline/order-enrich-pipeline
{
  "processors": [
    {
      "enrich": {
        "policy_name": "customer-policy",
        "field": "customer_id",
        "target_field": "customer_info",
        "ignore_missing": true
      }
    }
  ]
}

POST _ingest/pipeline/order-enrich-pipeline/_simulate
{
  "docs": [
    {
      "_source": {
        "order_id": "abc-001",
        "customer_id": 123
      }
    }
  ]
}

# Your code here

Hint

Use the POST _ingest/pipeline/order-enrich-pipeline/_simulate API with a docs array containing the order document.

4

Print the enriched order document

Write a command to print the enriched order document returned by the simulate API, showing the order_id, customer_id, and the added customer_info fields.

Elasticsearch

{
  "policy": {
    "match": {
      "indices": ["customers"],
      "match_field": "customer_id",
      "enrich_fields": ["name", "email"]
    }
  }
}

PUT _ingest/pipeline/order-enrich-pipeline
{
  "processors": [
    {
      "enrich": {
        "policy_name": "customer-policy",
        "field": "customer_id",
        "target_field": "customer_info",
        "ignore_missing": true
      }
    }
  ]
}

POST _ingest/pipeline/order-enrich-pipeline/_simulate
{
  "docs": [
    {
      "_source": {
        "order_id": "abc-001",
        "customer_id": 123
      }
    }
  ]
}

# Your code here to print the enriched document

Hint

Look at the response from the simulate API. It contains the enriched document under docs[0]._source. Print or display this JSON to see the added customer_info.

Practice

1. What is the main purpose of the enrich processor in Elasticsearch?

easy

A. To perform full-text search on documents

B. To delete documents based on a condition

C. To add extra data to documents by matching fields from an enrich policy

D. To create new indexes automatically

Solution

1. Step 1: Understand the enrich processor role

   The enrich processor is designed to add additional data to documents by matching fields from an external data source called an enrich policy.

2. Step 2: Compare options with enrich processor function

   Full-text search, document deletion, and automatic index creation describe other Elasticsearch features, not enrich processor functionality.

3. Final Answer:

   To add extra data to documents by matching fields from an enrich policy -> Option C

4. Quick Check:

   Enrich processor = add matched data [OK]

Hint: Enrich processor adds data by matching fields [OK]

Common Mistakes:

• Confusing enrich processor with delete or search operations
• Thinking enrich processor creates indexes
• Assuming it modifies original data sources

2. Which of the following is the correct syntax to define an enrich processor in an ingest pipeline?

easy

A. "enrich": {policy_name: "my_policy", field: "user_id", target_field: "user_info"}

B. "enrich": {"policy_name": "my_policy", "field": "user_id", "target_field": "user_info"}

C. "enrich": {"index": "my_index", "field": "user_id"}

D. "enrich": {"policy": "my_policy", "match_field": "user_id"}

Solution

1. Step 1: Recall enrich processor syntax

   The enrich processor requires the keys: policy_name, field, and target_field to specify the enrich policy, the field to match, and where to store enriched data.

2. Step 2: Identify correct option

   "enrich": {"policy_name": "my_policy", "field": "user_id", "target_field": "user_info"} correctly uses policy_name, field, and target_field with proper JSON quoting. Unquoted keys like policy_name cause syntax errors. Using index or match_field are incorrect keys.

3. Final Answer:

   "enrich": {"policy_name": "my_policy", "field": "user_id", "target_field": "user_info"} -> Option B

4. Quick Check:

   Correct keys = policy_name, field, target_field [OK]

Hint: Use policy_name, field, and target_field keys [OK]

Common Mistakes:

• Using wrong keys like 'index' or 'match_field'
• Omitting target_field key
• Incorrect JSON syntax or missing quotes

3. Given the following enrich processor configuration in an ingest pipeline:

{
  "enrich": {
    "policy_name": "user_policy",
    "field": "user_id",
    "target_field": "user_data"
  }
}

If a document {"user_id": "123"} matches the enrich policy with {"user_id": "123", "name": "Alice"}, what will be the resulting document after enrichment?

medium

A. {"user_id": "123", "user_data": {"user_id": "123", "name": "Alice"}}

B. {"user_id": "123", "name": "Alice"}

C. {"user_id": "123", "user_data": "Alice"}

D. {"user_id": "123"}

Solution

1. Step 1: Understand enrich processor output

   The enrich processor adds matched data under the target_field. Here, user_data will contain the matched document from the enrich policy.

2. Step 2: Apply to given document

   The original document has user_id: 123. The enrich policy matches and returns {"user_id": "123", "name": "Alice"}. This is added under user_data.

3. Final Answer:

   {"user_id": "123", "user_data": {"user_id": "123", "name": "Alice"}} -> Option A

4. Quick Check:

   Enrich adds matched doc under target_field [OK]

Hint: Matched data goes inside target_field key [OK]

Common Mistakes:

• Replacing original fields instead of adding under target_field
• Adding only a single field value instead of full matched doc
• Not nesting matched data inside target_field

4. You have this enrich processor configuration:

{
  "enrich": {
    "policy_name": "product_policy",
    "field": "product_id"
  }
}

When running the pipeline, you get an error. What is the likely cause?

medium

A. Missing target_field key in the enrich processor configuration

B. Incorrect policy_name value

C. The field value should be product_name

D. Enrich processor cannot be used in ingest pipelines

Solution

1. Step 1: Check required keys for enrich processor

   The enrich processor requires policy_name, field, and target_field. The configuration is missing target_field.

2. Step 2: Validate other options

   Incorrect policy_name value might cause errors but is not guaranteed. The field value should be product_name is incorrect because field should match the field to join on. Enrich processor cannot be used in ingest pipelines is false; enrich processor is valid in ingest pipelines.

3. Final Answer:

   Missing target_field key in the enrich processor configuration -> Option A

4. Quick Check:

   target_field is required in enrich processor [OK]

Hint: Always include target_field in enrich processor config [OK]

Common Mistakes:

• Forgetting target_field key
• Assuming enrich processor can't be in pipelines
• Changing field name incorrectly

5. You want to enrich incoming documents with user details but only if the user is active. Which approach correctly applies this condition using the enrich processor in an ingest pipeline?

hard

A. Use the enrich processor with a condition key to check user.active == true

B. Add a match condition inside the enrich processor to filter active users

C. Use the enrich processor with policy_name and then add a script processor to filter by user.active == true

D. Create an enrich policy that only indexes active users and use enrich processor normally

Solution

1. Step 1: Understand enrich processor filtering capabilities

   The enrich processor itself does not support filtering conditions inside its config. Filtering must be done before or by the enrich policy data.

2. Step 2: Choose correct approach for conditional enrichment

   Creating an enrich policy that only contains active users ensures only active user data is matched and added. Adding a match condition or condition key inside the enrich processor is unsupported syntax. Using a script processor after does not prevent enrichment for inactive users.

3. Final Answer:

   Create an enrich policy that only indexes active users and use enrich processor normally -> Option D

4. Quick Check:

   Filter data in enrich policy, not in processor [OK]

Hint: Filter data in enrich policy, not processor config [OK]

Common Mistakes:

• Trying to filter inside enrich processor config
• Using unsupported condition keys
• Filtering after enrichment instead of before