Enrich processor in Elasticsearch - Deep Dive

Overview - Enrich processor

What is it?

The Enrich processor in Elasticsearch is a tool used to add extra information to documents as they are ingested. It looks up data from an external source, called an enrich policy, and merges matching data into the document. This helps to enhance or complete the document with related details without storing all data in one place.

Why it matters

Without the Enrich processor, you would need to store all related data inside each document or perform costly joins at query time, which slows down searches. The Enrich processor solves this by enriching documents during ingestion, making searches faster and more efficient. This improves performance and reduces storage duplication.

Where it fits

Before learning about the Enrich processor, you should understand Elasticsearch basics like indexing and ingest pipelines. After mastering it, you can explore advanced data enrichment techniques, such as using scripted processors or integrating with external databases for enrichment.

Mental Model

Core Idea

The Enrich processor adds extra data to documents by looking up matching information from a separate data source during ingestion.

Think of it like...

Imagine mailing a letter and adding a sticker with extra info about the recipient from a separate address book before sending it out.

┌───────────────┐      ┌───────────────┐
│ Incoming Doc  │─────▶│ Enrich Policy │
│ (partial data)│      │ (lookup data) │
└───────────────┘      └───────────────┘
         │                    ▲
         │                    │
         ▼                    │
┌────────────────────────────┐
│ Enriched Document (merged)  │
└────────────────────────────┘

Build-Up - 7 Steps

1

Foundation

What is an Enrich Policy

Concept: An enrich policy defines the data source and matching rules used to add information to documents.

An enrich policy is created by specifying a source index and a matching field. This policy builds a special index that stores the data to be used for enrichment. For example, a policy might use a customer database to add customer details to logs.

Result

You get a ready-to-use enrich index that the Enrich processor can query during ingestion.

Understanding enrich policies is key because they hold the data that will be merged into your documents, separating enrichment data from your main data.

2

Foundation

How Enrich Processor Works in Pipelines

Concept: The Enrich processor is part of an ingest pipeline that modifies documents before indexing.

When a document passes through an ingest pipeline with an Enrich processor, Elasticsearch looks up the enrich index using the document's matching field. If a match is found, the processor adds fields from the enrich index to the document.

Result

Documents entering Elasticsearch are automatically enhanced with extra data before storage.

Knowing that enrichment happens during ingestion helps you design pipelines that improve search speed by avoiding runtime joins.

3

Intermediate

Configuring Enrich Processor Fields

🤔Before reading on: do you think you can enrich multiple fields at once or only one field per processor? Commit to your answer.

Concept: You can specify which fields to add or overwrite in the document from the enrich index.

The Enrich processor configuration allows you to define 'field' (the document field to match), 'target_field' (where to put the enriched data), and 'max_matches' (how many matches to allow). You can enrich multiple fields by adding multiple processors or using arrays.

Result

You control exactly what data is added and where, avoiding unwanted overwrites or data clutter.

Understanding field mapping in enrichment prevents data conflicts and keeps your documents clean and consistent.

4

Intermediate

Managing Enrich Policy Lifecycle

🤔Before reading on: do you think enrich policies update automatically with source data changes or require manual refresh? Commit to your answer.

Concept: Enrich policies must be executed to refresh their data after source changes.

After creating or updating an enrich policy, you run the 'execute' API to build or rebuild the enrich index. This step is manual and must be repeated whenever the source data changes to keep enrichment accurate.

Result

Your enrich data stays up-to-date only when you refresh the policy, ensuring correct enrichment.

Knowing the manual refresh requirement avoids stale data problems and ensures your enrichment reflects current information.

5

Advanced

Performance Considerations of Enrich Processor

🤔Before reading on: do you think enrichment slows down ingestion significantly or has minimal impact? Commit to your answer.

Concept: Enrich processor adds some overhead but is optimized for fast lookups using the enrich index.

The enrich index is optimized for quick key-value lookups, so enrichment is faster than runtime joins. However, large enrich indices or complex matching can slow ingestion. Proper sizing and limiting max_matches help maintain performance.

Result

You get faster searches with a small ingestion cost, balancing speed and resource use.

Understanding performance tradeoffs helps you design efficient pipelines and avoid bottlenecks.

6

Expert

Advanced Use: Chaining Enrich Processors

🤔Before reading on: do you think you can chain multiple enrich processors in one pipeline to enrich from different sources? Commit to your answer.

Concept: You can chain multiple enrich processors to enrich documents from different enrich policies sequentially.

By adding multiple enrich processors in an ingest pipeline, each referencing a different enrich policy, you can enrich documents with various data sets. This allows complex enrichment scenarios, like adding customer info and product details in one pass.

Result

Documents are enriched with multiple layers of data, improving search relevance and analytics.

Knowing how to chain enrich processors unlocks powerful multi-source enrichment strategies for complex data needs.

7

Expert

Internal Mechanics of Enrich Index Lookup

🤔Before reading on: do you think enrich lookups scan the entire enrich index or use optimized data structures? Commit to your answer.

Concept: Enrich lookups use specialized data structures for fast key-based retrieval.

The enrich index is built as a specialized index optimized for exact-match lookups on the matching field. It uses internal hash maps and caching to quickly find matching documents without scanning the whole index.

Result

Lookups are very fast and scalable even with large enrich data sets.

Understanding the internal lookup mechanism explains why enrich processor is efficient and how to optimize enrich index design.

Under the Hood

The Enrich processor queries a dedicated enrich index built from an enrich policy. This index stores key-value pairs optimized for exact-match lookups. During ingestion, the processor uses the document's matching field to quickly retrieve matching enrich data from this index and merges it into the document before indexing.

Why designed this way?

Elasticsearch designed the Enrich processor to avoid costly runtime joins by pre-building a fast lookup index. This design balances ingestion speed and query performance, allowing enrichment without duplicating data or slowing searches. Alternatives like runtime joins were rejected due to poor scalability.

┌───────────────┐       ┌───────────────────┐       ┌───────────────┐
│ Document In   │──────▶│ Enrich Processor  │──────▶│ Enrich Index  │
│ (with key)    │       │ (lookup & merge)  │       │ (fast lookup) │
└───────────────┘       └───────────────────┘       └───────────────┘
         │                        │                         ▲
         │                        │                         │
         ▼                        ▼                         │
┌────────────────────────────────────────────────────────────┐
│               Enriched Document Indexed                     │
└────────────────────────────────────────────────────────────┘

Myth Busters - 4 Common Misconceptions

Quick: Does the Enrich processor update enrichment data automatically when source data changes? Commit to yes or no.

Common Belief: The Enrich processor automatically updates enrichment data whenever the source index changes.

Reality: Enrich policies require manual execution to refresh the enrich index after source data changes; updates are not automatic.

Why it matters: Assuming automatic updates leads to stale enrichment data, causing inaccurate or outdated document information.

Quick: Can the Enrich processor perform fuzzy or partial matches? Commit to yes or no.

Common Belief: The Enrich processor can perform fuzzy or partial matches to enrich documents with approximate data.

Reality: The Enrich processor only supports exact matches on the specified field; fuzzy matching is not supported.

Why it matters: Expecting fuzzy matching can cause enrichment failures or missing data, leading to incomplete document enrichment.

Quick: Does using the Enrich processor eliminate the need for any joins at query time? Commit to yes or no.

Common Belief: Using the Enrich processor completely removes the need for joins during search queries.

Reality: While it reduces many join needs by enriching at ingestion, some complex joins or relationships may still require runtime joins or other methods.

Why it matters: Overreliance on enrichment can cause design blind spots where necessary query-time joins are overlooked, affecting search accuracy.

Quick: Is the Enrich processor suitable for very large enrich data sets without performance impact? Commit to yes or no.

Common Belief: The Enrich processor handles very large enrich data sets with no significant performance impact on ingestion.

Reality: Large enrich indices can slow ingestion and increase resource use; careful sizing and limits are needed.

Why it matters: Ignoring performance limits can cause slow ingestion pipelines and resource exhaustion in production.

Expert Zone

1

Enrich indices are optimized for exact-match lookups but do not support complex queries or aggregations, so enrichment data must be carefully structured.

2

The enrich processor merges data by overwriting or adding fields, so field naming conflicts can cause silent data loss if not managed.

3

Chaining multiple enrich processors can cause unexpected overwrites if target fields overlap, requiring careful pipeline design.

When NOT to use

Avoid using the Enrich processor when enrichment data changes very frequently and requires real-time updates; consider runtime joins or application-side enrichment instead. Also, if fuzzy or partial matching is needed, use alternative methods like scripted queries or external processing.

Production Patterns

In production, enrich processors are often used to add user profile data to logs, product details to sales events, or geo information to IP addresses. Pipelines typically include error handling and conditional enrichment to handle missing matches gracefully.

Connections

Database Join Operations

The Enrich processor performs a form of join at ingestion time, similar to SQL joins done at query time.

Understanding database joins helps grasp how enrichment merges related data, but Enrich processor shifts this work to ingestion for faster searches.

Cache Systems

Enrich indices act like a cache of lookup data optimized for fast retrieval during ingestion.

Knowing cache principles clarifies why enrich indices improve performance by avoiding repeated expensive lookups.

Supply Chain Management

Enriching documents is like adding supplier details to product shipments before delivery.

This connection shows how enrichment adds value by combining core data with related info early in a process, improving efficiency downstream.

Common Pitfalls

#1 Not refreshing enrich policy after source data changes.

Wrong approach:

PUT /_enrich/policy/customer_policy
{
"match": {
"indices": "customers",
"match_field": "customer_id",
"enrich_fields": ["name", "email"]
}
}

// Then ingest data without executing the policy

Correct approach:

PUT /_enrich/policy/customer_policy
{
"match": {
"indices": "customers",
"match_field": "customer_id",
"enrich_fields": ["name", "email"]
}
}

POST /_enrich/policy/customer_policy/_execute

// Then ingest data

Root cause:

Misunderstanding that enrich policies must be manually executed to build the enrich index before use.

#2 Expecting fuzzy matching in enrich processor.

Wrong approach:

"enrich": {
"policy_name": "customer_policy",
"field": "cust_name",
"target_field": "customer_info"
}
// where cust_name is partial or misspelled

Correct approach:

"enrich": {
"policy_name": "customer_policy",
"field": "customer_id",
"target_field": "customer_info"
}
// where customer_id is exact match

Root cause:

Assuming enrich processor supports approximate matching instead of exact key matching.

#3 Overwriting important fields unintentionally during enrichment.

Wrong approach:

"enrich": {
"policy_name": "product_policy",
"field": "product_id",
"target_field": "product_id"
}
// target_field same as source field

Correct approach:

"enrich": {
"policy_name": "product_policy",
"field": "product_id",
"target_field": "product_info"
}
// target_field different to avoid overwrite

Root cause:

Not separating enriched data fields from original document fields causes data loss.

Key Takeaways

The Enrich processor enhances documents by adding related data from a separate enrich index during ingestion.

Enrich policies define the source data and matching rules and must be manually executed to refresh enrichment data.

Enrichment uses exact-match lookups optimized for speed, not fuzzy or partial matching.

Proper configuration of fields and pipeline design prevents data conflicts and performance issues.

Advanced use includes chaining multiple enrich processors and understanding internal lookup mechanisms for efficient production use.

Practice

1. What is the main purpose of the enrich processor in Elasticsearch?

easy

A. To perform full-text search on documents

B. To delete documents based on a condition

C. To add extra data to documents by matching fields from an enrich policy

D. To create new indexes automatically

Solution

1. Step 1: Understand the enrich processor role

   The enrich processor is designed to add additional data to documents by matching fields from an external data source called an enrich policy.

2. Step 2: Compare options with enrich processor function

   Full-text search, document deletion, and automatic index creation describe other Elasticsearch features, not enrich processor functionality.

3. Final Answer:

   To add extra data to documents by matching fields from an enrich policy -> Option C

4. Quick Check:

   Enrich processor = add matched data [OK]

Hint: Enrich processor adds data by matching fields [OK]

Common Mistakes:

• Confusing enrich processor with delete or search operations
• Thinking enrich processor creates indexes
• Assuming it modifies original data sources

2. Which of the following is the correct syntax to define an enrich processor in an ingest pipeline?

easy

A. "enrich": {policy_name: "my_policy", field: "user_id", target_field: "user_info"}

B. "enrich": {"policy_name": "my_policy", "field": "user_id", "target_field": "user_info"}

C. "enrich": {"index": "my_index", "field": "user_id"}

D. "enrich": {"policy": "my_policy", "match_field": "user_id"}

Solution

1. Step 1: Recall enrich processor syntax

   The enrich processor requires the keys: policy_name, field, and target_field to specify the enrich policy, the field to match, and where to store enriched data.

2. Step 2: Identify correct option

   "enrich": {"policy_name": "my_policy", "field": "user_id", "target_field": "user_info"} correctly uses policy_name, field, and target_field with proper JSON quoting. Unquoted keys like policy_name cause syntax errors. Using index or match_field are incorrect keys.

3. Final Answer:

   "enrich": {"policy_name": "my_policy", "field": "user_id", "target_field": "user_info"} -> Option B

4. Quick Check:

   Correct keys = policy_name, field, target_field [OK]

Hint: Use policy_name, field, and target_field keys [OK]

Common Mistakes:

• Using wrong keys like 'index' or 'match_field'
• Omitting target_field key
• Incorrect JSON syntax or missing quotes

3. Given the following enrich processor configuration in an ingest pipeline:

{
  "enrich": {
    "policy_name": "user_policy",
    "field": "user_id",
    "target_field": "user_data"
  }
}

If a document {"user_id": "123"} matches the enrich policy with {"user_id": "123", "name": "Alice"}, what will be the resulting document after enrichment?

medium

A. {"user_id": "123", "user_data": {"user_id": "123", "name": "Alice"}}

B. {"user_id": "123", "name": "Alice"}

C. {"user_id": "123", "user_data": "Alice"}

D. {"user_id": "123"}

Solution

1. Step 1: Understand enrich processor output

   The enrich processor adds matched data under the target_field. Here, user_data will contain the matched document from the enrich policy.

2. Step 2: Apply to given document

   The original document has user_id: 123. The enrich policy matches and returns {"user_id": "123", "name": "Alice"}. This is added under user_data.

3. Final Answer:

   {"user_id": "123", "user_data": {"user_id": "123", "name": "Alice"}} -> Option A

4. Quick Check:

   Enrich adds matched doc under target_field [OK]

Hint: Matched data goes inside target_field key [OK]

Common Mistakes:

• Replacing original fields instead of adding under target_field
• Adding only a single field value instead of full matched doc
• Not nesting matched data inside target_field

4. You have this enrich processor configuration:

{
  "enrich": {
    "policy_name": "product_policy",
    "field": "product_id"
  }
}

When running the pipeline, you get an error. What is the likely cause?

medium

A. Missing target_field key in the enrich processor configuration

B. Incorrect policy_name value

C. The field value should be product_name

D. Enrich processor cannot be used in ingest pipelines

Solution

1. Step 1: Check required keys for enrich processor

   The enrich processor requires policy_name, field, and target_field. The configuration is missing target_field.

2. Step 2: Validate other options

   Incorrect policy_name value might cause errors but is not guaranteed. The field value should be product_name is incorrect because field should match the field to join on. Enrich processor cannot be used in ingest pipelines is false; enrich processor is valid in ingest pipelines.

3. Final Answer:

   Missing target_field key in the enrich processor configuration -> Option A

4. Quick Check:

   target_field is required in enrich processor [OK]

Hint: Always include target_field in enrich processor config [OK]

Common Mistakes:

• Forgetting target_field key
• Assuming enrich processor can't be in pipelines
• Changing field name incorrectly

5. You want to enrich incoming documents with user details but only if the user is active. Which approach correctly applies this condition using the enrich processor in an ingest pipeline?

hard

A. Use the enrich processor with a condition key to check user.active == true

B. Add a match condition inside the enrich processor to filter active users

C. Use the enrich processor with policy_name and then add a script processor to filter by user.active == true

D. Create an enrich policy that only indexes active users and use enrich processor normally

Solution

1. Step 1: Understand enrich processor filtering capabilities

   The enrich processor itself does not support filtering conditions inside its config. Filtering must be done before or by the enrich policy data.

2. Step 2: Choose correct approach for conditional enrichment

   Creating an enrich policy that only contains active users ensures only active user data is matched and added. Adding a match condition or condition key inside the enrich processor is unsupported syntax. Using a script processor after does not prevent enrichment for inactive users.

3. Final Answer:

   Create an enrich policy that only indexes active users and use enrich processor normally -> Option D

4. Quick Check:

   Filter data in enrich policy, not in processor [OK]

Hint: Filter data in enrich policy, not processor config [OK]

Common Mistakes:

• Trying to filter inside enrich processor config
• Using unsupported condition keys
• Filtering after enrichment instead of before