Enrich processor in Elasticsearch - Time & Space Complexity

Time Complexity: Enrich processor

O(n)

Understanding Time Complexity

When using the enrich processor in Elasticsearch, it's important to understand how the time to process documents changes as the number of documents grows.

We want to know how the processor's work increases when more documents need enrichment.

Scenario Under Consideration

Analyze the time complexity of the following enrich processor configuration snippet.

POST _ingest/pipeline/_simulate
{
  "pipeline": {
    "processors": [
      {
        "enrich": {
          "policy_name": "user_policy",
          "field": "user_id",
          "target_field": "user_info"
        }
      }
    ]
  },
  "docs": [
    {"_source": {"user_id": "123"}}
  ]
}
    

This snippet simulates enriching documents by looking up user info based on user_id using a stored enrich policy.

Identify Repeating Operations

In this process, the main repeating operation is the lookup for each document.

• Primary operation: For each document, the enrich processor performs a lookup in the enrich index.
• How many times: Once per document being processed.

How Execution Grows With Input

As the number of documents increases, the total number of lookups grows linearly.

Input Size (n)	Approx. Operations (lookups)
10	10 lookups
100	100 lookups
1000	1000 lookups

Pattern observation: Each new document adds one more lookup, so the work grows steadily with the number of documents.

Final Time Complexity

Time Complexity: O(n)

This means the time to enrich documents grows directly in proportion to how many documents you process.

Common Mistake

[X] Wrong: "The enrich processor does all lookups once and reuses results, so time stays the same no matter how many documents."

[OK] Correct: Each document requires its own lookup because different documents have different keys to enrich, so the processor must do work for each one.

Interview Connect

Understanding how the enrich processor scales helps you explain how Elasticsearch handles data enrichment efficiently, a useful skill when discussing data pipelines and search performance.

Self-Check

"What if the enrich index was cached in memory for faster lookups? How would that affect the time complexity?"

Practice

1. What is the main purpose of the enrich processor in Elasticsearch?

easy

A. To perform full-text search on documents

B. To delete documents based on a condition

C. To add extra data to documents by matching fields from an enrich policy

D. To create new indexes automatically

Solution

1. Step 1: Understand the enrich processor role

   The enrich processor is designed to add additional data to documents by matching fields from an external data source called an enrich policy.

2. Step 2: Compare options with enrich processor function

   Full-text search, document deletion, and automatic index creation describe other Elasticsearch features, not enrich processor functionality.

3. Final Answer:

   To add extra data to documents by matching fields from an enrich policy -> Option C

4. Quick Check:

   Enrich processor = add matched data [OK]

Hint: Enrich processor adds data by matching fields [OK]

Common Mistakes:

• Confusing enrich processor with delete or search operations
• Thinking enrich processor creates indexes
• Assuming it modifies original data sources

2. Which of the following is the correct syntax to define an enrich processor in an ingest pipeline?

easy

A. "enrich": {policy_name: "my_policy", field: "user_id", target_field: "user_info"}

B. "enrich": {"policy_name": "my_policy", "field": "user_id", "target_field": "user_info"}

C. "enrich": {"index": "my_index", "field": "user_id"}

D. "enrich": {"policy": "my_policy", "match_field": "user_id"}

Solution

1. Step 1: Recall enrich processor syntax

   The enrich processor requires the keys: policy_name, field, and target_field to specify the enrich policy, the field to match, and where to store enriched data.

2. Step 2: Identify correct option

   "enrich": {"policy_name": "my_policy", "field": "user_id", "target_field": "user_info"} correctly uses policy_name, field, and target_field with proper JSON quoting. Unquoted keys like policy_name cause syntax errors. Using index or match_field are incorrect keys.

3. Final Answer:

   "enrich": {"policy_name": "my_policy", "field": "user_id", "target_field": "user_info"} -> Option B

4. Quick Check:

   Correct keys = policy_name, field, target_field [OK]

Hint: Use policy_name, field, and target_field keys [OK]

Common Mistakes:

• Using wrong keys like 'index' or 'match_field'
• Omitting target_field key
• Incorrect JSON syntax or missing quotes

3. Given the following enrich processor configuration in an ingest pipeline:

{
  "enrich": {
    "policy_name": "user_policy",
    "field": "user_id",
    "target_field": "user_data"
  }
}

If a document {"user_id": "123"} matches the enrich policy with {"user_id": "123", "name": "Alice"}, what will be the resulting document after enrichment?

medium

A. {"user_id": "123", "user_data": {"user_id": "123", "name": "Alice"}}

B. {"user_id": "123", "name": "Alice"}

C. {"user_id": "123", "user_data": "Alice"}

D. {"user_id": "123"}

Solution

1. Step 1: Understand enrich processor output

   The enrich processor adds matched data under the target_field. Here, user_data will contain the matched document from the enrich policy.

2. Step 2: Apply to given document

   The original document has user_id: 123. The enrich policy matches and returns {"user_id": "123", "name": "Alice"}. This is added under user_data.

3. Final Answer:

   {"user_id": "123", "user_data": {"user_id": "123", "name": "Alice"}} -> Option A

4. Quick Check:

   Enrich adds matched doc under target_field [OK]

Hint: Matched data goes inside target_field key [OK]

Common Mistakes:

• Replacing original fields instead of adding under target_field
• Adding only a single field value instead of full matched doc
• Not nesting matched data inside target_field

4. You have this enrich processor configuration:

{
  "enrich": {
    "policy_name": "product_policy",
    "field": "product_id"
  }
}

When running the pipeline, you get an error. What is the likely cause?

medium

A. Missing target_field key in the enrich processor configuration

B. Incorrect policy_name value

C. The field value should be product_name

D. Enrich processor cannot be used in ingest pipelines

Solution

1. Step 1: Check required keys for enrich processor

   The enrich processor requires policy_name, field, and target_field. The configuration is missing target_field.

2. Step 2: Validate other options

   Incorrect policy_name value might cause errors but is not guaranteed. The field value should be product_name is incorrect because field should match the field to join on. Enrich processor cannot be used in ingest pipelines is false; enrich processor is valid in ingest pipelines.

3. Final Answer:

   Missing target_field key in the enrich processor configuration -> Option A

4. Quick Check:

   target_field is required in enrich processor [OK]

Hint: Always include target_field in enrich processor config [OK]

Common Mistakes:

• Forgetting target_field key
• Assuming enrich processor can't be in pipelines
• Changing field name incorrectly

5. You want to enrich incoming documents with user details but only if the user is active. Which approach correctly applies this condition using the enrich processor in an ingest pipeline?

hard

A. Use the enrich processor with a condition key to check user.active == true

B. Add a match condition inside the enrich processor to filter active users

C. Use the enrich processor with policy_name and then add a script processor to filter by user.active == true

D. Create an enrich policy that only indexes active users and use enrich processor normally

Solution

1. Step 1: Understand enrich processor filtering capabilities

   The enrich processor itself does not support filtering conditions inside its config. Filtering must be done before or by the enrich policy data.

2. Step 2: Choose correct approach for conditional enrichment

   Creating an enrich policy that only contains active users ensures only active user data is matched and added. Adding a match condition or condition key inside the enrich processor is unsupported syntax. Using a script processor after does not prevent enrichment for inactive users.

3. Final Answer:

   Create an enrich policy that only indexes active users and use enrich processor normally -> Option D

4. Quick Check:

   Filter data in enrich policy, not in processor [OK]

Hint: Filter data in enrich policy, not processor config [OK]

Common Mistakes:

• Trying to filter inside enrich processor config
• Using unsupported condition keys
• Filtering after enrichment instead of before