Percolate queries (reverse search) in Elasticsearch - Step-by-Step Execution

Concept Flow - Percolate queries (reverse search)

Index percolator queries

↓

New document arrives

↓

Percolate query runs

↓

Match queries found

↓

Return matching query IDs

Percolate queries let you register queries first, then check which queries match a new document.

Execution Sample

Elasticsearch

PUT /my-index
{
  "mappings": {
    "properties": {
      "query": { "type": "percolator" },
      "message": { "type": "text" }
    }
  }
}

PUT /my-index/_doc/1
{
  "query": { "match": { "message": "error" } }
}

GET /my-index/_search
{
  "query": {
    "percolate": {
      "field": "query",
      "document": {
        "message": "this is an error message"
      }
    }
  }
}

This code registers a percolator query matching 'error' in 'message', then searches which queries match a new document.

Execution Table

Step	Action	Input/Condition	Result/Output
1	Create index with percolator field	Define 'query' as percolator type	Index ready to store queries
2	Index percolator query	Query: match 'message' contains 'error'	Query stored with ID 1
3	New document arrives	Document: message='this is an error message'	Document ready for percolation
4	Run percolate query	Check which stored queries match document	Query ID 1 matches document
5	Return matching queries	Matches found	Response includes query ID 1
6	End	No more queries	Percolate search complete

💡 All stored queries checked against the new document; matching queries returned.

Variable Tracker

Variable	Start	After Step 2	After Step 3	After Step 4	Final
Index	empty	created with percolator field	unchanged	unchanged	unchanged
Stored Queries	none	1 query stored (match 'error')	unchanged	unchanged	unchanged
New Document	none	none	document with message='this is an error message'	unchanged	unchanged
Matching Queries	none	none	none	query ID 1 found	query ID 1 returned

Key Moments - 3 Insights

Why do we index queries instead of documents in percolate?

Because percolate reverses the usual search: it stores queries first, then checks which queries match new documents (see execution_table step 2 and 4).

What does the 'percolate' query field refer to?

It refers to the field of type 'percolator' where queries are stored, as shown in execution_table step 1 and 4.

Why does the new document not get indexed normally?

Because percolate queries check the document against stored queries without indexing it as a normal document (see execution_table step 3 and 4).

Visual Quiz - 3 Questions

Look at the execution_table, at which step is the new document prepared for percolation?

A Step 3

B Step 2

C Step 4

D Step 5

💡 Hint

Check the 'Action' and 'Input/Condition' columns for document arrival.

According to variable_tracker, what is the state of 'Matching Queries' after Step 4?

A 1 query stored

B none

C query ID 1 found

D document with message

💡 Hint

Look at the 'Matching Queries' row under 'After Step 4' column.

If we add another percolator query matching 'warning', what changes in the execution_table?

A Step 3 will change the document content

B Step 2 will show 2 queries stored

C Step 4 will find 2 matches for the same document

D Step 5 will return no matches

💡 Hint

Adding queries affects stored queries count (see variable_tracker 'Stored Queries').

Concept Snapshot

Percolate queries let you store queries first.
When a new document arrives, Elasticsearch checks which stored queries match it.
Define a 'percolator' field in mapping to store queries.
Index queries as documents with 'query' field.
Use 'percolate' query to find matching queries for a document.
Useful for alerting or reverse search scenarios.

Full Transcript

Percolate queries in Elasticsearch reverse the usual search process. Instead of storing documents and searching with queries, you store queries first in a special 'percolator' field. When a new document arrives, Elasticsearch runs these stored queries against the document to find matches. This is useful for alerting or matching documents to predefined criteria. The process involves creating an index with a percolator field, indexing queries as documents, then running a percolate query with a new document to find which queries match. The execution table shows each step from index creation, query storage, document arrival, percolate query execution, to returning matching queries. Variable tracking shows how stored queries and matching queries change over time. Key moments clarify why queries are stored, what the percolate field means, and why documents are not indexed normally. The visual quiz tests understanding of steps and variable states. The snapshot summarizes the concept in simple lines.

Practice

1.

What is the main purpose of a percolate query in Elasticsearch?

easy

A. To find stored queries that match a new document

B. To update documents in an index

C. To delete documents based on a condition

D. To aggregate data by terms

Solution

1. Step 1: Understand percolate query concept

   A percolate query is used to find stored queries that match a new document, reversing the usual search direction.

2. Step 2: Compare options with concept

   The other options describe other Elasticsearch operations, not percolate queries.

3. Final Answer:

   To find stored queries that match a new document -> Option A

4. Quick Check:

   Percolate query = find matching stored queries [OK]

Hint: Percolate queries match queries to documents, not documents to queries [OK]

Common Mistakes:

• Confusing percolate query with regular search
• Thinking it updates or deletes documents
• Mixing it with aggregation queries

2.

Which mapping type must be included in an Elasticsearch index to use percolate queries?

{
  "mappings": {
    "properties": {
      "query": {
        "type": "???"
      }
    }
  }
}

easy

A. "percolator"

B. "text"

C. "keyword"

D. "nested"

Solution

1. Step 1: Identify required field type for percolate queries

   Elasticsearch requires a special field type called "percolator" to store queries for percolate queries.

2. Step 2: Match options with required type

   Only "percolator" uses "percolator" type; others are for different purposes.

3. Final Answer:

   "percolator" -> Option A

4. Quick Check:

   Percolate field type = "percolator" [OK]

Hint: Use "percolator" type for storing queries in mapping [OK]

Common Mistakes:

• Using "text" or "keyword" instead of "percolator"
• Confusing nested type with percolator
• Omitting the percolator field in mapping

3.

Given the following percolate query, what will it return?

{
  "query": {
    "percolate": {
      "field": "query",
      "document": {
        "message": "Elasticsearch alerting"
      }
    }
  }
}

Assuming the index has stored queries matching documents containing "alerting".

medium

A. Documents containing the word "alerting"

B. An error because "document" is missing an ID

C. All documents in the index

D. Stored queries that match the document with message "Elasticsearch alerting"

Solution

1. Step 1: Understand percolate query behavior

   The percolate query matches stored queries against the provided document, returning matching stored queries.

2. Step 2: Analyze the given query

   The query uses "document" with a message field; it will find stored queries matching this document's content.

3. Final Answer:

   Stored queries that match the document with message "Elasticsearch alerting" -> Option D

4. Quick Check:

   Percolate query returns matching stored queries [OK]

Hint: Percolate queries return stored queries matching the input document [OK]

Common Mistakes:

• Thinking it returns documents instead of queries
• Assuming document ID is required for percolate query
• Confusing percolate with regular search

4.

Identify the error in this percolate query:

{
  "query": {
    "percolate": {
      "field": "query"
      "document": {
        "content": "Test document"
      }
    }
  }
}

medium

A. "field" should be "query_field"

B. Missing comma between "field" and "document" fields

C. "document" must include an "id" field

D. Percolate query cannot use 'content' field in document

Solution

1. Step 1: Check JSON syntax in query

   Between "field" and "document" keys, a comma is missing, causing invalid JSON.

2. Step 2: Validate other parts

   "field" name is correct, "document" can omit "id", and "content" is valid as document content.

3. Final Answer:

   Missing comma between "field" and "document" fields -> Option B

4. Quick Check:

   JSON syntax error = missing comma [OK]

Hint: Check commas between JSON fields carefully [OK]

Common Mistakes:

• Forgetting commas between JSON keys
• Assuming document must have an ID
• Changing field names unnecessarily

5.

You want to build an alert system that triggers when new documents match any stored queries. Which steps are necessary to implement this using percolate queries?

hard

A. Use aggregation queries on documents to find alerts

B. Store documents in a normal index, then run a regular search for alerts

C. Create an index with a percolator field, store queries, then percolate new documents against stored queries

D. Create a nested field for queries and filter documents manually

Solution

1. Step 1: Setup index with percolator field

   Define an index mapping with a "percolator" type field to store queries for reverse matching.

2. Step 2: Store queries and percolate new documents

   Index the alert queries into the percolator field, then use percolate queries to check if new documents match any stored queries.

3. Final Answer:

   Create an index with a percolator field, store queries, then percolate new documents against stored queries -> Option C

4. Quick Check:

   Percolate queries enable alerting by matching docs to stored queries [OK]

Hint: Store queries in percolator field, then percolate new docs [OK]

Common Mistakes:

• Using regular search instead of percolate queries
• Not defining percolator field in mapping
• Trying to use aggregations for alerting