Why Filter aggregation in Elasticsearch? - Purpose & Use Cases

The Big Idea

What if you could instantly find exactly what you need in a mountain of data without lifting a finger?

The Scenario

Imagine you have a huge pile of documents and you want to find how many of them match a specific condition, like all documents where the status is "active". Doing this by checking each document one by one is like searching for a needle in a haystack manually.

The Problem

Manually filtering documents is slow and tiring. It takes a lot of time to scan through every document, and it's easy to make mistakes or miss some. Also, if you want to count how many match, you have to keep track yourself, which can be confusing and error-prone.

The Solution

Filter aggregation lets Elasticsearch quickly find and count documents that match your condition without looking at every single one manually. It's like having a smart assistant who instantly points out the matching documents and tells you how many there are.

Before vs After

✗ Before

{ "query": { "match": { "status": "active" } } }
// Then count results manually

✓ After

{ "aggs": { "active_docs": { "filter": { "term": { "status": "active" } } } } }

What It Enables

It enables fast, accurate counting and grouping of documents based on specific conditions, making data analysis simple and efficient.

Real Life Example

For example, an online store can quickly find how many orders are "shipped" versus "pending" without scanning every order manually, helping them track shipments in real time.

Key Takeaways

Manually filtering large data is slow and error-prone.

Filter aggregation quickly finds and counts matching documents.

This makes data analysis faster and more reliable.

Practice

1. What is the main purpose of a filter aggregation in Elasticsearch?

easy

A. To delete documents matching a query

B. To sort documents by a field value

C. To update documents in the index

D. To select documents that match a specific condition

Solution

1. Step 1: Understand filter aggregation role

   A filter aggregation is used to select documents that meet a certain condition or criteria.

2. Step 2: Compare with other options

   Sorting, updating, or deleting documents are not functions of filter aggregation.

3. Final Answer:

   To select documents that match a specific condition -> Option D

4. Quick Check:

   Filter aggregation = select matching documents [OK]

Hint: Filter aggregation selects matching documents only [OK]

Common Mistakes:

• Confusing filter aggregation with sorting
• Thinking filter aggregation modifies documents
• Mixing filter aggregation with delete operations

2. Which of the following is the correct syntax to define a filter aggregation in an Elasticsearch query?

easy

A. "aggs": { "my_filter": { "filter": { "term": { "status": "active" } } } }

B. "aggs": { "my_filter": { "terms": { "field": "status" } } }

C. "aggs": { "my_filter": { "range": { "field": "age", "gte": 30 } } }

D. "aggs": { "my_filter": { "match": { "status": "active" } } }

Solution

1. Step 1: Identify filter aggregation syntax

   The filter aggregation uses the key "filter" with a query inside, like "term" for exact match.

2. Step 2: Check each option

   "aggs": { "my_filter": { "filter": { "term": { "status": "active" } } } } correctly uses "filter" with a "term" query. Options A, C, and D use other aggregation types or query types incorrectly.

3. Final Answer:

   "aggs": { "my_filter": { "filter": { "term": { "status": "active" } } } } -> Option A

4. Quick Check:

   Filter aggregation syntax = option with "filter" key [OK]

Hint: Filter aggregation always uses "filter" key inside aggs [OK]

Common Mistakes:

• Using "terms" instead of "filter" for filter aggregation
• Confusing query types inside aggregation
• Missing the "filter" key in aggregation

3. Given this Elasticsearch aggregation query snippet, what will be the count of documents in the filter aggregation named active_users?

{
  "aggs": {
    "active_users": {
      "filter": { "term": { "status": "active" } }
    }
  }
}

Assuming the index has 1000 documents, 300 of which have status set to active.

medium

A. 300

B. 1000

C. 700

D. 0

Solution

1. Step 1: Understand filter aggregation result

   The filter aggregation counts documents matching the filter condition, here "status" = "active".

2. Step 2: Apply given data

   Out of 1000 documents, 300 have status "active", so the aggregation count is 300.

3. Final Answer:

   300 -> Option A

4. Quick Check:

   Filter count = matching docs = 300 [OK]

Hint: Filter aggregation count = number of matching documents [OK]

Common Mistakes:

• Choosing total documents instead of filtered count
• Confusing filter aggregation with terms aggregation count
• Assuming zero if filter is not exact

4. Identify the error in this filter aggregation query:

{
  "aggs": {
    "filtered_data": {
      "filter": {
        "terms": {
          "status": "active"
        }
      }
    }
  }
}

medium

A. The filter aggregation must use "term" not "terms" for single value

B. The "terms" query requires an array of values, not a single string

C. The aggregation name "filtered_data" is invalid

D. The filter aggregation cannot be nested inside "aggs"

Solution

1. Step 1: Check the "terms" query usage

   The "terms" query expects an array of values, e.g., ["active"], not a single string.

2. Step 2: Validate other parts

   The aggregation name is valid, and filter aggregation can be inside "aggs". Using "term" is an alternative but not an error here.

3. Final Answer:

   The "terms" query requires an array of values, not a single string -> Option B

4. Quick Check:

   "terms" query needs array, not string [OK]

Hint: "terms" query needs array of values, not single string [OK]

Common Mistakes:

• Using string instead of array for "terms" query
• Thinking aggregation names have restrictions
• Believing filter aggregation can't be nested

5. You want to find the average price of products only in the category "electronics" using filter aggregation. Which of the following query snippets correctly combines filter aggregation with an average sub-aggregation?

hard

A. {
"query": { "term": { "category": "electronics" } },
"aggs": { "avg_price": { "avg": { "field": "price" } } }
}

B. {
"aggs": {
"avg_price": { "avg": { "field": "price" } },
"filter": { "term": { "category": "electronics" } }
}
}

C. {
"aggs": {
"electronics_filter": {
"filter": { "term": { "category": "electronics" } },
"aggs": { "avg_price": { "avg": { "field": "price" } } }
}
}
}

D. {
"aggs": {
"electronics_filter": {
"terms": { "field": "category" },
"aggs": { "avg_price": { "avg": { "field": "price" } } }
}
}
}

Solution

1. Step 1: Understand filter aggregation with sub-aggregation

   Filter aggregation uses "filter" key and can contain sub-aggregations inside "aggs".

2. Step 2: Check each option

   { "aggs": { "electronics_filter": { "filter": { "term": { "category": "electronics" } }, "aggs": { "avg_price": { "avg": { "field": "price" } } } } } } correctly nests an average aggregation inside a filter aggregation for category "electronics". { "aggs": { "avg_price": { "avg": { "field": "price" } }, "filter": { "term": { "category": "electronics" } } } } misplaces filter outside aggregation. { "query": { "term": { "category": "electronics" } }, "aggs": { "avg_price": { "avg": { "field": "price" } } } } uses query filtering, not aggregation filtering. { "aggs": { "electronics_filter": { "terms": { "field": "category" }, "aggs": { "avg_price": { "avg": { "field": "price" } } } } } } uses "terms" aggregation, not filter.

3. Final Answer:

   { "aggs": { "electronics_filter": { "filter": { "term": { "category": "electronics" } }, "aggs": { "avg_price": { "avg": { "field": "price" } } } } } } -> Option C

4. Quick Check:

   Filter aggregation with sub-aggregation = { "aggs": { "electronics_filter": { "filter": { "term": { "category": "electronics" } }, "aggs": { "avg_price": { "avg": { "field": "price" } } } } } } [OK]

Hint: Nest avg aggregation inside filter aggregation's "aggs" [OK]

Common Mistakes:

• Placing filter outside aggregation block
• Using query filter instead of filter aggregation
• Confusing terms aggregation with filter aggregation