Saved searches and filters in Elasticsearch - Deep Dive

Overview - Saved searches and filters

What is it?

Saved searches and filters in Elasticsearch let you store specific search queries or filter conditions so you can reuse them easily without rewriting. A saved search remembers the exact criteria you used to find data, while filters narrow down results by specific rules. This helps you quickly find the same data again or apply consistent rules across different searches.

Why it matters

Without saved searches and filters, you would have to write the same complex queries every time you want to find similar data. This wastes time and can cause mistakes if you forget details. Saved searches and filters make your work faster, more reliable, and consistent, especially when dealing with large amounts of data.

Where it fits

Before learning saved searches and filters, you should understand basic Elasticsearch queries and how to search data. After mastering saved searches and filters, you can explore dashboards, visualizations, and automation that use these saved queries to monitor and analyze data continuously.

Mental Model

Core Idea

Saved searches and filters are like bookmarks and stencils for your data queries, letting you quickly reuse and apply precise search rules without rewriting them.

Think of it like...

Imagine you have a recipe book where you mark your favorite recipes (saved searches) and use cookie cutters (filters) to shape dough in specific ways every time you bake. This saves you from writing the recipe or shaping dough from scratch each time.

┌───────────────────────────────┐
│         Saved Search           │
│  (Stored Query + Filters)      │
├───────────────┬───────────────┤
│ Query Part    │ Filter Part   │
│ (What to find)│ (Narrow rules)│
└───────────────┴───────────────┘

Build-Up - 7 Steps

1

Foundation

Understanding Basic Elasticsearch Queries

Concept: Learn how Elasticsearch searches data using queries and filters.

Elasticsearch lets you search data by writing queries that describe what you want. For example, you can search for documents where the field 'status' is 'active'. Filters are special parts of queries that quickly narrow down results without scoring them.

Result

You can find documents matching your criteria, like all active users.

Understanding queries and filters is essential because saved searches and filters build on these basic search rules.

2

Foundation

Difference Between Queries and Filters

Concept: Distinguish how queries score results and filters just include/exclude data.

Queries calculate how well each document matches and rank them. Filters simply say yes or no to each document, making searches faster when you only want to include or exclude data based on exact conditions.

Result

You know when to use filters for speed and queries for relevance.

Knowing this difference helps you create efficient saved searches that run quickly.

3

Intermediate

Creating and Saving Searches

🤔Before reading on: Do you think saved searches store only the query text or the full search including filters and settings? Commit to your answer.

Concept: Saved searches store the entire search setup, including queries, filters, and display options.

In Elasticsearch tools like Kibana, you can build a search with queries and filters, then save it with a name. This saved search remembers all parts so you can run it again exactly or share it with others.

Result

You have a reusable search that returns the same results anytime you run it.

Understanding that saved searches capture the full search context prevents confusion when results differ if you try to recreate searches manually.

4

Intermediate

Using Filters to Refine Saved Searches

🤔Before reading on: Can filters in saved searches be changed later without editing the original saved search? Commit to your answer.

Concept: Filters can be layered on top of saved searches or edited within them to refine results.

You can add filters to a saved search to narrow results further or modify existing filters. This flexibility lets you adapt saved searches to new needs without creating new ones from scratch.

Result

You get more precise results by adjusting filters on saved searches.

Knowing filters are flexible tools within saved searches helps you maintain efficient workflows without cluttering your saved searches list.

5

Intermediate

Sharing and Reusing Saved Searches

Concept: Saved searches can be shared with others or reused in dashboards and reports.

Once saved, searches can be accessed by team members or embedded in dashboards to provide consistent views of data. This promotes collaboration and standardizes how data is analyzed.

Result

Teams work with the same data views, reducing errors and improving communication.

Recognizing saved searches as collaboration tools elevates their value beyond personal convenience.

6

Advanced

Optimizing Filters for Performance

🤔Before reading on: Do you think all filters have the same impact on search speed? Commit to your answer.

Concept: Some filters are faster because they use cached results or simple conditions, improving search speed.

Filters like term or range filters are fast because Elasticsearch caches their results. Complex filters or scripts slow down searches. Designing saved searches with efficient filters improves performance, especially on large datasets.

Result

Saved searches run faster and use fewer resources.

Understanding filter performance helps you build saved searches that scale well in production.

7

Expert

Internals of Saved Searches and Filter Caching

🤔Before reading on: Do you think saved searches store data or just the query definitions? Commit to your answer.

Concept: Saved searches store query definitions, and Elasticsearch caches filter results to speed repeated queries.

Elasticsearch does not save the actual data results of a saved search. Instead, it saves the query and filter definitions. When you run a saved search, Elasticsearch uses filter caches to quickly find matching documents without reprocessing filters each time.

Result

Saved searches are lightweight and fast, relying on caching mechanisms.

Knowing that saved searches rely on caching explains why changing filters or data can affect results and performance.

Under the Hood

Saved searches store the JSON query and filter definitions in Elasticsearch or Kibana's saved objects. When executed, Elasticsearch parses these definitions and applies filters using its inverted index and filter cache. The filter cache stores results of frequent filters to avoid recomputation, speeding up repeated queries. Filters are applied before scoring, making searches efficient. Saved searches do not store actual data snapshots, so results reflect current data.

Why designed this way?

This design separates query definitions from data, keeping saved searches lightweight and flexible. Caching filters improves performance without duplicating data. Alternatives like storing full result sets would be heavy and stale quickly. The approach balances speed, storage, and freshness.

┌───────────────┐      ┌───────────────┐      ┌───────────────┐
│ Saved Search  │─────▶│ Query & Filter│─────▶│ Elasticsearch │
│ (Query JSON)  │      │ Parsing       │      │ Engine        │
└───────────────┘      └───────────────┘      └───────────────┘
                                   │
                                   ▼
                         ┌───────────────────┐
                         │ Filter Cache      │
                         │ (Cached Results)  │
                         └───────────────────┘

Myth Busters - 4 Common Misconceptions

Quick: Does saving a search store the actual data results or just the search rules? Commit to your answer.

Common Belief: Saving a search stores the data results so you can view the same snapshot anytime.

Reality: Saved searches only store the query and filter definitions, not the data itself. Results reflect current data when run.

Why it matters: Expecting saved searches to show fixed data can cause confusion when data changes and results differ.

Quick: Do filters always improve search speed? Commit to your answer.

Common Belief: Adding filters always makes searches faster because they reduce data.

Reality: Some filters are slow, especially complex or scripted ones, and can hurt performance.

Why it matters: Blindly adding filters without understanding their cost can degrade search speed.

Quick: Can you edit a saved search's filters without changing the saved search itself? Commit to your answer.

Common Belief: Filters in saved searches are fixed and cannot be changed without editing the saved search.

Reality: You can add or modify filters on top of saved searches temporarily without altering the saved search definition.

Why it matters: Knowing this flexibility prevents unnecessary duplication of saved searches.

Quick: Are saved searches only useful for personal use? Commit to your answer.

Common Belief: Saved searches are just personal shortcuts and not meant for sharing or dashboards.

Reality: Saved searches are designed for sharing and integration into dashboards and reports for teams.

Why it matters: Underestimating saved searches limits collaboration and consistent data analysis.

Expert Zone

1

Filters that use exact matches (term filters) benefit most from caching, while range or script filters may not cache well, affecting performance.

2

Saved searches do not version data; if underlying data changes, results change, so saved searches are not snapshots but live queries.

3

In Kibana, saved searches are stored as saved objects with metadata, enabling export/import and integration with other tools.

When NOT to use

Avoid saved searches when you need fixed snapshots of data at a point in time; instead, use data exports or snapshots. For highly dynamic or complex queries with many scripted filters, consider custom query optimization or pre-aggregated data to improve performance.

Production Patterns

In production, saved searches are often used as building blocks for dashboards, alerts, and reports. Teams create shared saved searches to standardize monitoring views. Filters are carefully designed to balance precision and speed, and saved searches are version-controlled to track changes.

Connections

Caching in Web Browsers

Both use caching to speed up repeated requests by storing previous results or computations.

Understanding how browsers cache web pages helps grasp why Elasticsearch caches filter results to avoid reprocessing.

Bookmarks in Web Browsers

Saved searches are like bookmarks that store a location (query) so you can return quickly without retyping.

Knowing how bookmarks save URLs helps understand saved searches as stored queries for quick reuse.

Recipe Management in Cooking

Saved searches and filters are like saved recipes and tools (cookie cutters) that standardize and speed up cooking.

This connection shows how saving and reusing instructions and tools improves efficiency in different fields.

Common Pitfalls

#1 Expecting saved searches to show the same data snapshot over time.

Wrong approach:

Run saved search and assume results never change even if data updates.

Correct approach:

Understand saved searches run live queries reflecting current data each time.

Root cause:

Misunderstanding that saved searches store queries, not data snapshots.

#2 Adding complex scripted filters without considering performance impact.

Wrong approach:

Use script filters in saved searches without testing speed.

Correct approach:

Use simple term or range filters and test performance before adding scripts.

Root cause:

Lack of awareness about filter caching and execution cost.

#3 Creating many similar saved searches instead of reusing and layering filters.

Wrong approach:

Save multiple searches differing only by small filter changes.

Correct approach:

Save one base search and apply filters dynamically as needed.

Root cause:

Not knowing filters can be added or changed without editing saved searches.

Key Takeaways

Saved searches store the full query and filter setup, not the actual data results.

Filters narrow down search results efficiently and can be layered on saved searches for flexibility.

Filter caching in Elasticsearch speeds up repeated queries but depends on filter type.

Saved searches enable collaboration by sharing consistent queries across teams and dashboards.

Understanding saved searches as live queries helps avoid confusion about changing results.