Wildcard and prefix queries in Elasticsearch - Deep Dive

Overview - Wildcard and prefix queries

What is it?

Wildcard and prefix queries are ways to search text in Elasticsearch when you don't know the exact word or want to find words starting with certain letters. Wildcard queries let you use special symbols like * or ? to match parts of words. Prefix queries find all words that begin with a specific set of letters. These help find data even if you only remember part of a word or want to include many similar words.

Why it matters

Without wildcard and prefix queries, searching would require exact matches, making it hard to find information if you misspell words or only know part of them. These queries make search flexible and user-friendly, helping people find what they need quickly, like searching for all products starting with 'phone' or names containing 'ann'.

Where it fits

Before learning these queries, you should understand basic Elasticsearch search concepts like term and match queries. After mastering wildcard and prefix queries, you can explore more advanced search features like fuzzy queries, regex queries, and performance tuning for large datasets.

Mental Model

Core Idea

Wildcard and prefix queries let you search for words by matching patterns or beginnings instead of exact full words.

Think of it like...

It's like using a treasure map with clues: instead of knowing the exact spot, you know part of the location or a pattern, so you look around all places that fit that hint.

Search Query Types
┌─────────────────────────────┐
│       Search Input           │
└─────────────┬───────────────┘
              │
      ┌───────┴────────┐
      │                │
Wildcard Query     Prefix Query
(uses * and ?)     (uses starting letters)
      │                │
Matches words like  Matches words like
"ca*" → cat, car   "pre" → prefix, prepare
"te?t" → test, text

Build-Up - 7 Steps

1

Foundation

Basic concept of wildcard queries

Concept: Introduce wildcard queries and their special symbols * and ? for flexible matching.

Wildcard queries use * to match any number of characters and ? to match exactly one character. For example, searching 'ca*' finds 'cat', 'car', 'camera'. Searching 'te?t' finds 'test' and 'text'. This helps when you only know part of a word or want to include variations.

Result

A search for 'ca*' returns documents containing words like 'cat', 'car', 'castle'.

Understanding wildcard symbols lets you search beyond exact words, making search more forgiving and powerful.

2

Foundation

Basic concept of prefix queries

Concept: Introduce prefix queries that find words starting with a given string.

Prefix queries look for words that start with the letters you provide. For example, a prefix query for 'pre' finds 'prefix', 'prepare', 'prevent'. It is simpler than wildcard queries and faster because it only matches beginnings.

Result

A prefix query for 'pre' returns documents with words like 'prefix', 'prepare', 'prevent'.

Knowing prefix queries helps you quickly find all words starting with certain letters, useful for autocomplete or filtering.

3

Intermediate

Syntax and usage in Elasticsearch

🤔Before reading on: do you think wildcard and prefix queries use the same syntax in Elasticsearch? Commit to your answer.

Concept: Learn the exact JSON syntax to write wildcard and prefix queries in Elasticsearch.

In Elasticsearch, a wildcard query looks like: { "wildcard": { "field": "ca*" } } A prefix query looks like: { "prefix": { "field": "pre" } } You specify the field to search and the pattern or prefix. Wildcard queries allow * and ?, prefix queries only need the starting string.

Result

Queries run and return matching documents based on the pattern or prefix.

Knowing the syntax is essential to use these queries correctly and avoid errors.

4

Intermediate

Performance considerations of wildcard queries

🤔Before reading on: do you think wildcard queries with * at the start are fast or slow? Commit to your answer.

Concept: Understand how wildcard queries affect search speed and why some patterns are slower.

Wildcard queries that start with * (like '*ing') are slow because Elasticsearch must scan many terms. Queries with * only at the end (like 'pre*') are faster. Prefix queries are optimized for speed. Using wildcard queries carefully improves performance.

Result

Queries with leading * run slower and can impact system responsiveness.

Knowing performance impacts helps you write efficient queries and avoid slow searches.

5

Intermediate

Combining wildcard and prefix queries

Concept: Learn how to combine these queries with others for flexible search.

You can combine wildcard or prefix queries with other query types using bool queries. For example, find documents where a field starts with 'pre' and another field contains 'te?t'. This allows complex search conditions.

Result

Combined queries return documents matching all conditions, improving search precision.

Combining queries lets you build powerful searches tailored to user needs.

6

Advanced

Limitations and alternatives to wildcard queries

🤔Before reading on: do you think wildcard queries can replace fuzzy or regex queries? Commit to your answer.

Concept: Explore when wildcard queries are not enough and what other query types exist.

Wildcard queries do not handle typos well and can be slow with complex patterns. Fuzzy queries find words with small spelling mistakes. Regex queries allow more complex patterns but are slower. Choosing the right query depends on your needs.

Result

Using the right query type improves search accuracy and speed.

Knowing query limits prevents misuse and guides better search design.

7

Expert

Internal indexing and query execution details

🤔Before reading on: do you think wildcard queries search raw text or indexed terms? Commit to your answer.

Concept: Understand how Elasticsearch indexes terms and executes wildcard and prefix queries internally.

Elasticsearch indexes text by breaking it into terms. Wildcard and prefix queries search these terms, not raw text. Prefix queries use efficient data structures like prefix trees. Wildcard queries with leading * require scanning many terms, causing slowdowns. Understanding this helps optimize queries and index design.

Result

Queries run faster or slower depending on how they interact with the index structure.

Knowing internal mechanics helps experts write performant queries and design indexes for search speed.

Under the Hood

Elasticsearch stores text fields as inverted indexes, mapping terms to documents. Prefix queries use a prefix tree (trie) to quickly find all terms starting with the prefix. Wildcard queries translate patterns into term expansions. If the wildcard starts with *, Elasticsearch must scan many terms, which is slow. Queries operate on indexed terms, not raw text, so analysis affects results.

Why designed this way?

This design balances search speed and flexibility. Prefix queries are optimized for fast lookups using tries. Wildcard queries offer flexible pattern matching but at a cost. Alternatives like regex or fuzzy queries exist but are slower or more complex. The tradeoff allows users to choose based on their needs.

Inverted Index Structure
┌───────────────┐
│  Documents    │
└──────┬────────┘
       │
       ▼
┌───────────────┐
│  Text Fields  │
└──────┬────────┘
       │ Tokenize & Analyze
       ▼
┌───────────────┐
│  Terms Index  │
│ (Inverted)    │
└──────┬────────┘
       │
       ▼
┌─────────────────────────────┐
│ Prefix Query: Uses Trie Tree │
│ Wildcard Query: Expands * ? │
│ Leading * scans many terms  │
└─────────────────────────────┘

Myth Busters - 4 Common Misconceptions

Quick: Do wildcard queries with * at the start run fast or slow? Commit to your answer.

Common Belief: Wildcard queries always run fast because they just match patterns.

Reality: Wildcard queries with * at the start are slow because Elasticsearch must scan many terms in the index.

Why it matters: Using leading * wildcard queries in production can cause slow searches and overload the system.

Quick: Does a prefix query match anywhere in the word or only at the start? Commit to your answer.

Common Belief: Prefix queries match the pattern anywhere inside the word.

Reality: Prefix queries only match words starting with the given prefix, not inside or at the end.

Why it matters: Misunderstanding this leads to wrong query design and missed search results.

Quick: Can wildcard queries handle typos like fuzzy queries? Commit to your answer.

Common Belief: Wildcard queries can find words with spelling mistakes like fuzzy queries.

Reality: Wildcard queries do not handle typos; they only match patterns literally.

Why it matters: Relying on wildcard queries for typo tolerance results in poor user search experience.

Quick: Do wildcard and prefix queries search raw text or indexed terms? Commit to your answer.

Common Belief: They search the raw text directly.

Reality: They search the indexed terms created by analyzing the text, so analysis affects results.

Why it matters: Ignoring analysis can cause unexpected search results or misses.

Expert Zone

1

Wildcard queries with multiple * and ? can explode the number of term expansions, causing heavy load.

2

Prefix queries are case-sensitive depending on the analyzer used; choosing the right analyzer affects results.

3

Using keyword fields for wildcard queries avoids tokenization issues but limits flexibility.

When NOT to use

Avoid wildcard queries with leading * on large datasets; use prefix queries or n-gram indexing instead. For typo tolerance, use fuzzy queries. For complex patterns, use regex queries but expect slower performance.

Production Patterns

In production, prefix queries power autocomplete features. Wildcard queries are used sparingly for admin tools or small datasets. Combining prefix queries with filters improves speed. Index design often includes keyword subfields to support wildcard queries efficiently.

Connections

Regular expressions

Wildcard queries are a simpler form of pattern matching compared to regex.

Understanding regex helps grasp the power and limits of wildcard queries and when to upgrade to regex for complex patterns.

Trie data structure

Prefix queries use tries internally to quickly find terms starting with a prefix.

Knowing tries explains why prefix queries are fast and how indexing supports efficient search.

Autocompletion in user interfaces

Prefix queries are often the backend mechanism for autocomplete suggestions.

Understanding prefix queries helps design responsive and relevant autocomplete features.

Common Pitfalls

#1 Using wildcard queries with * at the start on large fields.

Wrong approach:

{ "wildcard": { "field": "*ing" } }

Correct approach:

{ "prefix": { "field": "ing" } }

Root cause:

Misunderstanding that leading * causes slow scans and that prefix queries only match starts.

#2 Expecting prefix queries to match substrings inside words.

Wrong approach:

{ "prefix": { "field": "fix" } } // expects to match 'prefix'

Correct approach:

{ "wildcard": { "field": "*fix" } }

Root cause:

Confusing prefix queries with substring matching.

#3 Using wildcard queries on analyzed text fields without keyword subfields.

Wrong approach:

{ "wildcard": { "text_field": "ca*" } }

Correct approach:

{ "wildcard": { "text_field.keyword": "ca*" } }

Root cause:

Not realizing analyzed fields break text into tokens, making wildcard queries ineffective.

Key Takeaways

Wildcard queries use * and ? to match flexible patterns but can be slow if used with leading *.

Prefix queries find words starting with given letters and are faster and simpler than wildcard queries.

Both queries operate on indexed terms, so text analysis affects their behavior and results.

Choosing the right query type and understanding performance impacts is key to efficient search.

Combining these queries with others enables powerful, user-friendly search experiences.