Wildcard and prefix queries in Elasticsearch - Deep Dive

Overview - Wildcard and prefix queries

What is it?

Wildcard and prefix queries are ways to search text in Elasticsearch when you don't know the exact word or want to find words starting with certain letters. Wildcard queries let you use special symbols like * or ? to match parts of words. Prefix queries find all words that begin with a specific set of letters. These help find data even if you only remember part of a word or want to include many similar words.

Why it matters

Without wildcard and prefix queries, searching would require exact matches, making it hard to find information if you misspell words or only know part of them. These queries make search flexible and user-friendly, helping people find what they need quickly, like searching for all products starting with 'phone' or names containing 'ann'.

Where it fits

Before learning these queries, you should understand basic Elasticsearch search concepts like term and match queries. After mastering wildcard and prefix queries, you can explore more advanced search features like fuzzy queries, regex queries, and performance tuning for large datasets.

Mental Model

Core Idea

Wildcard and prefix queries let you search for words by matching patterns or beginnings instead of exact full words.

Think of it like...

It's like using a treasure map with clues: instead of knowing the exact spot, you know part of the location or a pattern, so you look around all places that fit that hint.

Search Query Types
┌─────────────────────────────┐
│       Search Input           │
└─────────────┬───────────────┘
              │
      ┌───────┴────────┐
      │                │
Wildcard Query     Prefix Query
(uses * and ?)     (uses starting letters)
      │                │
Matches words like  Matches words like
"ca*" → cat, car   "pre" → prefix, prepare
"te?t" → test, text

Build-Up - 7 Steps

1

Foundation

Basic concept of wildcard queries

Concept: Introduce wildcard queries and their special symbols * and ? for flexible matching.

Wildcard queries use * to match any number of characters and ? to match exactly one character. For example, searching 'ca*' finds 'cat', 'car', 'camera'. Searching 'te?t' finds 'test' and 'text'. This helps when you only know part of a word or want to include variations.

Result

A search for 'ca*' returns documents containing words like 'cat', 'car', 'castle'.

Understanding wildcard symbols lets you search beyond exact words, making search more forgiving and powerful.

2

Foundation

Basic concept of prefix queries

Concept: Introduce prefix queries that find words starting with a given string.

Prefix queries look for words that start with the letters you provide. For example, a prefix query for 'pre' finds 'prefix', 'prepare', 'prevent'. It is simpler than wildcard queries and faster because it only matches beginnings.

Result

A prefix query for 'pre' returns documents with words like 'prefix', 'prepare', 'prevent'.

Knowing prefix queries helps you quickly find all words starting with certain letters, useful for autocomplete or filtering.

3

Intermediate

Syntax and usage in Elasticsearch

🤔Before reading on: do you think wildcard and prefix queries use the same syntax in Elasticsearch? Commit to your answer.

Concept: Learn the exact JSON syntax to write wildcard and prefix queries in Elasticsearch.

In Elasticsearch, a wildcard query looks like: { "wildcard": { "field": "ca*" } } A prefix query looks like: { "prefix": { "field": "pre" } } You specify the field to search and the pattern or prefix. Wildcard queries allow * and ?, prefix queries only need the starting string.

Result

Queries run and return matching documents based on the pattern or prefix.

Knowing the syntax is essential to use these queries correctly and avoid errors.

4

Intermediate

Performance considerations of wildcard queries

🤔Before reading on: do you think wildcard queries with * at the start are fast or slow? Commit to your answer.

Concept: Understand how wildcard queries affect search speed and why some patterns are slower.

Wildcard queries that start with * (like '*ing') are slow because Elasticsearch must scan many terms. Queries with * only at the end (like 'pre*') are faster. Prefix queries are optimized for speed. Using wildcard queries carefully improves performance.

Result

Queries with leading * run slower and can impact system responsiveness.

Knowing performance impacts helps you write efficient queries and avoid slow searches.

5

Intermediate

Combining wildcard and prefix queries

Concept: Learn how to combine these queries with others for flexible search.

You can combine wildcard or prefix queries with other query types using bool queries. For example, find documents where a field starts with 'pre' and another field contains 'te?t'. This allows complex search conditions.

Result

Combined queries return documents matching all conditions, improving search precision.

Combining queries lets you build powerful searches tailored to user needs.

6

Advanced

Limitations and alternatives to wildcard queries

🤔Before reading on: do you think wildcard queries can replace fuzzy or regex queries? Commit to your answer.

Concept: Explore when wildcard queries are not enough and what other query types exist.

Wildcard queries do not handle typos well and can be slow with complex patterns. Fuzzy queries find words with small spelling mistakes. Regex queries allow more complex patterns but are slower. Choosing the right query depends on your needs.

Result

Using the right query type improves search accuracy and speed.

Knowing query limits prevents misuse and guides better search design.

7

Expert

Internal indexing and query execution details

🤔Before reading on: do you think wildcard queries search raw text or indexed terms? Commit to your answer.

Concept: Understand how Elasticsearch indexes terms and executes wildcard and prefix queries internally.

Elasticsearch indexes text by breaking it into terms. Wildcard and prefix queries search these terms, not raw text. Prefix queries use efficient data structures like prefix trees. Wildcard queries with leading * require scanning many terms, causing slowdowns. Understanding this helps optimize queries and index design.

Result

Queries run faster or slower depending on how they interact with the index structure.

Knowing internal mechanics helps experts write performant queries and design indexes for search speed.

Under the Hood

Elasticsearch stores text fields as inverted indexes, mapping terms to documents. Prefix queries use a prefix tree (trie) to quickly find all terms starting with the prefix. Wildcard queries translate patterns into term expansions. If the wildcard starts with *, Elasticsearch must scan many terms, which is slow. Queries operate on indexed terms, not raw text, so analysis affects results.

Why designed this way?

This design balances search speed and flexibility. Prefix queries are optimized for fast lookups using tries. Wildcard queries offer flexible pattern matching but at a cost. Alternatives like regex or fuzzy queries exist but are slower or more complex. The tradeoff allows users to choose based on their needs.

Inverted Index Structure
┌───────────────┐
│  Documents    │
└──────┬────────┘
       │
       ▼
┌───────────────┐
│  Text Fields  │
└──────┬────────┘
       │ Tokenize & Analyze
       ▼
┌───────────────┐
│  Terms Index  │
│ (Inverted)    │
└──────┬────────┘
       │
       ▼
┌─────────────────────────────┐
│ Prefix Query: Uses Trie Tree │
│ Wildcard Query: Expands * ? │
│ Leading * scans many terms  │
└─────────────────────────────┘

Myth Busters - 4 Common Misconceptions

Quick: Do wildcard queries with * at the start run fast or slow? Commit to your answer.

Common Belief: Wildcard queries always run fast because they just match patterns.

Reality: Wildcard queries with * at the start are slow because Elasticsearch must scan many terms in the index.

Why it matters: Using leading * wildcard queries in production can cause slow searches and overload the system.

Quick: Does a prefix query match anywhere in the word or only at the start? Commit to your answer.

Common Belief: Prefix queries match the pattern anywhere inside the word.

Reality: Prefix queries only match words starting with the given prefix, not inside or at the end.

Why it matters: Misunderstanding this leads to wrong query design and missed search results.

Quick: Can wildcard queries handle typos like fuzzy queries? Commit to your answer.

Common Belief: Wildcard queries can find words with spelling mistakes like fuzzy queries.

Reality: Wildcard queries do not handle typos; they only match patterns literally.

Why it matters: Relying on wildcard queries for typo tolerance results in poor user search experience.

Quick: Do wildcard and prefix queries search raw text or indexed terms? Commit to your answer.

Common Belief: They search the raw text directly.

Reality: They search the indexed terms created by analyzing the text, so analysis affects results.

Why it matters: Ignoring analysis can cause unexpected search results or misses.

Expert Zone

1

Wildcard queries with multiple * and ? can explode the number of term expansions, causing heavy load.

2

Prefix queries are case-sensitive depending on the analyzer used; choosing the right analyzer affects results.

3

Using keyword fields for wildcard queries avoids tokenization issues but limits flexibility.

When NOT to use

Avoid wildcard queries with leading * on large datasets; use prefix queries or n-gram indexing instead. For typo tolerance, use fuzzy queries. For complex patterns, use regex queries but expect slower performance.

Production Patterns

In production, prefix queries power autocomplete features. Wildcard queries are used sparingly for admin tools or small datasets. Combining prefix queries with filters improves speed. Index design often includes keyword subfields to support wildcard queries efficiently.

Connections

Regular expressions

Wildcard queries are a simpler form of pattern matching compared to regex.

Understanding regex helps grasp the power and limits of wildcard queries and when to upgrade to regex for complex patterns.

Trie data structure

Prefix queries use tries internally to quickly find terms starting with a prefix.

Knowing tries explains why prefix queries are fast and how indexing supports efficient search.

Autocompletion in user interfaces

Prefix queries are often the backend mechanism for autocomplete suggestions.

Understanding prefix queries helps design responsive and relevant autocomplete features.

Common Pitfalls

#1 Using wildcard queries with * at the start on large fields.

Wrong approach:

{ "wildcard": { "field": "*ing" } }

Correct approach:

{ "prefix": { "field": "ing" } }

Root cause:

Misunderstanding that leading * causes slow scans and that prefix queries only match starts.

#2 Expecting prefix queries to match substrings inside words.

Wrong approach:

{ "prefix": { "field": "fix" } } // expects to match 'prefix'

Correct approach:

{ "wildcard": { "field": "*fix" } }

Root cause:

Confusing prefix queries with substring matching.

#3 Using wildcard queries on analyzed text fields without keyword subfields.

Wrong approach:

{ "wildcard": { "text_field": "ca*" } }

Correct approach:

{ "wildcard": { "text_field.keyword": "ca*" } }

Root cause:

Not realizing analyzed fields break text into tokens, making wildcard queries ineffective.

Key Takeaways

Wildcard queries use * and ? to match flexible patterns but can be slow if used with leading *.

Prefix queries find words starting with given letters and are faster and simpler than wildcard queries.

Both queries operate on indexed terms, so text analysis affects their behavior and results.

Choosing the right query type and understanding performance impacts is key to efficient search.

Combining these queries with others enables powerful, user-friendly search experiences.

Practice

1. What does a wildcard query with * symbol do in Elasticsearch?

easy

A. Matches terms ending with a specific suffix

B. Matches only the exact search term

C. Matches terms starting with a specific prefix

D. Matches any sequence of characters in the search term

Solution

1. Step 1: Understand the wildcard symbol *

   The * symbol in wildcard queries matches zero or more characters in any position of the term.

2. Step 2: Compare with other options

   Exact match requires no wildcard, prefix queries use a different query type, and suffix matching is not directly done by * alone.

3. Final Answer:

   Matches any sequence of characters in the search term -> Option D

4. Quick Check:

   wildcard * = match any chars [OK]

Hint: Remember * matches any characters, ? matches one character [OK]

Common Mistakes:

• Confusing wildcard * with prefix query
• Thinking * matches only one character
• Assuming wildcard matches exact terms only

2. Which of the following is the correct syntax for a prefix query in Elasticsearch?

easy

A. { "prefix": { "field_name": "pre" } }

B. { "wildcard": { "field_name": "pre*" } }

C. { "match": { "field_name": "pre*" } }

D. { "term": { "field_name": "pre*" } }

Solution

1. Step 1: Identify prefix query syntax

   Prefix queries use the prefix keyword with the field and the prefix string.

2. Step 2: Check other options

   Wildcard queries use wildcard, match and term queries do not support prefix syntax with asterisk.

3. Final Answer:

   { "prefix": { "field_name": "pre" } } -> Option A

4. Quick Check:

   prefix query uses "prefix" keyword [OK]

Hint: Prefix queries use "prefix" keyword, not wildcard or match [OK]

Common Mistakes:

• Using wildcard syntax for prefix query
• Using match query with asterisk
• Confusing term query with prefix query

3. Given the following wildcard query, which documents will match?

{ "query": { "wildcard": { "name": "jo?n*" } } }

Assuming documents have names: "john", "joan", "jonas", "joanathan", "jo".

medium

A. ["joan", "jonas", "joanathan"]

B. ["john", "joan", "jo"]

C. ["john", "joan", "joanathan"]

D. ["john", "joan", "jonas", "joanathan", "jo"]

Solution

1. Step 1: Analyze the wildcard pattern "jo?n*"

   The pattern means: 'jo' followed by any single character (due to '?'), then 'n', then any characters (due to '*').

2. Step 2: Check each name against the pattern

   "john" matches ('h' replaces '?'), "joan" matches ('a' replaces '?'), "joanathan" matches ('a' replaces '?'). "jonas" does not match ('n' replaces '?' but next 'a' ≠ 'n'). "jo" is too short to match.

3. Final Answer:

   ["john", "joan", "joanathan"] -> Option C

4. Quick Check:

   wildcard ? = one char, * = any chars [OK]

Hint: ? matches one char, * matches zero or more chars [OK]

Common Mistakes:

• Assuming ? matches zero characters
• Including too short strings like "jo"
• Confusing wildcard with prefix query

4. You wrote this prefix query but it returns no results:

{ "query": { "prefix": { "title": "prog*" } } }

What is the error?

medium

A. Field name "title" is invalid

B. Prefix query should not include the asterisk (*)

C. Prefix queries require wildcard keyword

D. Prefix query must use match instead of prefix

Solution

1. Step 1: Understand prefix query syntax

   Prefix queries expect a string prefix without wildcards like '*'. The asterisk is not needed and causes no matches.

2. Step 2: Check other options

   Field name is assumed valid, wildcard keyword is different query type, and match query is unrelated to prefix syntax.

3. Final Answer:

   Prefix query should not include the asterisk (*) -> Option B

4. Quick Check:

   Prefix query uses plain prefix string, no * [OK]

Hint: Remove * from prefix query string to fix [OK]

Common Mistakes:

• Including * in prefix query string
• Confusing prefix with wildcard query
• Using match query instead of prefix

5. You want to find all documents where the "username" field starts with "alex" but also want to allow any characters after "alex" including none. Which query is best and why?

{ "query": { "wildcard": { "username": "alex*" } } }

or

{ "query": { "prefix": { "username": "alex" } } }

hard

A. Prefix query is better because it is faster and matches all starting with "alex"

B. Wildcard query is better because it matches "alex" exactly only

C. Wildcard query is better because it matches any characters including none after "alex"

D. Prefix query is better because it requires the * symbol

Solution

1. Step 1: Compare prefix and wildcard queries for this case

   Both queries match terms starting with "alex" and any characters after. Wildcard uses "alex*", prefix uses "alex" without *.

2. Step 2: Consider performance and usage

   Prefix queries are optimized and faster for this use case. Wildcard queries are slower and more flexible but unnecessary here.

3. Final Answer:

   Prefix query is better because it is faster and matches all starting with "alex" -> Option A

4. Quick Check:

   Prefix query faster for starts-with matching [OK]

Hint: Use prefix query for starts-with, wildcard for complex patterns [OK]

Common Mistakes:

• Using wildcard when prefix is enough
• Adding * in prefix query string
• Thinking wildcard matches exact only