Autocomplete with edge n-gram in Elasticsearch - Step-by-Step Execution

Concept Flow - Autocomplete with edge n-gram

User types prefix

↓

Elasticsearch receives query

↓

Search edge n-gram tokens

↓

Match tokens starting with prefix

↓

Return autocomplete suggestions

↓

User sees suggestions

When a user types a prefix, Elasticsearch searches tokens created by edge n-gram to find matching suggestions starting with that prefix.

Execution Sample

Elasticsearch

PUT /autocomplete_example
{
  "settings": {
    "analysis": {
      "analyzer": {
        "autocomplete_analyzer": {
          "tokenizer": "autocomplete_tokenizer"
        }
      },
      "tokenizer": {
        "autocomplete_tokenizer": {
          "type": "edge_ngram",
          "min_gram": 1,
          "max_gram": 10,
          "token_chars": ["letter"]
        }
      }
    }
  },
  "mappings": {
    "properties": {
      "name": {
        "type": "text",
        "analyzer": "autocomplete_analyzer",
        "search_analyzer": "standard"
      }
    }
  }
}

GET /autocomplete_example/_search
{
  "query": {
    "match": {
      "name": "mic"
    }
  }
}

This code creates an index with an edge n-gram tokenizer for autocomplete on the 'name' field, then searches for prefix 'mic'.

Execution Table

Step	Action	Input Text	Tokens Created	Query Tokens	Matched Tokens	Output Suggestions
1	Index document	michael	["m", "mi", "mic", "mich", "micha", "michae", "michael"]	-	-	-
2	Index document	michelle	["m", "mi", "mic", "mich", "miche", "michel", "michell", "michelle"]	-	-	-
3	User types prefix	mic	-	-	["mic"]	-
4	Search edge n-gram tokens	-	-	["mic"]	["mic"]	-
5	Match tokens starting with prefix	-	-	["mic"]	["mic"]	-
6	Return autocomplete suggestions	-	-	-	-	["michael", "michelle"]
7	User sees suggestions	-	-	-	-	["michael", "michelle"]

💡 Autocomplete suggestions returned when prefix 'mic' matches edge n-gram tokens.

Variable Tracker

Variable	Start	After Step 1	After Step 2	After Step 3	After Step 4	After Step 6	Final
Input Text	-	michael	michelle	mic	mic	mic	mic
Tokens Created	-	["m", "mi", "mic", "mich", "micha", "michae", "michael"]	["m", "mi", "mic", "mich", "miche", "michel", "michell", "michelle"]	-	-	-	-
Query Tokens	-	-	-	["mic"]	["mic"]	["mic"]	["mic"]
Matched Tokens	-	-	-	-	["mic"]	["mic"]	["mic"]
Output Suggestions	-	-	-	-	-	["michael", "michelle"]	["michael", "michelle"]

Key Moments - 3 Insights

Why do we create multiple tokens like "m", "mi", "mic" for a single word?

Because edge n-gram tokenizer breaks words into prefixes of increasing length to match user input prefixes, as shown in execution_table rows 1 and 2.

Why is the search analyzer set to standard instead of autocomplete_analyzer?

To avoid breaking the search query into n-grams and instead search the full prefix typed by the user, matching the edge n-gram tokens indexed (see execution_table row 3).

What happens if the user types a prefix longer than max_gram?

Tokens are only created up to max_gram length, so longer prefixes won't match beyond that length, limiting autocomplete suggestions (max_gram is 10 here).

Visual Quiz - 3 Questions

Look at the execution_table at Step 1. What tokens are created for the word "michael"?

A ["michael"]

B ["m", "mi", "mic", "mich", "micha", "michae", "michael"]

C ["mic", "mich", "micha"]

D ["m", "mi", "michelle"]

💡 Hint

Check the Tokens Created column at Step 1 in execution_table.

At which step does Elasticsearch match the user query tokens with the indexed tokens?

A Step 3

B Step 6

C Step 4

D Step 2

💡 Hint

Look for the step where 'Matched Tokens' column is filled in execution_table.

If we change max_gram to 3, how would the tokens for "michelle" change at Step 2?

A ["m", "mi", "mic"]

B ["michelle"]

C ["m", "mi", "mic", "mich"]

D No tokens would be created

💡 Hint

Tokens are created up to max_gram length, see Tokens Created column at Step 2.

Concept Snapshot

Autocomplete with edge n-gram:
- Use edge_ngram tokenizer to create prefix tokens
- Index tokens like 'm', 'mi', 'mic' for words
- Search uses standard analyzer for full prefix
- Matches tokens starting with user input
- Returns suggestions starting with typed prefix

Full Transcript

This visual trace shows how Elasticsearch uses edge n-gram tokenizer to support autocomplete. When indexing, words like 'michael' are broken into prefix tokens such as 'm', 'mi', 'mic', etc. When a user types a prefix like 'mic', Elasticsearch searches for tokens starting with 'mic' and returns matching suggestions like 'michael' and 'michelle'. The search analyzer is standard to keep the query as a full prefix. The execution table tracks each step from indexing to returning suggestions, and the variable tracker shows how tokens and queries evolve. Key moments clarify why multiple tokens are created and why the search analyzer differs. The quiz tests understanding of token creation and matching steps.

Practice

1. What is the main purpose of using an edge_ngram tokenizer in Elasticsearch autocomplete?

easy

A. To break words into prefixes for fast autocomplete suggestions

B. To sort search results alphabetically

C. To remove stop words from the search query

D. To index only complete words without partial matches

Solution

1. Step 1: Understand edge n-gram tokenizer function

   The edge n-gram tokenizer splits words into prefixes, like 'app', 'appl', 'apple' for 'apple'.

2. Step 2: Relate tokenizer to autocomplete

   This prefix splitting allows Elasticsearch to quickly suggest words as users type partial input.

3. Final Answer:

   To break words into prefixes for fast autocomplete suggestions -> Option A

4. Quick Check:

   edge_ngram tokenizer = prefix splitting [OK]

Hint: Edge n-gram creates prefixes for autocomplete speed [OK]

Common Mistakes:

• Thinking edge n-gram sorts results
• Confusing with stop word removal
• Assuming it indexes only full words

2. Which of the following is the correct way to define an edge_ngram tokenizer in Elasticsearch settings?

easy

A. {
"tokenizer": {
"my_edge_ngram": {
"type": "edge_ngram",
"min_gram": 2,
"max_gram": 5,
"token_chars": ["letter"]
}
}
}

B. {
"tokenizer": {
"my_edge_ngram": {
"type": "standard",
"min_gram": 2,
"max_gram": 5
}
}
}

C. {
"analyzer": {
"my_edge_ngram": {
"type": "edge_ngram"
}
}
}

D. {
"tokenizer": {
"my_edge_ngram": {
"type": "ngram",
"min_gram": 1,
"max_gram": 3
}
}
}

Solution

1. Step 1: Identify correct tokenizer type and parameters

   The edge_ngram tokenizer requires type 'edge_ngram' with min_gram and max_gram settings.

2. Step 2: Check token_chars usage

   { "tokenizer": { "my_edge_ngram": { "type": "edge_ngram", "min_gram": 2, "max_gram": 5, "token_chars": ["letter"] } } } correctly sets token_chars to 'letter' to include only letters in tokens.

3. Final Answer:

   Option A with type 'edge_ngram' and proper min_gram, max_gram -> Option A

4. Quick Check:

   edge_ngram tokenizer syntax = { "tokenizer": { "my_edge_ngram": { "type": "edge_ngram", "min_gram": 2, "max_gram": 5, "token_chars": ["letter"] } } } [OK]

Hint: Look for type 'edge_ngram' with min_gram and max_gram [OK]

Common Mistakes:

• Using 'standard' type instead of 'edge_ngram'
• Defining edge_ngram under analyzer instead of tokenizer
• Confusing ngram with edge_ngram tokenizer

3. Given this analyzer configuration using edge n-gram tokenizer: { "settings": { "analysis": { "tokenizer": { "autocomplete_tokenizer": { "type": "edge_ngram", "min_gram": 1, "max_gram": 3, "token_chars": ["letter"] } }, "analyzer": { "autocomplete": { "type": "custom", "tokenizer": "autocomplete_tokenizer", "filter": ["lowercase"] } } } }} What tokens will be generated for the word "cat" during indexing?

medium

A. ["cat"]

B. ["c", "a", "t"]

C. ["ca", "cat", "cat"]

D. ["c", "ca", "cat"]

Solution

1. Step 1: Understand edge n-gram tokenizer with min_gram=1 and max_gram=3

   It creates tokens from the start of the word with lengths 1 to 3 letters.

2. Step 2: Generate tokens for 'cat'

   Tokens are 'c' (1 letter), 'ca' (2 letters), and 'cat' (3 letters).

3. Final Answer:

   ["c", "ca", "cat"] -> Option D

4. Quick Check:

   edge n-gram tokens for 'cat' = ["c", "ca", "cat"] [OK]

Hint: Edge n-gram tokens start from first letter up to max_gram [OK]

Common Mistakes:

• Assuming only full word token is generated
• Including tokens from middle or end of word
• Confusing n-gram with edge n-gram tokens

4. You have this analyzer configuration for autocomplete but your search results do not return expected matches: { "settings": { "analysis": { "tokenizer": { "autocomplete_tokenizer": { "type": "edge_ngram", "min_gram": 2, "max_gram": 5 } }, "analyzer": { "autocomplete": { "type": "custom", "tokenizer": "autocomplete_tokenizer", "filter": ["lowercase"] }, "search_analyzer": { "type": "standard" } } } }} What is the likely cause of the problem?

medium

A. The tokenizer type should be 'ngram' not 'edge_ngram'

B. The search analyzer is incorrectly defined as 'standard' instead of using the same analyzer

C. The min_gram value is too low causing too many tokens

D. The filter 'lowercase' should not be used in the analyzer

Solution

1. Step 1: Understand analyzer and search_analyzer roles

   Indexing uses edge n-gram tokenizer, but searching uses standard analyzer which does not generate prefixes.

2. Step 2: Identify mismatch causing no autocomplete matches

   Because search analyzer does not tokenize input into prefixes, queries won't match edge n-gram tokens.

3. Final Answer:

   Search analyzer should match or complement edge n-gram analyzer, not standard -> Option B

4. Quick Check:

   Analyzer mismatch = The search analyzer is incorrectly defined as 'standard' instead of using the same analyzer [OK]

Hint: Search analyzer must complement edge n-gram tokenizer [OK]

Common Mistakes:

• Changing min_gram instead of fixing analyzer mismatch
• Confusing ngram and edge_ngram tokenizer types
• Removing lowercase filter unnecessarily

5. You want to implement autocomplete for a product search using edge n-gram tokenizer. Which approach correctly combines indexing and searching analyzers for best autocomplete results?

hard

A. Use edge n-gram tokenizer in the indexing analyzer and standard analyzer in the search analyzer

B. Use edge n-gram tokenizer in both indexing and search analyzers

C. Use edge n-gram tokenizer in indexing analyzer and a lowercase + standard tokenizer in search analyzer

D. Use standard tokenizer in indexing analyzer and edge n-gram tokenizer in search analyzer

Solution

1. Step 1: Understand indexing with edge n-gram tokenizer

   Indexing uses edge n-gram tokenizer to create prefix tokens for autocomplete.

2. Step 2: Understand searching with standard tokenizer plus lowercase filter

   Search analyzer uses standard tokenizer to tokenize user input normally and lowercase filter to match indexed tokens.

3. Step 3: Why not use edge n-gram tokenizer in search analyzer?

   Using edge n-gram in search analyzer can cause unexpected partial matches and performance issues.

4. Final Answer:

   Index with edge n-gram, search with standard + lowercase analyzer -> Option C

5. Quick Check:

   Index edge n-gram + search standard = Use edge n-gram tokenizer in indexing analyzer and a lowercase + standard tokenizer in search analyzer [OK]

Hint: Index with edge n-gram, search with standard + lowercase [OK]

Common Mistakes:

• Using edge n-gram tokenizer in search analyzer
• Using standard tokenizer in indexing analyzer
• Not applying lowercase filter in search analyzer