Autocomplete with edge n-gram in Elasticsearch

Introduction

Autocomplete helps users find words or phrases quickly as they type. Edge n-gram breaks words into smaller parts from the start, making autocomplete fast and accurate.

When you want to suggest search terms as users type in a search box.

When you want to speed up search by matching partial beginnings of words.

When you want to support prefix matching for product names or tags.

When you want to improve user experience by showing instant suggestions.

When you want to handle large text fields but still provide quick autocomplete.

Syntax

Elasticsearch

{
  "settings": {
    "analysis": {
      "analyzer": {
        "autocomplete_analyzer": {
          "tokenizer": "autocomplete_tokenizer",
          "filter": ["lowercase"]
        }
      },
      "tokenizer": {
        "autocomplete_tokenizer": {
          "type": "edge_ngram",
          "min_gram": 1,
          "max_gram": 20,
          "token_chars": ["letter"]
        }
      }
    }
  },
  "mappings": {
    "properties": {
      "field_name": {
        "type": "text",
        "analyzer": "autocomplete_analyzer",
        "search_analyzer": "standard"
      }
    }
  }
}

The edge_ngram tokenizer breaks words into prefixes starting from the first letter.

Use search_analyzer as standard to avoid breaking search input into n-grams.

Examples

This example sets max_gram to 10, suitable for shorter autocomplete prefixes.

Elasticsearch

{
  "settings": {
    "analysis": {
      "analyzer": {
        "autocomplete_analyzer": {
          "tokenizer": "autocomplete_tokenizer",
          "filter": ["lowercase"]
        }
      },
      "tokenizer": {
        "autocomplete_tokenizer": {
          "type": "edge_ngram",
          "min_gram": 1,
          "max_gram": 10,
          "token_chars": ["letter"]
        }
      }
    }
  },
  "mappings": {
    "properties": {
      "product_name": {
        "type": "text",
        "analyzer": "autocomplete_analyzer",
        "search_analyzer": "standard"
      }
    }
  }
}

This example includes digits in tokens and starts from 2 characters to avoid too short prefixes.

Elasticsearch

{
  "settings": {
    "analysis": {
      "analyzer": {
        "autocomplete_analyzer": {
          "tokenizer": "autocomplete_tokenizer",
          "filter": ["lowercase"]
        }
      },
      "tokenizer": {
        "autocomplete_tokenizer": {
          "type": "edge_ngram",
          "min_gram": 2,
          "max_gram": 15,
          "token_chars": ["letter", "digit"]
        }
      }
    }
  },
  "mappings": {
    "properties": {
      "username": {
        "type": "text",
        "analyzer": "autocomplete_analyzer",
        "search_analyzer": "standard"
      }
    }
  }
}

Edge case: If the field is empty, no tokens are generated, so autocomplete returns no suggestions.

Elasticsearch

{
  "settings": {
    "analysis": {
      "analyzer": {
        "autocomplete_analyzer": {
          "tokenizer": "autocomplete_tokenizer",
          "filter": ["lowercase"]
        }
      },
      "tokenizer": {
        "autocomplete_tokenizer": {
          "type": "edge_ngram",
          "min_gram": 1,
          "max_gram": 20,
          "token_chars": ["letter"]
        }
      }
    }
  },
  "mappings": {
    "properties": {
      "empty_field": {
        "type": "text",
        "analyzer": "autocomplete_analyzer",
        "search_analyzer": "standard"
      }
    }
  }
}

Sample Program

This program creates an index with edge n-gram autocomplete on product_name. It adds three products and searches for "App" to get autocomplete suggestions starting with "App".

Elasticsearch

PUT /autocomplete_example
{
  "settings": {
    "analysis": {
      "analyzer": {
        "autocomplete_analyzer": {
          "tokenizer": "autocomplete_tokenizer",
          "filter": ["lowercase"]
        }
      },
      "tokenizer": {
        "autocomplete_tokenizer": {
          "type": "edge_ngram",
          "min_gram": 1,
          "max_gram": 10,
          "token_chars": ["letter"]
        }
      }
    }
  },
  "mappings": {
    "properties": {
      "product_name": {
        "type": "text",
        "analyzer": "autocomplete_analyzer",
        "search_analyzer": "standard"
      }
    }
  }
}

POST /autocomplete_example/_doc/1
{
  "product_name": "Apple MacBook Pro"
}

POST /autocomplete_example/_doc/2
{
  "product_name": "Apple Watch"
}

POST /autocomplete_example/_doc/3
{
  "product_name": "Samsung Galaxy"
}

GET /autocomplete_example/_search
{
  "query": {
    "match": {
      "product_name": "App"
    }
  }
}

Important Notes

Time complexity: Indexing is slower due to many tokens from edge n-grams; search is fast because of prefix matching.

Space complexity: Uses more disk space because each word creates multiple tokens.

Common mistake: Using edge n-gram analyzer for search input instead of standard analyzer causes poor search results.

Use edge n-gram for autocomplete prefix matching; use completion suggester for faster but less flexible autocomplete.

Summary

Edge n-gram tokenizer breaks words into prefixes for fast autocomplete.

Use a custom analyzer with edge n-gram tokenizer for indexing and standard analyzer for searching.

Autocomplete improves user experience by suggesting matches as users type.

Practice

1. What is the main purpose of using an edge_ngram tokenizer in Elasticsearch autocomplete?

easy

A. To break words into prefixes for fast autocomplete suggestions

B. To sort search results alphabetically

C. To remove stop words from the search query

D. To index only complete words without partial matches

Solution

1. Step 1: Understand edge n-gram tokenizer function

   The edge n-gram tokenizer splits words into prefixes, like 'app', 'appl', 'apple' for 'apple'.

2. Step 2: Relate tokenizer to autocomplete

   This prefix splitting allows Elasticsearch to quickly suggest words as users type partial input.

3. Final Answer:

   To break words into prefixes for fast autocomplete suggestions -> Option A

4. Quick Check:

   edge_ngram tokenizer = prefix splitting [OK]

Hint: Edge n-gram creates prefixes for autocomplete speed [OK]

Common Mistakes:

• Thinking edge n-gram sorts results
• Confusing with stop word removal
• Assuming it indexes only full words

2. Which of the following is the correct way to define an edge_ngram tokenizer in Elasticsearch settings?

easy

A. {
"tokenizer": {
"my_edge_ngram": {
"type": "edge_ngram",
"min_gram": 2,
"max_gram": 5,
"token_chars": ["letter"]
}
}
}

B. {
"tokenizer": {
"my_edge_ngram": {
"type": "standard",
"min_gram": 2,
"max_gram": 5
}
}
}

C. {
"analyzer": {
"my_edge_ngram": {
"type": "edge_ngram"
}
}
}

D. {
"tokenizer": {
"my_edge_ngram": {
"type": "ngram",
"min_gram": 1,
"max_gram": 3
}
}
}

Solution

1. Step 1: Identify correct tokenizer type and parameters

   The edge_ngram tokenizer requires type 'edge_ngram' with min_gram and max_gram settings.

2. Step 2: Check token_chars usage

   { "tokenizer": { "my_edge_ngram": { "type": "edge_ngram", "min_gram": 2, "max_gram": 5, "token_chars": ["letter"] } } } correctly sets token_chars to 'letter' to include only letters in tokens.

3. Final Answer:

   Option A with type 'edge_ngram' and proper min_gram, max_gram -> Option A

4. Quick Check:

   edge_ngram tokenizer syntax = { "tokenizer": { "my_edge_ngram": { "type": "edge_ngram", "min_gram": 2, "max_gram": 5, "token_chars": ["letter"] } } } [OK]

Hint: Look for type 'edge_ngram' with min_gram and max_gram [OK]

Common Mistakes:

• Using 'standard' type instead of 'edge_ngram'
• Defining edge_ngram under analyzer instead of tokenizer
• Confusing ngram with edge_ngram tokenizer

3. Given this analyzer configuration using edge n-gram tokenizer: { "settings": { "analysis": { "tokenizer": { "autocomplete_tokenizer": { "type": "edge_ngram", "min_gram": 1, "max_gram": 3, "token_chars": ["letter"] } }, "analyzer": { "autocomplete": { "type": "custom", "tokenizer": "autocomplete_tokenizer", "filter": ["lowercase"] } } } }} What tokens will be generated for the word "cat" during indexing?

medium

A. ["cat"]

B. ["c", "a", "t"]

C. ["ca", "cat", "cat"]

D. ["c", "ca", "cat"]

Solution

1. Step 1: Understand edge n-gram tokenizer with min_gram=1 and max_gram=3

   It creates tokens from the start of the word with lengths 1 to 3 letters.

2. Step 2: Generate tokens for 'cat'

   Tokens are 'c' (1 letter), 'ca' (2 letters), and 'cat' (3 letters).

3. Final Answer:

   ["c", "ca", "cat"] -> Option D

4. Quick Check:

   edge n-gram tokens for 'cat' = ["c", "ca", "cat"] [OK]

Hint: Edge n-gram tokens start from first letter up to max_gram [OK]

Common Mistakes:

• Assuming only full word token is generated
• Including tokens from middle or end of word
• Confusing n-gram with edge n-gram tokens

4. You have this analyzer configuration for autocomplete but your search results do not return expected matches: { "settings": { "analysis": { "tokenizer": { "autocomplete_tokenizer": { "type": "edge_ngram", "min_gram": 2, "max_gram": 5 } }, "analyzer": { "autocomplete": { "type": "custom", "tokenizer": "autocomplete_tokenizer", "filter": ["lowercase"] }, "search_analyzer": { "type": "standard" } } } }} What is the likely cause of the problem?

medium

A. The tokenizer type should be 'ngram' not 'edge_ngram'

B. The search analyzer is incorrectly defined as 'standard' instead of using the same analyzer

C. The min_gram value is too low causing too many tokens

D. The filter 'lowercase' should not be used in the analyzer

Solution

1. Step 1: Understand analyzer and search_analyzer roles

   Indexing uses edge n-gram tokenizer, but searching uses standard analyzer which does not generate prefixes.

2. Step 2: Identify mismatch causing no autocomplete matches

   Because search analyzer does not tokenize input into prefixes, queries won't match edge n-gram tokens.

3. Final Answer:

   Search analyzer should match or complement edge n-gram analyzer, not standard -> Option B

4. Quick Check:

   Analyzer mismatch = The search analyzer is incorrectly defined as 'standard' instead of using the same analyzer [OK]

Hint: Search analyzer must complement edge n-gram tokenizer [OK]

Common Mistakes:

• Changing min_gram instead of fixing analyzer mismatch
• Confusing ngram and edge_ngram tokenizer types
• Removing lowercase filter unnecessarily

5. You want to implement autocomplete for a product search using edge n-gram tokenizer. Which approach correctly combines indexing and searching analyzers for best autocomplete results?

hard

A. Use edge n-gram tokenizer in the indexing analyzer and standard analyzer in the search analyzer

B. Use edge n-gram tokenizer in both indexing and search analyzers

C. Use edge n-gram tokenizer in indexing analyzer and a lowercase + standard tokenizer in search analyzer

D. Use standard tokenizer in indexing analyzer and edge n-gram tokenizer in search analyzer

Solution

1. Step 1: Understand indexing with edge n-gram tokenizer

   Indexing uses edge n-gram tokenizer to create prefix tokens for autocomplete.

2. Step 2: Understand searching with standard tokenizer plus lowercase filter

   Search analyzer uses standard tokenizer to tokenize user input normally and lowercase filter to match indexed tokens.

3. Step 3: Why not use edge n-gram tokenizer in search analyzer?

   Using edge n-gram in search analyzer can cause unexpected partial matches and performance issues.

4. Final Answer:

   Index with edge n-gram, search with standard + lowercase analyzer -> Option C

5. Quick Check:

   Index edge n-gram + search standard = Use edge n-gram tokenizer in indexing analyzer and a lowercase + standard tokenizer in search analyzer [OK]

Hint: Index with edge n-gram, search with standard + lowercase [OK]

Common Mistakes:

• Using edge n-gram tokenizer in search analyzer
• Using standard tokenizer in indexing analyzer
• Not applying lowercase filter in search analyzer