Token filters (lowercase, stemmer, synonym) in Elasticsearch - Step-by-Step Execution

Concept Flow - Token filters (lowercase, stemmer, synonym)

Input Text

↓

Tokenizer splits text

↓

Lowercase filter: convert to lowercase

↓

Stemmer filter: reduce words to root form

↓

Synonym filter: replace words with synonyms

↓

Output tokens for indexing/search

Text is split into tokens, then filters apply in order: lowercase makes all letters small, stemmer reduces words to roots, synonym replaces words with their synonyms.

Execution Sample

Elasticsearch

PUT /my_index
{
  "settings": {
    "analysis": {
      "filter": {
        "my_synonym": {
          "type": "synonym",
          "synonyms": ["quick,fast"]
        },
        "my_stemmer": {
          "type": "stemmer",
          "language": "english"
        },
        "my_lowercase": {
          "type": "lowercase"
        }
      },
      "analyzer": {
        "my_analyzer": {
          "tokenizer": "standard",
          "filter": ["my_lowercase", "my_stemmer", "my_synonym"]
        }
      }
    }
  }
}

GET /my_index/_analyze
{
  "analyzer": "my_analyzer",
  "text": "Quickly running fast runners"
}

This code creates an index with a custom analyzer that lowercases, stems, and applies synonyms to the input text, then analyzes the sample text.

Execution Table

Step	Input Token	Filter Applied	Output Token	Notes
1	Quickly	Lowercase	quickly	Convert to lowercase
2	quickly	Stemmer	quick	Stem to root form
3	quick	Synonym	quick, fast	Replace with synonyms (expands to two tokens)
4	running	Lowercase	running	Already lowercase
5	running	Stemmer	run	Stem to root form
6	run	Synonym	run	No synonym found
7	fast	Lowercase	fast	Already lowercase
8	fast	Stemmer	fast	Stemmer keeps 'fast'
9	fast	Synonym	quick, fast	Synonym expands to two tokens
10	runners	Lowercase	runners	Already lowercase
11	runners	Stemmer	runner	Stem to root form
12	runner	Synonym	runner	No synonym found
13	End	-	-	All tokens processed

💡 All tokens processed through lowercase, stemmer, and synonym filters.

Variable Tracker

Token	Original	After Lowercase	After Stemmer	After Synonym
Token1	Quickly	quickly	quick	quick, fast
Token2	running	running	run	run
Token3	fast	fast	fast	quick, fast
Token4	runners	runners	runner	runner

Key Moments - 3 Insights

Why does the token 'Quickly' become two tokens 'quick' and 'fast' after the synonym filter?

Because the synonym filter replaces 'quick' with both 'quick' and 'fast', expanding one token into two as shown in execution_table row 3.

Does the stemmer always shorten words to their root form?

Yes, the stemmer reduces words like 'running' to 'run' and 'runners' to 'runner' as seen in rows 5 and 11.

Why is the lowercase filter applied before the stemmer and synonym filters?

Lowercase filter ensures all tokens are lowercase so stemmer and synonym filters work consistently, as shown in the order of filters in the execution_table.

Visual Quiz - 3 Questions

Look at the execution_table, what is the output token after the stemmer filter for 'running'?

A run

B running

C runner

D ran

💡 Hint

Check execution_table row 5 under 'Output Token' after 'Stemmer' filter.

At which step does the synonym filter expand a token into two tokens?

A Step 9

B Step 6

C Step 3

D Step 12

💡 Hint

Look for 'Synonym' filter steps that output two tokens in execution_table.

If the lowercase filter was removed, what would happen to the token 'Quickly'?

A It would become 'quickly' anyway

B It would remain 'Quickly' and might not match synonyms or stem correctly

C It would be removed

D It would become uppercase

💡 Hint

Refer to the importance of lowercase filter in key_moments and execution_table row 1.

Concept Snapshot

Token filters process tokens after splitting text.
Lowercase filter makes all letters small.
Stemmer reduces words to their root form.
Synonym filter replaces words with synonyms, possibly expanding tokens.
Filters apply in order and affect search indexing and matching.

Full Transcript

This visual execution shows how Elasticsearch token filters work step-by-step. First, the input text is split into tokens. Then, each token passes through the lowercase filter, which converts all letters to lowercase. Next, the stemmer filter reduces words to their root forms, like 'running' to 'run'. Finally, the synonym filter replaces tokens with their synonyms, sometimes expanding one token into multiple tokens, such as 'quick' becoming 'quick' and 'fast'. The execution table traces each token through these filters, showing how tokens change at each step. The variable tracker summarizes token states after each filter. Key moments clarify common confusions, like why synonyms expand tokens and why lowercase is applied first. The quiz tests understanding by asking about specific steps and effects of filters. This process helps Elasticsearch index and search text more effectively by normalizing and expanding tokens.

Practice

1.

What does the lowercase token filter do in Elasticsearch?

easy

A. Converts all tokens to lowercase letters

B. Removes stop words from the tokens

C. Finds the root form of tokens

D. Adds synonyms to tokens

Solution

1. Step 1: Understand the lowercase filter purpose

   The lowercase filter changes all tokens to lowercase to make searches case-insensitive.

2. Step 2: Compare with other filters

   Stemmer finds root words, synonym adds similar words, stop word filter removes common words. Only lowercase converts to lowercase.

3. Final Answer:

   Converts all tokens to lowercase letters -> Option A

4. Quick Check:

   Lowercase filter = converts tokens to lowercase [OK]

Hint: Lowercase filter always makes words small letters [OK]

Common Mistakes:

• Confusing lowercase with stemmer
• Thinking lowercase adds synonyms
• Mixing lowercase with stop word removal

2.

Which of the following is the correct syntax to define a synonym token filter in Elasticsearch?

{
  "filter": {
    "my_synonym": {
      "type": "synonym",
      "synonyms": ["quick,fast"]
    }
  }
}

easy

A. { "type": "stemmer", "synonyms": ["quick,fast"] }

B. { "type": "stop", "synonyms": ["quick,fast"] }

C. { "type": "synonym", "synonyms": ["quick,fast"] }

D. { "type": "lowercase", "synonyms": ["quick,fast"] }

Solution

1. Step 1: Identify the token filter type for synonyms

   The synonym filter must have "type": "synonym" to work correctly.

2. Step 2: Check the synonyms property

   The synonyms list is correctly provided as an array of strings with comma-separated words.

3. Final Answer:

   { "type": "synonym", "synonyms": ["quick,fast"] } -> Option C

4. Quick Check:

   Synonym filter syntax = type synonym with synonyms list [OK]

Hint: Synonym filter always uses type 'synonym' with synonyms array [OK]

Common Mistakes:

• Using wrong type like stemmer or lowercase
• Not providing synonyms as an array
• Misspelling 'synonym' in type

3.

Given this analyzer configuration in Elasticsearch:

{
  "analyzer": {
    "my_analyzer": {
      "tokenizer": "standard",
      "filter": ["lowercase", "porter_stem"]
    }
  }
}

What will be the output tokens for the input text "Running Runs"?

medium

A. ["run", "runs"]

B. ["running", "runs"]

C. ["Running", "Runs"]

D. ["run", "run"]

Solution

1. Step 1: Apply lowercase filter

   Input "Running Runs" becomes "running runs" after lowercase filter.

2. Step 2: Apply porter_stem filter

   Porter stemmer reduces "running" and "runs" to root "run" for both tokens.

3. Final Answer:

   ["run", "run"] -> Option D

4. Quick Check:

   Lowercase + stemmer = root forms in lowercase [OK]

Hint: Lowercase then stemmer gives root lowercase tokens [OK]

Common Mistakes:

• Ignoring stemming effect
• Not applying lowercase first
• Expecting original case tokens

4.

Identify the error in this token filter definition:

{
  "filter": {
    "my_stemmer": {
      "type": "stemmer",
      "language": "englishy"
    }
  }
}

medium

A. "type" should be "stemmer_override"

B. "language" value is misspelled; should be "english"

C. "filter" key must be inside "analyzer"

D. Missing "synonyms" array

Solution

1. Step 1: Check the "language" parameter spelling

   The language "englishy" is incorrect; the correct value is "english".

2. Step 2: Verify the stemmer type and parameters

   Type "stemmer" is correct; no synonyms needed here; filter can be defined outside analyzer.

3. Final Answer:

   "language" value is misspelled; should be "english" -> Option B

4. Quick Check:

   Stemmer language spelling must be exact [OK]

Hint: Check spelling of language names in stemmer filters [OK]

Common Mistakes:

• Using invalid language names
• Confusing stemmer with synonym filter
• Expecting synonyms in stemmer filter

5.

You want to create an analyzer that makes searches case-insensitive, finds root forms of words, and also matches synonyms like "fast" for "quick". Which token filters should you include in your analyzer?

hard

A. ["lowercase", "stemmer", "synonym"]

B. ["stop", "lowercase", "synonym"]

C. ["lowercase", "synonym"]

D. ["stemmer", "stop"]

Solution

1. Step 1: Identify filters for case-insensitivity and root forms

   Lowercase filter makes tokens lowercase; stemmer finds root forms.

2. Step 2: Add synonym filter for matching similar words

   Synonym filter adds synonyms like "fast" for "quick" to improve matching.

3. Final Answer:

   ["lowercase", "stemmer", "synonym"] -> Option A

4. Quick Check:

   Lowercase + stemmer + synonym = case-insensitive, root, and synonym matching [OK]

Hint: Combine lowercase, stemmer, and synonym filters for full matching [OK]

Common Mistakes:

• Omitting stemmer or synonym filters
• Using stop filter instead of stemmer
• Not including lowercase for case-insensitivity