Token filters (lowercase, stemmer, synonym) in Elasticsearch - Mini Project: Build & Apply

Create a Custom Elasticsearch Analyzer with Token Filters

📖 Scenario: You are setting up a search engine for a book store. You want to make sure that searches find books even if users type words in different forms or cases. For example, searching for "Running" should find books with "running" or "run". Also, some words have synonyms, like "quick" and "fast".

🎯 Goal: Build an Elasticsearch index with a custom analyzer that uses token filters: lowercase, stemmer, and synonym filters. This will help the search engine understand different word forms and synonyms.

📋 What You'll Learn

Create an index called books.

Define a custom analyzer named custom_analyzer.

Use the standard tokenizer in the analyzer.

Add three token filters to the analyzer: lowercase, english_stemmer, and synonym_filter.

Define the english_stemmer filter as a stemmer for English.

Define the synonym_filter with synonyms: quick,fast and jumps,leaps.

💡 Why This Matters

🌍 Real World

Search engines often need to understand different word forms and synonyms to give better results. This project shows how to set up such features in Elasticsearch.

💼 Career

Many jobs in search engineering, data engineering, and backend development require knowledge of text analysis and Elasticsearch configuration.

1

Create the index with basic settings

Create an Elasticsearch index called books with an empty settings object.

Elasticsearch

{
  "settings": {
    
  }
}

# Your code here

Hint

Use the PUT method to create the index books with empty settings.

2

Add the custom analyzer with tokenizer and filters

Inside the settings, add an analysis section. Define a custom analyzer named custom_analyzer that uses the standard tokenizer and the token filters lowercase, english_stemmer, and synonym_filter in that order.

Elasticsearch

PUT /books
{
  "settings": {
    "analysis": {
      "analyzer": {
        "custom_analyzer": {
          "tokenizer": "standard",
          "filter": ["lowercase", "english_stemmer", "synonym_filter"]
        }
      }
    }
  }
}

# Your code here

Hint

Remember to put the analysis section inside settings. Define the analyzer with the exact name custom_analyzer.

3

Define the stemmer and synonym token filters

Add the token filters english_stemmer and synonym_filter inside the analysis section. Define english_stemmer as a stemmer filter with stemmer type and english language. Define synonym_filter as a synonym filter with synonyms quick,fast and jumps,leaps.

Elasticsearch

PUT /books
{
  "settings": {
    "analysis": {
      "analyzer": {
        "custom_analyzer": {
          "tokenizer": "standard",
          "filter": ["lowercase", "english_stemmer", "synonym_filter"]
        }
      },
      "filter": {
        "english_stemmer": {
          "type": "stemmer",
          "language": "english"
        },
        "synonym_filter": {
          "type": "synonym",
          "synonyms": ["quick,fast", "jumps,leaps"]
        }
      }
    }
  }
}

# Your code here

Hint

Define the filters inside the filter section under analysis. Use the exact names english_stemmer and synonym_filter.

4

Test the analyzer with a sample text

Use the _analyze API to test the custom_analyzer on the text "The quick brown fox jumps running fast". Print the tokens produced by the analyzer.

Elasticsearch

POST /books/_analyze
{
  "analyzer": "custom_analyzer",
  "text": "The quick brown fox jumps running fast"
}

# Your code here

Hint

Use the POST /books/_analyze API with the custom_analyzer and the given text. The output tokens should include stemmed and synonym words.

Practice

1.

What does the lowercase token filter do in Elasticsearch?

easy

A. Converts all tokens to lowercase letters

B. Removes stop words from the tokens

C. Finds the root form of tokens

D. Adds synonyms to tokens

Solution

1. Step 1: Understand the lowercase filter purpose

   The lowercase filter changes all tokens to lowercase to make searches case-insensitive.

2. Step 2: Compare with other filters

   Stemmer finds root words, synonym adds similar words, stop word filter removes common words. Only lowercase converts to lowercase.

3. Final Answer:

   Converts all tokens to lowercase letters -> Option A

4. Quick Check:

   Lowercase filter = converts tokens to lowercase [OK]

Hint: Lowercase filter always makes words small letters [OK]

Common Mistakes:

• Confusing lowercase with stemmer
• Thinking lowercase adds synonyms
• Mixing lowercase with stop word removal

2.

Which of the following is the correct syntax to define a synonym token filter in Elasticsearch?

{
  "filter": {
    "my_synonym": {
      "type": "synonym",
      "synonyms": ["quick,fast"]
    }
  }
}

easy

A. { "type": "stemmer", "synonyms": ["quick,fast"] }

B. { "type": "stop", "synonyms": ["quick,fast"] }

C. { "type": "synonym", "synonyms": ["quick,fast"] }

D. { "type": "lowercase", "synonyms": ["quick,fast"] }

Solution

1. Step 1: Identify the token filter type for synonyms

   The synonym filter must have "type": "synonym" to work correctly.

2. Step 2: Check the synonyms property

   The synonyms list is correctly provided as an array of strings with comma-separated words.

3. Final Answer:

   { "type": "synonym", "synonyms": ["quick,fast"] } -> Option C

4. Quick Check:

   Synonym filter syntax = type synonym with synonyms list [OK]

Hint: Synonym filter always uses type 'synonym' with synonyms array [OK]

Common Mistakes:

• Using wrong type like stemmer or lowercase
• Not providing synonyms as an array
• Misspelling 'synonym' in type

3.

Given this analyzer configuration in Elasticsearch:

{
  "analyzer": {
    "my_analyzer": {
      "tokenizer": "standard",
      "filter": ["lowercase", "porter_stem"]
    }
  }
}

What will be the output tokens for the input text "Running Runs"?

medium

A. ["run", "runs"]

B. ["running", "runs"]

C. ["Running", "Runs"]

D. ["run", "run"]

Solution

1. Step 1: Apply lowercase filter

   Input "Running Runs" becomes "running runs" after lowercase filter.

2. Step 2: Apply porter_stem filter

   Porter stemmer reduces "running" and "runs" to root "run" for both tokens.

3. Final Answer:

   ["run", "run"] -> Option D

4. Quick Check:

   Lowercase + stemmer = root forms in lowercase [OK]

Hint: Lowercase then stemmer gives root lowercase tokens [OK]

Common Mistakes:

• Ignoring stemming effect
• Not applying lowercase first
• Expecting original case tokens

4.

Identify the error in this token filter definition:

{
  "filter": {
    "my_stemmer": {
      "type": "stemmer",
      "language": "englishy"
    }
  }
}

medium

A. "type" should be "stemmer_override"

B. "language" value is misspelled; should be "english"

C. "filter" key must be inside "analyzer"

D. Missing "synonyms" array

Solution

1. Step 1: Check the "language" parameter spelling

   The language "englishy" is incorrect; the correct value is "english".

2. Step 2: Verify the stemmer type and parameters

   Type "stemmer" is correct; no synonyms needed here; filter can be defined outside analyzer.

3. Final Answer:

   "language" value is misspelled; should be "english" -> Option B

4. Quick Check:

   Stemmer language spelling must be exact [OK]

Hint: Check spelling of language names in stemmer filters [OK]

Common Mistakes:

• Using invalid language names
• Confusing stemmer with synonym filter
• Expecting synonyms in stemmer filter

5.

You want to create an analyzer that makes searches case-insensitive, finds root forms of words, and also matches synonyms like "fast" for "quick". Which token filters should you include in your analyzer?

hard

A. ["lowercase", "stemmer", "synonym"]

B. ["stop", "lowercase", "synonym"]

C. ["lowercase", "synonym"]

D. ["stemmer", "stop"]

Solution

1. Step 1: Identify filters for case-insensitivity and root forms

   Lowercase filter makes tokens lowercase; stemmer finds root forms.

2. Step 2: Add synonym filter for matching similar words

   Synonym filter adds synonyms like "fast" for "quick" to improve matching.

3. Final Answer:

   ["lowercase", "stemmer", "synonym"] -> Option A

4. Quick Check:

   Lowercase + stemmer + synonym = case-insensitive, root, and synonym matching [OK]

Hint: Combine lowercase, stemmer, and synonym filters for full matching [OK]

Common Mistakes:

• Omitting stemmer or synonym filters
• Using stop filter instead of stemmer
• Not including lowercase for case-insensitivity