Custom analyzers in Elasticsearch - Step-by-Step Execution

Concept Flow - Custom analyzers

Define tokenizer

↓

Define filters

↓

Create custom analyzer

↓

Apply analyzer to text

↓

Tokenize text

↓

Apply filters to tokens

↓

Output processed tokens

Custom analyzers combine a tokenizer and filters to process text into tokens for searching.

Execution Sample

Elasticsearch

PUT /my_index
{
  "settings": {
    "analysis": {
      "analyzer": {
        "my_custom_analyzer": {
          "type": "custom",
          "tokenizer": "standard",
          "filter": ["lowercase", "asciifolding"]
        }
      }
    }
  }
}

This code creates an index with a custom analyzer that tokenizes text, lowercases it, and removes accents.

Execution Table

Step	Action	Input Text / Tokens	Output Tokens
1	Input text to analyze	"Café Déjà Vu"	"Café Déjà Vu"
2	Tokenize with standard tokenizer	"Café Déjà Vu"	["Café", "Déjà", "Vu"]
3	Apply lowercase filter	["Café", "Déjà", "Vu"]	["café", "déjà", "vu"]
4	Apply asciifolding filter	["café", "déjà", "vu"]	["cafe", "deja", "vu"]
5	Output final tokens	["cafe", "deja", "vu"]	["cafe", "deja", "vu"]

💡 All filters applied; final tokens ready for indexing or searching.

Variable Tracker

Variable	Start	After Tokenize	After Lowercase	After Asciifolding	Final
tokens	N/A	["Café", "Déjà", "Vu"]	["café", "déjà", "vu"]	["cafe", "deja", "vu"]	["cafe", "deja", "vu"]

Key Moments - 3 Insights

Why do tokens change after applying the lowercase filter?

Because the lowercase filter converts all tokens to lowercase, as shown in step 3 of the execution_table where tokens change from ["Café", "Déjà", "Vu"] to ["café", "déjà", "vu"].

What does the asciifolding filter do to tokens?

It removes accents and converts characters to their ASCII equivalents, as seen in step 4 where ["café", "déjà", "vu"] becomes ["cafe", "deja", "vu"].

Why is the tokenizer step important before filters?

Because the tokenizer splits the input text into tokens, which filters then process. Without tokenization (step 2), filters cannot work on individual words.

Visual Quiz - 3 Questions

Look at the execution_table, what are the tokens immediately after tokenization (step 2)?

A ["cafe", "deja", "vu"]

B ["Café", "Déjà", "Vu"]

C ["café", "déjà", "vu"]

D ["Cafe", "Deja", "Vu"]

💡 Hint

Check the Output Tokens column at step 2 in the execution_table.

At which step do tokens lose their accents?

A Step 4 - Asciifolding filter

B Step 3 - Lowercase filter

C Step 2 - Tokenize

D Step 5 - Output final tokens

💡 Hint

Look at the change from accented to unaccented tokens in the Output Tokens column.

If we remove the lowercase filter, what would be the tokens after asciifolding?

A ["cafe", "deja", "vu"]

B ["Café", "Déjà", "Vu"]

C ["Cafe", "Deja", "Vu"]

D ["café", "déjà", "vu"]

💡 Hint

Consider that asciifolding removes accents but does not change case; lowercase filter changes case.

Concept Snapshot

Custom analyzers in Elasticsearch combine a tokenizer and filters.
Tokenizer splits text into tokens.
Filters modify tokens (e.g., lowercase, asciifolding).
Define in index settings under analysis.analyzer.
Used to control how text is indexed and searched.

Full Transcript

Custom analyzers in Elasticsearch let you control how text is broken into tokens and processed. First, a tokenizer splits the text into words or tokens. Then, filters change these tokens, for example by making them lowercase or removing accents. In the example, the text "Café Déjà Vu" is tokenized into ["Café", "Déjà", "Vu"]. The lowercase filter changes these to ["café", "déjà", "vu"]. The asciifolding filter then removes accents, resulting in ["cafe", "deja", "vu"]. This process helps make searching more flexible and accurate. You define custom analyzers in the index settings under the analysis section, specifying the tokenizer and filters you want to use.

Practice

1. What is the main purpose of a custom analyzer in Elasticsearch?

easy

A. To control how text is broken down and processed before indexing

B. To store documents in the database

C. To create visualizations of search results

D. To manage user access permissions

Solution

1. Step 1: Understand the role of analyzers

   Analyzers in Elasticsearch process text by breaking it into tokens and applying filters.

2. Step 2: Identify the purpose of custom analyzers

   Custom analyzers let you define specific tokenizers and filters to control text processing.

3. Final Answer:

   To control how text is broken down and processed before indexing -> Option A

4. Quick Check:

   Custom analyzer = text processing control [OK]

Hint: Custom analyzers shape text before search indexing [OK]

Common Mistakes:

• Confusing analyzers with data storage
• Thinking analyzers create visualizations
• Mixing analyzers with security settings

2. Which of the following is the correct way to define a custom analyzer with a lowercase filter in Elasticsearch settings?

easy

A. "analyzer": {"my_analyzer": {"type": "custom", "tokenizer": "standard", "filter": ["lowercase"]}}

B. "analyzer": {"my_analyzer": {"type": "standard", "tokenizer": "lowercase", "filter": ["custom"]}}

C. "analyzer": {"my_analyzer": {"type": "custom", "tokenizer": "lowercase", "filter": ["standard"]}}

D. "analyzer": {"my_analyzer": {"type": "filter", "tokenizer": "standard", "filter": ["lowercase"]}}

Solution

1. Step 1: Review custom analyzer syntax

   A custom analyzer requires "type": "custom", a tokenizer, and filters as an array.

2. Step 2: Match correct tokenizer and filter

   "standard" tokenizer and "lowercase" filter are valid and correctly placed in "analyzer": {"my_analyzer": {"type": "custom", "tokenizer": "standard", "filter": ["lowercase"]}}.

3. Final Answer:

   "analyzer": {"my_analyzer": {"type": "custom", "tokenizer": "standard", "filter": ["lowercase"]}} -> Option A

4. Quick Check:

   Custom analyzer syntax = "analyzer": {"my_analyzer": {"type": "custom", "tokenizer": "standard", "filter": ["lowercase"]}} [OK]

Hint: Custom analyzer needs type, tokenizer, and filter array [OK]

Common Mistakes:

• Using wrong type like 'filter' instead of 'custom'
• Swapping tokenizer and filter names
• Missing array brackets for filters

3. Given this custom analyzer configuration:

{
  "settings": {
    "analysis": {
      "analyzer": {
        "my_analyzer": {
          "type": "custom",
          "tokenizer": "standard",
          "filter": ["lowercase", "stop"]
        }
      },
      "filter": {
        "stop": {
          "type": "stop",
          "stopwords": ["the", "is"]
        }
      }
    }
  }
}

What is the output tokens when analyzing the text "The cat is on the mat" with my_analyzer?

medium

A. ["The", "cat", "is", "on", "the", "mat"]

B. ["cat", "is", "on", "mat"]

C. ["the", "cat", "is", "on", "the", "mat"]

D. ["cat", "on", "mat"]

Solution

1. Step 1: Apply lowercase filter

   All words become lowercase: "the", "cat", "is", "on", "the", "mat".

2. Step 2: Apply stop filter with stopwords ["the", "is"]

   Remove "the" and "is" from tokens, leaving "cat", "on", "mat".

3. Final Answer:

   ["cat", "on", "mat"] -> Option D

4. Quick Check:

   Lowercase + stop filter removes 'the' and 'is' [OK]

Hint: Lowercase then remove stopwords from tokens [OK]

Common Mistakes:

• Ignoring lowercase filter effect
• Not removing stopwords correctly
• Confusing original text with tokens

4. Identify the error in this custom analyzer definition:

{
  "analysis": {
    "analyzer": {
      "my_analyzer": {
        "type": "custom",
        "tokenizer": "standard",
        "filter": "lowercase"
      }
    }
  }
}

medium

A. Missing stopwords definition

B. The tokenizer name is invalid

C. The filter should be an array, not a string

D. The analyzer type should be "standard"

Solution

1. Step 1: Check filter syntax in custom analyzer

   Filters must be provided as an array, e.g., ["lowercase"], not a string.

2. Step 2: Verify other parts

   Tokenizer "standard" and analyzer type "custom" are correct; stopwords are optional.

3. Final Answer:

   The filter should be an array, not a string -> Option C

4. Quick Check:

   Filter must be array syntax [OK]

Hint: Filters always use array brackets [] in custom analyzers [OK]

Common Mistakes:

• Using string instead of array for filters
• Changing tokenizer to invalid types
• Assuming stopwords are mandatory

5. You want to create a custom analyzer that removes English stopwords, lowercases text, and also applies a stemmer filter named english_stemmer. Which of the following is the correct way to define this analyzer in Elasticsearch settings?

hard

A. {
"analysis": {
"analyzer": {
"custom_english": {
"type": "standard",
"tokenizer": "standard",
"filter": ["lowercase", "stop", "english_stemmer"]
}
},
"filter": {
"stop": {
"type": "stop",
"stopwords": "_english_"
},
"english_stemmer": {
"type": "stemmer",
"language": "english"
}
}
}
}

B. {
"analysis": {
"analyzer": {
"custom_english": {
"type": "custom",
"tokenizer": "standard",
"filter": ["lowercase", "stop", "english_stemmer"]
}
},
"filter": {
"stop": {
"type": "stop",
"stopwords": "_english_"
},
"english_stemmer": {
"type": "stemmer",
"language": "english"
}
}
}
}

C. {
"analysis": {
"analyzer": {
"custom_english": {
"type": "custom",
"tokenizer": "standard",
"filter": ["stop", "lowercase", "english_stemmer"]
}
},
"filter": {
"stop": {
"type": "stop",
"stopwords": "_english_"
},
"english_stemmer": {
"type": "stemmer",
"language": "english"
}
}
}
}

D. {
"analysis": {
"analyzer": {
"custom_english": {
"type": "custom",
"tokenizer": "standard",
"filter": ["lowercase", "english_stemmer", "stop"]
}
},
"filter": {
"stop": {
"type": "stop",
"stopwords": "_english_"
},
"english_stemmer": {
"type": "stemmer",
"language": "english"
}
}
}
}

Solution

1. Step 1: Confirm analyzer type and tokenizer

   The analyzer must be "custom" with "standard" tokenizer to apply filters correctly.

2. Step 2: Check filter order and definitions

   Filters should be ["lowercase", "stop", "english_stemmer"] in that order to lowercase first, remove stopwords next, then stem.

3. Step 3: Verify filter definitions

   Stop filter uses "_english_" stopwords, and stemmer filter is correctly defined with language "english".

4. Final Answer:

   { "analysis": { "analyzer": { "custom_english": { "type": "custom", "tokenizer": "standard", "filter": ["lowercase", "stop", "english_stemmer"] } }, "filter": { "stop": { "type": "stop", "stopwords": "_english_" }, "english_stemmer": { "type": "stemmer", "language": "english" } } } } -> Option B

5. Quick Check:

   Custom analyzer with correct filter order and definitions = { "analysis": { "analyzer": { "custom_english": { "type": "custom", "tokenizer": "standard", "filter": ["lowercase", "stop", "english_stemmer"] } }, "filter": { "stop": { "type": "stop", "stopwords": "_english_" }, "english_stemmer": { "type": "stemmer", "language": "english" } } } } [OK]

Hint: Filter order matters: lowercase, stop, then stemmer [OK]

Common Mistakes:

• Using wrong analyzer type like 'standard'
• Incorrect filter order causing wrong results
• Missing or wrong stopwords definition