Analyzer components (tokenizer, filters) in Elasticsearch - Step-by-Step Execution

Concept Flow - Analyzer components (tokenizer, filters)

Input Text

↓

Tokenizer: splits text into tokens

↓

Filter 1: modifies tokens

↓

Filter 2: modifies tokens

↓

Output: final tokens for indexing/search

Text goes through a tokenizer to split it into words, then filters change these words step-by-step to prepare for search.

Execution Sample

Elasticsearch

POST _analyze
{
  "tokenizer": "standard",
  "filter": ["lowercase", "stop"],
  "text": "The Quick Brown Fox"
}

This example splits text into words, makes them lowercase, and removes common stop words.

Execution Table

Step	Action	Input Tokens	Output Tokens
1	Tokenizer splits text	["The Quick Brown Fox"]	["The", "Quick", "Brown", "Fox"]
2	Lowercase filter	["The", "Quick", "Brown", "Fox"]	["the", "quick", "brown", "fox"]
3	Stop filter removes stop words	["the", "quick", "brown", "fox"]	["quick", "brown", "fox"]
4	End of analysis	["quick", "brown", "fox"]	["quick", "brown", "fox"]

💡 All filters applied, final tokens ready for indexing/search.

Variable Tracker

Variable	Start	After Tokenizer	After Lowercase Filter	After Stop Filter	Final
tokens	N/A	["The", "Quick", "Brown", "Fox"]	["the", "quick", "brown", "fox"]	["quick", "brown", "fox"]	["quick", "brown", "fox"]

Key Moments - 2 Insights

Why does the token list change after each filter?

Each filter modifies the tokens step-by-step, as shown in execution_table rows 2 and 3, changing case or removing words.

Why is 'The' removed after the stop filter?

'The' is a common stop word removed by the stop filter, as seen in execution_table row 3 where it disappears from tokens.

Visual Quiz - 3 Questions

Look at the execution_table, what tokens are output after the lowercase filter (step 2)?

A ["The", "Quick", "Brown", "Fox"]

B ["the", "quick", "brown", "fox"]

C ["quick", "brown", "fox"]

D ["THE", "QUICK", "BROWN", "FOX"]

💡 Hint

Check the Output Tokens column at step 2 in execution_table.

At which step are stop words removed from the tokens?

A Step 3

B Step 1

C Step 2

D Step 4

💡 Hint

Look for the step where tokens like 'the' disappear in execution_table.

If we remove the lowercase filter, what would be the output tokens after the stop filter?

A ["Quick", "Brown", "Fox"]

B ["the", "quick", "brown", "fox"]

C ["The", "Quick", "Brown", "Fox"]

D ["quick", "brown", "fox"]

💡 Hint

Without lowercase filter, stop filter does not remove 'The' due to case sensitivity, leaving ["The", "Quick", "Brown", "Fox"].

Concept Snapshot

Analyzer components process text in steps:
1. Tokenizer splits text into words.
2. Filters modify tokens (e.g., lowercase, remove stop words).
3. Final tokens are used for search indexing.
Each filter changes tokens step-by-step.

Full Transcript

In Elasticsearch, an analyzer breaks text into tokens using a tokenizer, then applies filters to modify these tokens. For example, the standard tokenizer splits 'The Quick Brown Fox' into ['The', 'Quick', 'Brown', 'Fox']. Then the lowercase filter changes them to ['the', 'quick', 'brown', 'fox']. Next, the stop filter removes common words like 'the', resulting in ['quick', 'brown', 'fox']. This step-by-step process prepares text for efficient searching.