Standard analyzer in Elasticsearch - Step-by-Step Execution

Concept Flow - Standard analyzer

Input Text

↓

Lowercase Filter

↓

Standard Tokenizer

↓

Stop Words Filter

↓

Output Tokens

The standard analyzer processes text by first splitting into tokens, then lowercasing them, removing common stop words, and outputting the cleaned tokens.

Execution Sample

Elasticsearch

{
  "analyzer": "standard",
  "text": "The Quick Brown Foxes jumped over the lazy dogs."
}

This example analyzes the input text using the standard analyzer to produce tokens.

Execution Table

Step	Action	Input	Output
1	Input text received	The Quick Brown Foxes jumped over the lazy dogs.	The Quick Brown Foxes jumped over the lazy dogs.
2	Standard tokenizer splits text	The Quick Brown Foxes jumped over the lazy dogs.	["The", "Quick", "Brown", "Foxes", "jumped", "over", "the", "lazy", "dogs"]
3	Lowercase filter applied	["The", "Quick", "Brown", "Foxes", "jumped", "over", "the", "lazy", "dogs"]	["the", "quick", "brown", "foxes", "jumped", "over", "the", "lazy", "dogs"]
4	Stop words removed	["the", "quick", "brown", "foxes", "jumped", "over", "the", "lazy", "dogs"]	["quick", "brown", "foxes", "jumped", "lazy", "dogs"]

💡 All stop words removed, final tokens ready for indexing or searching.

Variable Tracker

Variable	Start	After Step 2	After Step 3	After Step 4
text	The Quick Brown Foxes jumped over the lazy dogs.	["The", "Quick", "Brown", "Foxes", "jumped", "over", "the", "lazy", "dogs"]	["the", "quick", "brown", "foxes", "jumped", "over", "the", "lazy", "dogs"]	["quick", "brown", "foxes", "jumped", "lazy", "dogs"]

Key Moments - 3 Insights

Why does the analyzer convert all letters to lowercase?

Lowercasing ensures that searches are case-insensitive, so 'Quick' and 'quick' are treated the same, as shown in step 3 of the execution_table.

What does the standard tokenizer do with punctuation?

The standard tokenizer splits text into words by removing punctuation and spaces, as seen in step 2 where the sentence is split into tokens without periods.

Why are some words like 'the' removed in the final output?

Common words called stop words are removed to reduce noise and improve search relevance, demonstrated in step 4 where 'the' is removed.

Visual Quiz - 3 Questions

Look at the execution_table, what is the output after the lowercase filter (Step 3)?

A "the quick brown foxes jumped over the lazy dogs."

B ["the", "quick", "brown", "foxes"]

C "The Quick Brown Foxes jumped over the lazy dogs."

D ["quick", "brown", "foxes", "jumped"]

💡 Hint

Check the Output column in Step 3 of the execution_table.

At which step are stop words removed from the tokens?

A Step 1

B Step 2

C Step 4

D Step 3

💡 Hint

Look at the Action column and Output tokens in the execution_table.

If the input text was 'Cats and dogs', which token would likely be removed by the stop words filter?

A "and"

B "cats"

C "dogs"

D "cats and dogs"

💡 Hint

Refer to the stop words removal step in the execution_table and variable_tracker.

Concept Snapshot

Standard analyzer:
- Splits text into tokens by words
- Lowercases tokens
- Removes common stop words
- Used for indexing and searching
- Helps match queries case-insensitively and ignore noise words

Full Transcript

The standard analyzer in Elasticsearch processes text by first splitting the text into individual words using the standard tokenizer, which removes punctuation. Then it converts all letters to lowercase to ensure case-insensitive matching. After tokenizing and lowercasing, it removes common stop words like 'the' and 'and' to reduce noise. The final output is a list of clean tokens ready for indexing or searching. This process helps Elasticsearch find relevant matches regardless of case and ignores common words that do not add meaning.

Practice

1. What does the standard analyzer in Elasticsearch do by default?

easy

A. Keeps punctuation and case intact

B. Breaks text into words, ignores punctuation and case

C. Only analyzes numbers and symbols

D. Removes all spaces and joins words

Solution

1. Step 1: Understand the role of the standard analyzer

   The standard analyzer splits text into simple words by removing punctuation and ignoring case differences.

2. Step 2: Recall default behavior in Elasticsearch

   It is the default analyzer and is designed for general text processing, making text lowercase and splitting on punctuation.

3. Final Answer:

   Breaks text into words, ignores punctuation and case -> Option B

4. Quick Check:

   Standard analyzer = splits words, ignores punctuation and case [OK]

Hint: Standard analyzer splits words and lowercases text [OK]

Common Mistakes:

• Thinking it keeps punctuation
• Assuming it preserves case
• Believing it only analyzes numbers

2. Which of the following is the correct way to specify the standard analyzer in an Elasticsearch index mapping?

easy

A. "analyzer": "whitespace"

B. "analyzer": "simple"

C. "analyzer": "keyword"

D. "analyzer": "standard"

Solution

1. Step 1: Identify the analyzer name for standard analyzer

   The standard analyzer is referenced by the string "standard" in Elasticsearch mappings.

2. Step 2: Compare with other analyzer names

   Other analyzers like "simple", "keyword", and "whitespace" are different and not the standard analyzer.

3. Final Answer:

   "analyzer": "standard" -> Option D

4. Quick Check:

   Standard analyzer name = "standard" [OK]

Hint: Use "analyzer": "standard" to apply standard analyzer [OK]

Common Mistakes:

• Using other analyzer names by mistake
• Forgetting quotes around analyzer name
• Confusing standard with simple analyzer

3. Given the text "Hello, World! Elasticsearch 7.10", what tokens will the standard analyzer produce?

medium

A. ["Hello", "World", "Elasticsearch", "7.10"]

B. ["hello,", "world!", "elasticsearch", "7.10"]

C. ["hello", "world", "elasticsearch", "7", "10"]

D. ["hello world elasticsearch 7 10"]

Solution

1. Step 1: Apply standard analyzer tokenization rules

   The standard analyzer lowercases text and splits on punctuation and whitespace, so "Hello," becomes "hello", "World!" becomes "world".

2. Step 2: Analyze numbers and punctuation

   Numbers separated by punctuation like "7.10" are split into "7" and "10" tokens.

3. Final Answer:

   ["hello", "world", "elasticsearch", "7", "10"] -> Option C

4. Quick Check:

   Standard analyzer tokens = lowercase words split on punctuation [OK]

Hint: Standard analyzer lowercases and splits on punctuation [OK]

Common Mistakes:

• Keeping punctuation attached to words
• Not splitting numbers on dots
• Preserving original case

4. You wrote this mapping snippet but your field is not analyzed as expected:

{
  "mappings": {
    "properties": {
      "title": {
        "type": "text",
        "analyzer": "standard_analyzer"
      }
    }
  }
}

What is the error?

medium

A. Analyzer name should be "standard", not "standard_analyzer"

B. Field type should be "keyword" instead of "text"

C. Analyzer must be defined under "settings", not "mappings"

D. Missing "index": true property

Solution

1. Step 1: Check the analyzer name used in mapping

   The standard analyzer is named "standard". Using "standard_analyzer" is incorrect and causes the analyzer to not be found.

2. Step 2: Verify field type and analyzer placement

   Field type "text" is correct for analyzed fields, and analyzer is correctly placed under "mappings". "index": true is default and not required.

3. Final Answer:

   Analyzer name should be "standard", not "standard_analyzer" -> Option A

4. Quick Check:

   Analyzer name must be exact "standard" [OK]

Hint: Use exact analyzer name "standard" in mappings [OK]

Common Mistakes:

• Adding extra suffix to analyzer name
• Changing field type to keyword for analyzed text
• Misplacing analyzer in settings instead of mappings

5. You want to create a custom analyzer that uses the standard tokenizer but adds a lowercase filter and a stop word filter. Which of the following JSON snippets correctly defines this analyzer in Elasticsearch?

hard

A. {
"settings": {
"analysis": {
"analyzer": {
"custom_std": {
"type": "custom",
"tokenizer": "standard",
"filter": ["lowercase", "stop"]
}
}
}
}
}

B. {
"settings": {
"analysis": {
"analyzer": {
"custom_std": {
"type": "standard",
"filter": ["lowercase", "stop"]
}
}
}
}
}

C. {
"settings": {
"analysis": {
"analyzer": {
"custom_std": {
"type": "custom",
"tokenizer": "simple",
"filter": ["lowercase", "stop"]
}
}
}
}
}

D. {
"settings": {
"analysis": {
"analyzer": {
"custom_std": {
"type": "custom",
"tokenizer": "standard",
"filter": ["stop", "uppercase"]
}
}
}
}
}

Solution

1. Step 1: Define a custom analyzer with standard tokenizer

   The analyzer must have "type": "custom" and use "tokenizer": "standard" to start with the standard tokenizer.

2. Step 2: Add lowercase and stop filters in correct order

   The filters array should include "lowercase" and "stop" filters to lowercase tokens and remove stop words.

3. Step 3: Verify other options

   { "settings": { "analysis": { "analyzer": { "custom_std": { "type": "standard", "filter": ["lowercase", "stop"] } } } } } uses "type": "standard" which does not allow custom filters. { "settings": { "analysis": { "analyzer": { "custom_std": { "type": "custom", "tokenizer": "simple", "filter": ["lowercase", "stop"] } } } } } uses "simple" tokenizer, not standard. { "settings": { "analysis": { "analyzer": { "custom_std": { "type": "custom", "tokenizer": "standard", "filter": ["stop", "uppercase"] } } } } } uses "uppercase" filter which is incorrect here.

4. Final Answer:

   Custom analyzer with type custom, standard tokenizer, and filters lowercase and stop -> Option A

5. Quick Check:

   Custom analyzer = type custom + standard tokenizer + lowercase + stop filters [OK]

Hint: Custom analyzer needs type custom and tokenizer standard [OK]

Common Mistakes:

• Using type standard instead of custom for filters
• Choosing wrong tokenizer like simple
• Wrong filter order or filters like uppercase