Testing analyzers (_analyze API) in Elasticsearch - Step-by-Step Execution

Concept Flow - Testing analyzers (_analyze API)

Send text to _analyze API

↓

Elasticsearch applies analyzer

↓

Text is tokenized

↓

Tokens are filtered

↓

Return tokens and details

↓

Display output

The flow shows how text is sent to the _analyze API, processed by the analyzer, tokenized, filtered, and then returned as tokens.

Execution Sample

Elasticsearch

POST _analyze
{
  "analyzer": "standard",
  "text": "Quick Brown Foxes"
}

This request tests the 'standard' analyzer on the text 'Quick Brown Foxes' and returns the tokens.

Execution Table

Step	Action	Input Text	Analyzer Used	Tokens Produced
1	Send request to _analyze API	Quick Brown Foxes	standard
2	Apply standard analyzer	Quick Brown Foxes	standard
3	Tokenize text	Quick Brown Foxes	standard	["quick", "brown", "foxes"]
4	Return tokens	Quick Brown Foxes	standard	["quick", "brown", "foxes"]
5	Display output	Quick Brown Foxes	standard	["quick", "brown", "foxes"]

💡 All tokens produced and returned by the analyzer; execution ends.

Variable Tracker

Variable	Start	After Step 2	After Step 3	Final
Input Text	"Quick Brown Foxes"	"Quick Brown Foxes"	"Quick Brown Foxes"	"Quick Brown Foxes"
Analyzer	none	"standard"	"standard"	"standard"
Tokens	none	none	["quick", "brown", "foxes"]	["quick", "brown", "foxes"]

Key Moments - 2 Insights

Why are the tokens all lowercase even though the input text has uppercase letters?

Because the 'standard' analyzer lowercases tokens during analysis, as shown in execution_table step 3 where tokens are all lowercase.

What happens if we change the analyzer to 'whitespace'?

The analyzer will split tokens only on whitespace without lowercasing, so tokens keep original case; this would change tokens in step 3.

Visual Quiz - 3 Questions

Look at the execution table, what tokens are produced at step 3?

A ["Quick", "Brown", "Foxes"]

B ["quickbrownfoxes"]

C ["quick", "brown", "foxes"]

D ["QuickBrownFoxes"]

💡 Hint

Check the 'Tokens Produced' column at step 3 in the execution_table.

At which step does the analyzer actually split the text into tokens?

A Step 3

B Step 1

C Step 2

D Step 5

💡 Hint

Look for the step where 'Tokenize text' happens in the execution_table.

If the input text was 'Quick-Brown Foxes' and the analyzer is 'standard', how would tokens change?

A ["quick-brown", "foxes"]

B ["quick", "brown", "foxes"]

C ["quickbrown", "foxes"]

D ["Quick-Brown", "Foxes"]

💡 Hint

The 'standard' analyzer splits on punctuation and lowercases tokens, similar to step 3 tokens.

Concept Snapshot

Testing analyzers with _analyze API:
- Send POST request to _analyze with 'text' and 'analyzer'
- Elasticsearch tokenizes and filters text
- Returns tokens array
- Useful to see how text is broken down
- Helps debug indexing and search behavior

Full Transcript

This visual trace shows how the Elasticsearch _analyze API processes input text. First, the text 'Quick Brown Foxes' is sent with the 'standard' analyzer. The analyzer lowercases and splits the text into tokens 'quick', 'brown', and 'foxes'. These tokens are returned and displayed. The trace highlights key steps: sending request, applying analyzer, tokenizing, and returning tokens. It also clarifies why tokens are lowercase and how changing analyzers affects output.

Practice

1. What is the main purpose of using the /_analyze API in Elasticsearch?

easy

A. To create a new index

B. To update the index mapping settings

C. To see how input text is broken into tokens by an analyzer

D. To delete documents from an index

Solution

1. Step 1: Understand the role of /_analyze

   The /_analyze API shows how text is processed by an analyzer, including tokenization and filtering.

2. Step 2: Compare with other API functions

   Updating mappings, deleting documents, or creating indexes are done by other APIs, not /_analyze.

3. Final Answer:

   To see how input text is broken into tokens by an analyzer -> Option C

4. Quick Check:

   Analyze API = tokenizes text [OK]

Hint: Remember: /_analyze shows token breakdown [OK]

Common Mistakes:

• Confusing /_analyze with index creation APIs
• Thinking it modifies data instead of analyzing text
• Assuming it deletes or updates documents

2. Which of the following is the correct syntax to test the standard analyzer on the text "Hello World" using the /_analyze API?

easy

A. { "analyzer": "standard", "text": "Hello World" }

B. { "analyzer": "standard" "text": "Hello World" }

C. { "text": "Hello World", "analyzer": standard }

D. { analyzer: "standard", text: "Hello World" }

Solution

1. Step 1: Check JSON syntax for /_analyze request body

   The body must be valid JSON with keys as strings and values properly quoted. { "analyzer": "standard", "text": "Hello World" } uses correct JSON syntax.

2. Step 2: Identify errors in other options

   { "analyzer": "standard" "text": "Hello World" } misses a comma between keys. { "text": "Hello World", "analyzer": standard } lacks quotes around "standard". { analyzer: "standard", text: "Hello World" } uses unquoted keys, invalid in JSON.

3. Final Answer:

   { "analyzer": "standard", "text": "Hello World" } -> Option A

4. Quick Check:

   Valid JSON with quoted keys and values = { "analyzer": "standard", "text": "Hello World" } [OK]

Hint: Always use quotes around keys and string values in JSON [OK]

Common Mistakes:

• Missing commas between JSON fields
• Not quoting string values
• Using unquoted keys in JSON

3. Given this request to /_analyze API with the standard analyzer:

{ "analyzer": "standard", "text": "Quick brown fox" }

What is the expected list of tokens returned?

medium

A. ["quick brown", "fox"]

B. ["quick", "brown", "fox"]

C. ["Quick", "brown", "fox"]

D. ["quick", "brown fox"]

Solution

1. Step 1: Understand standard analyzer behavior

   The standard analyzer lowercases text and splits on whitespace and punctuation, so "Quick brown fox" becomes tokens "quick", "brown", "fox".

2. Step 2: Check options for token case and splitting

   ["quick", "brown", "fox"] shows all lowercase separate tokens. ["Quick", "brown", "fox"] keeps uppercase 'Quick' which standard analyzer lowercases. Options that combine words are incorrect.

3. Final Answer:

   ["quick", "brown", "fox"] -> Option B

4. Quick Check:

   Standard analyzer lowercases and splits words [OK]

Hint: Standard analyzer lowercases and splits on spaces [OK]

Common Mistakes:

• Expecting tokens to keep original case
• Thinking tokens combine multiple words
• Confusing analyzer output with raw text

4. You run this /_analyze request:

{ "analyzer": "whitespace", "text": "Hello, World!" }

But the output tokens include punctuation marks like "Hello," instead of just "Hello". What is the likely cause?

medium

A. Using the whitespace analyzer which only splits on spaces, not removing punctuation

B. The standard analyzer was used instead of whitespace

C. The text was not provided in the request body

D. The analyzer automatically removes punctuation by default

Solution

1. Step 1: Recall whitespace analyzer behavior

   The whitespace analyzer splits text only on spaces and does not remove punctuation, so tokens keep punctuation marks.

2. Step 2: Eliminate other options

   The standard analyzer was used instead of whitespace is incorrect because the standard analyzer removes punctuation. The text was not provided in the request body is invalid because text is provided. The analyzer automatically removes punctuation by default is false; whitespace analyzer does not remove punctuation.

3. Final Answer:

   Using the whitespace analyzer which only splits on spaces, not removing punctuation -> Option A

4. Quick Check:

   Whitespace analyzer keeps punctuation in tokens [OK]

Hint: Whitespace analyzer splits only on spaces, keeps punctuation [OK]

Common Mistakes:

• Expecting punctuation removal with whitespace analyzer
• Confusing analyzer names
• Omitting text in request body

5. You want to test a custom analyzer named "my_custom_analyzer" that uses a lowercase filter and a stopword filter removing "the" and "and". Which /_analyze request body will correctly show tokens for the text "The quick brown fox and the lazy dog"?

hard

A. { "analyzer": "standard", "filter": ["lowercase", "stop"], "text": "The quick brown fox and the lazy dog" }

B. { "analyzer": "my_custom_analyzer", "text": "The quick brown fox and the lazy dog", "stopwords": ["the", "and"] }

C. { "tokenizer": "standard", "filter": ["lowercase", "stop"], "text": "The quick brown fox and the lazy dog" }

D. { "analyzer": "my_custom_analyzer", "text": "The quick brown fox and the lazy dog" }

Solution

1. Step 1: Use the custom analyzer name in the /_analyze request

   To test a custom analyzer, specify its name under the "analyzer" key with the text to analyze. { "analyzer": "my_custom_analyzer", "text": "The quick brown fox and the lazy dog" } does this correctly.

2. Step 2: Understand why other options are incorrect

   { "analyzer": "my_custom_analyzer", "text": "The quick brown fox and the lazy dog", "stopwords": ["the", "and"] } incorrectly tries to add stopwords in the request body, which is not supported here. { "tokenizer": "standard", "filter": ["lowercase", "stop"], "text": "The quick brown fox and the lazy dog" } tries to define tokenizer and filters inline, which is invalid for /_analyze without full analyzer definition. { "analyzer": "standard", "filter": ["lowercase", "stop"], "text": "The quick brown fox and the lazy dog" } uses standard analyzer, not the custom one.

3. Final Answer:

   { "analyzer": "my_custom_analyzer", "text": "The quick brown fox and the lazy dog" } -> Option D

4. Quick Check:

   Use custom analyzer name directly in /_analyze [OK]

Hint: Pass custom analyzer name directly to /_analyze [OK]

Common Mistakes:

• Trying to redefine filters in /_analyze request
• Using standard analyzer instead of custom
• Adding unsupported keys like stopwords in request body