Testing analyzers (_analyze API) in Elasticsearch

Introduction

Testing analyzers helps you see how text is broken down and processed before searching. It shows how your search engine understands words.

You want to check how your text is split into words (tokens).

You want to see which parts of words are kept or removed.

You want to test if your custom analyzer works as expected.

You want to debug why some search results are missing.

You want to understand how filters like lowercase or stop words affect text.

Syntax

Elasticsearch

POST /_analyze
{
  "analyzer": "standard",
  "text": "Your text here"
}

Use POST method with the /_analyze endpoint.

Specify the analyzer name and the text to test inside the JSON body.

Examples

This tests the built-in standard analyzer on the text.

Elasticsearch

POST /_analyze
{
  "analyzer": "standard",
  "text": "Quick Brown Fox"
}

This uses a whitespace tokenizer and lowercase filter to see tokens.

Elasticsearch

POST /_analyze
{
  "tokenizer": "whitespace",
  "filter": ["lowercase"],
  "text": "Hello World"
}

This tests the analyzer configured in your index my_index.

Elasticsearch

POST /my_index/_analyze
{
  "text": "Testing custom analyzer"
}

Sample Program

This example sends a request to Elasticsearch to analyze the sentence using the standard analyzer. It will show how the sentence is split into tokens.

Elasticsearch

POST /_analyze
{
  "analyzer": "standard",
  "text": "The Quick Brown Fox"
}

Important Notes

You can test any analyzer or tokenizer configured in your Elasticsearch setup.

Use this API to quickly check how text is processed without indexing data.

Summary

Use /_analyze to see how text is split into tokens.

It helps debug and understand your search text processing.

You can test built-in or custom analyzers easily.

Practice

1. What is the main purpose of using the /_analyze API in Elasticsearch?

easy

A. To create a new index

B. To update the index mapping settings

C. To see how input text is broken into tokens by an analyzer

D. To delete documents from an index

Solution

1. Step 1: Understand the role of /_analyze

   The /_analyze API shows how text is processed by an analyzer, including tokenization and filtering.

2. Step 2: Compare with other API functions

   Updating mappings, deleting documents, or creating indexes are done by other APIs, not /_analyze.

3. Final Answer:

   To see how input text is broken into tokens by an analyzer -> Option C

4. Quick Check:

   Analyze API = tokenizes text [OK]

Hint: Remember: /_analyze shows token breakdown [OK]

Common Mistakes:

• Confusing /_analyze with index creation APIs
• Thinking it modifies data instead of analyzing text
• Assuming it deletes or updates documents

2. Which of the following is the correct syntax to test the standard analyzer on the text "Hello World" using the /_analyze API?

easy

A. { "analyzer": "standard", "text": "Hello World" }

B. { "analyzer": "standard" "text": "Hello World" }

C. { "text": "Hello World", "analyzer": standard }

D. { analyzer: "standard", text: "Hello World" }

Solution

1. Step 1: Check JSON syntax for /_analyze request body

   The body must be valid JSON with keys as strings and values properly quoted. { "analyzer": "standard", "text": "Hello World" } uses correct JSON syntax.

2. Step 2: Identify errors in other options

   { "analyzer": "standard" "text": "Hello World" } misses a comma between keys. { "text": "Hello World", "analyzer": standard } lacks quotes around "standard". { analyzer: "standard", text: "Hello World" } uses unquoted keys, invalid in JSON.

3. Final Answer:

   { "analyzer": "standard", "text": "Hello World" } -> Option A

4. Quick Check:

   Valid JSON with quoted keys and values = { "analyzer": "standard", "text": "Hello World" } [OK]

Hint: Always use quotes around keys and string values in JSON [OK]

Common Mistakes:

• Missing commas between JSON fields
• Not quoting string values
• Using unquoted keys in JSON

3. Given this request to /_analyze API with the standard analyzer:

{ "analyzer": "standard", "text": "Quick brown fox" }

What is the expected list of tokens returned?

medium

A. ["quick brown", "fox"]

B. ["quick", "brown", "fox"]

C. ["Quick", "brown", "fox"]

D. ["quick", "brown fox"]

Solution

1. Step 1: Understand standard analyzer behavior

   The standard analyzer lowercases text and splits on whitespace and punctuation, so "Quick brown fox" becomes tokens "quick", "brown", "fox".

2. Step 2: Check options for token case and splitting

   ["quick", "brown", "fox"] shows all lowercase separate tokens. ["Quick", "brown", "fox"] keeps uppercase 'Quick' which standard analyzer lowercases. Options that combine words are incorrect.

3. Final Answer:

   ["quick", "brown", "fox"] -> Option B

4. Quick Check:

   Standard analyzer lowercases and splits words [OK]

Hint: Standard analyzer lowercases and splits on spaces [OK]

Common Mistakes:

• Expecting tokens to keep original case
• Thinking tokens combine multiple words
• Confusing analyzer output with raw text

4. You run this /_analyze request:

{ "analyzer": "whitespace", "text": "Hello, World!" }

But the output tokens include punctuation marks like "Hello," instead of just "Hello". What is the likely cause?

medium

A. Using the whitespace analyzer which only splits on spaces, not removing punctuation

B. The standard analyzer was used instead of whitespace

C. The text was not provided in the request body

D. The analyzer automatically removes punctuation by default

Solution

1. Step 1: Recall whitespace analyzer behavior

   The whitespace analyzer splits text only on spaces and does not remove punctuation, so tokens keep punctuation marks.

2. Step 2: Eliminate other options

   The standard analyzer was used instead of whitespace is incorrect because the standard analyzer removes punctuation. The text was not provided in the request body is invalid because text is provided. The analyzer automatically removes punctuation by default is false; whitespace analyzer does not remove punctuation.

3. Final Answer:

   Using the whitespace analyzer which only splits on spaces, not removing punctuation -> Option A

4. Quick Check:

   Whitespace analyzer keeps punctuation in tokens [OK]

Hint: Whitespace analyzer splits only on spaces, keeps punctuation [OK]

Common Mistakes:

• Expecting punctuation removal with whitespace analyzer
• Confusing analyzer names
• Omitting text in request body

5. You want to test a custom analyzer named "my_custom_analyzer" that uses a lowercase filter and a stopword filter removing "the" and "and". Which /_analyze request body will correctly show tokens for the text "The quick brown fox and the lazy dog"?

hard

A. { "analyzer": "standard", "filter": ["lowercase", "stop"], "text": "The quick brown fox and the lazy dog" }

B. { "analyzer": "my_custom_analyzer", "text": "The quick brown fox and the lazy dog", "stopwords": ["the", "and"] }

C. { "tokenizer": "standard", "filter": ["lowercase", "stop"], "text": "The quick brown fox and the lazy dog" }

D. { "analyzer": "my_custom_analyzer", "text": "The quick brown fox and the lazy dog" }

Solution

1. Step 1: Use the custom analyzer name in the /_analyze request

   To test a custom analyzer, specify its name under the "analyzer" key with the text to analyze. { "analyzer": "my_custom_analyzer", "text": "The quick brown fox and the lazy dog" } does this correctly.

2. Step 2: Understand why other options are incorrect

   { "analyzer": "my_custom_analyzer", "text": "The quick brown fox and the lazy dog", "stopwords": ["the", "and"] } incorrectly tries to add stopwords in the request body, which is not supported here. { "tokenizer": "standard", "filter": ["lowercase", "stop"], "text": "The quick brown fox and the lazy dog" } tries to define tokenizer and filters inline, which is invalid for /_analyze without full analyzer definition. { "analyzer": "standard", "filter": ["lowercase", "stop"], "text": "The quick brown fox and the lazy dog" } uses standard analyzer, not the custom one.

3. Final Answer:

   { "analyzer": "my_custom_analyzer", "text": "The quick brown fox and the lazy dog" } -> Option D

4. Quick Check:

   Use custom analyzer name directly in /_analyze [OK]

Hint: Pass custom analyzer name directly to /_analyze [OK]

Common Mistakes:

• Trying to redefine filters in /_analyze request
• Using standard analyzer instead of custom
• Adding unsupported keys like stopwords in request body