Testing analyzers (_analyze API) in Elasticsearch - Mini Project: Build & Apply

Testing analyzers (_analyze API)

📖 Scenario: You are working with Elasticsearch, a search engine that breaks down text into smaller parts called tokens. These tokens help search work better. To see how Elasticsearch breaks down text, you use the _analyze API.

🎯 Goal: You will create a simple request to test how Elasticsearch analyzes a piece of text using the _analyze API. You will set up the text, choose an analyzer, send the request, and see the tokens it produces.

📋 What You'll Learn

Create a JSON object with the text to analyze

Add a field to specify the analyzer to use

Send the request to the _analyze API

Print the tokens returned by the API

💡 Why This Matters

🌍 Real World

Testing analyzers helps developers understand how their text data is processed before searching, improving search accuracy and relevance.

💼 Career

Knowledge of Elasticsearch analyzers and the _analyze API is important for roles in search engineering, backend development, and data engineering.

1

DATA SETUP: Create the text to analyze

Create a JSON object called request_body with a field text set to the string "Quick Brown Foxes".

Elasticsearch

request_body = {
    "text": "Quick Brown Foxes"
}
# Your code here

Hint

Use a Python dictionary with the key text and the exact string value.

2

CONFIGURATION: Add the analyzer to the request

Add a field analyzer to the request_body dictionary and set it to the string "standard".

Elasticsearch

request_body = {
    "text": "Quick Brown Foxes",
    # Add the analyzer field here
}
# Your code here

Hint

Add the analyzer key with value "standard" inside the dictionary.

3

CORE LOGIC: Send the analyze request

Use the client.indices.analyze method to send request_body and store the result in a variable called response.

Elasticsearch

response = client.indices.analyze(body=request_body)
# Your code here

Hint

Call client.indices.analyze with body=request_body and save the result as response.

4

OUTPUT: Print the tokens from the response

Print the list of tokens from response by accessing response["tokens"].

Elasticsearch

print(response["tokens"])
# Your code here

Hint

Print response["tokens"] to see the tokens generated by the analyzer.

Practice

1. What is the main purpose of using the /_analyze API in Elasticsearch?

easy

A. To create a new index

B. To update the index mapping settings

C. To see how input text is broken into tokens by an analyzer

D. To delete documents from an index

Solution

1. Step 1: Understand the role of /_analyze

   The /_analyze API shows how text is processed by an analyzer, including tokenization and filtering.

2. Step 2: Compare with other API functions

   Updating mappings, deleting documents, or creating indexes are done by other APIs, not /_analyze.

3. Final Answer:

   To see how input text is broken into tokens by an analyzer -> Option C

4. Quick Check:

   Analyze API = tokenizes text [OK]

Hint: Remember: /_analyze shows token breakdown [OK]

Common Mistakes:

• Confusing /_analyze with index creation APIs
• Thinking it modifies data instead of analyzing text
• Assuming it deletes or updates documents

2. Which of the following is the correct syntax to test the standard analyzer on the text "Hello World" using the /_analyze API?

easy

A. { "analyzer": "standard", "text": "Hello World" }

B. { "analyzer": "standard" "text": "Hello World" }

C. { "text": "Hello World", "analyzer": standard }

D. { analyzer: "standard", text: "Hello World" }

Solution

1. Step 1: Check JSON syntax for /_analyze request body

   The body must be valid JSON with keys as strings and values properly quoted. { "analyzer": "standard", "text": "Hello World" } uses correct JSON syntax.

2. Step 2: Identify errors in other options

   { "analyzer": "standard" "text": "Hello World" } misses a comma between keys. { "text": "Hello World", "analyzer": standard } lacks quotes around "standard". { analyzer: "standard", text: "Hello World" } uses unquoted keys, invalid in JSON.

3. Final Answer:

   { "analyzer": "standard", "text": "Hello World" } -> Option A

4. Quick Check:

   Valid JSON with quoted keys and values = { "analyzer": "standard", "text": "Hello World" } [OK]

Hint: Always use quotes around keys and string values in JSON [OK]

Common Mistakes:

• Missing commas between JSON fields
• Not quoting string values
• Using unquoted keys in JSON

3. Given this request to /_analyze API with the standard analyzer:

{ "analyzer": "standard", "text": "Quick brown fox" }

What is the expected list of tokens returned?

medium

A. ["quick brown", "fox"]

B. ["quick", "brown", "fox"]

C. ["Quick", "brown", "fox"]

D. ["quick", "brown fox"]

Solution

1. Step 1: Understand standard analyzer behavior

   The standard analyzer lowercases text and splits on whitespace and punctuation, so "Quick brown fox" becomes tokens "quick", "brown", "fox".

2. Step 2: Check options for token case and splitting

   ["quick", "brown", "fox"] shows all lowercase separate tokens. ["Quick", "brown", "fox"] keeps uppercase 'Quick' which standard analyzer lowercases. Options that combine words are incorrect.

3. Final Answer:

   ["quick", "brown", "fox"] -> Option B

4. Quick Check:

   Standard analyzer lowercases and splits words [OK]

Hint: Standard analyzer lowercases and splits on spaces [OK]

Common Mistakes:

• Expecting tokens to keep original case
• Thinking tokens combine multiple words
• Confusing analyzer output with raw text

4. You run this /_analyze request:

{ "analyzer": "whitespace", "text": "Hello, World!" }

But the output tokens include punctuation marks like "Hello," instead of just "Hello". What is the likely cause?

medium

A. Using the whitespace analyzer which only splits on spaces, not removing punctuation

B. The standard analyzer was used instead of whitespace

C. The text was not provided in the request body

D. The analyzer automatically removes punctuation by default

Solution

1. Step 1: Recall whitespace analyzer behavior

   The whitespace analyzer splits text only on spaces and does not remove punctuation, so tokens keep punctuation marks.

2. Step 2: Eliminate other options

   The standard analyzer was used instead of whitespace is incorrect because the standard analyzer removes punctuation. The text was not provided in the request body is invalid because text is provided. The analyzer automatically removes punctuation by default is false; whitespace analyzer does not remove punctuation.

3. Final Answer:

   Using the whitespace analyzer which only splits on spaces, not removing punctuation -> Option A

4. Quick Check:

   Whitespace analyzer keeps punctuation in tokens [OK]

Hint: Whitespace analyzer splits only on spaces, keeps punctuation [OK]

Common Mistakes:

• Expecting punctuation removal with whitespace analyzer
• Confusing analyzer names
• Omitting text in request body

5. You want to test a custom analyzer named "my_custom_analyzer" that uses a lowercase filter and a stopword filter removing "the" and "and". Which /_analyze request body will correctly show tokens for the text "The quick brown fox and the lazy dog"?

hard

A. { "analyzer": "standard", "filter": ["lowercase", "stop"], "text": "The quick brown fox and the lazy dog" }

B. { "analyzer": "my_custom_analyzer", "text": "The quick brown fox and the lazy dog", "stopwords": ["the", "and"] }

C. { "tokenizer": "standard", "filter": ["lowercase", "stop"], "text": "The quick brown fox and the lazy dog" }

D. { "analyzer": "my_custom_analyzer", "text": "The quick brown fox and the lazy dog" }

Solution

1. Step 1: Use the custom analyzer name in the /_analyze request

   To test a custom analyzer, specify its name under the "analyzer" key with the text to analyze. { "analyzer": "my_custom_analyzer", "text": "The quick brown fox and the lazy dog" } does this correctly.

2. Step 2: Understand why other options are incorrect

   { "analyzer": "my_custom_analyzer", "text": "The quick brown fox and the lazy dog", "stopwords": ["the", "and"] } incorrectly tries to add stopwords in the request body, which is not supported here. { "tokenizer": "standard", "filter": ["lowercase", "stop"], "text": "The quick brown fox and the lazy dog" } tries to define tokenizer and filters inline, which is invalid for /_analyze without full analyzer definition. { "analyzer": "standard", "filter": ["lowercase", "stop"], "text": "The quick brown fox and the lazy dog" } uses standard analyzer, not the custom one.

3. Final Answer:

   { "analyzer": "my_custom_analyzer", "text": "The quick brown fox and the lazy dog" } -> Option D

4. Quick Check:

   Use custom analyzer name directly in /_analyze [OK]

Hint: Pass custom analyzer name directly to /_analyze [OK]

Common Mistakes:

• Trying to redefine filters in /_analyze request
• Using standard analyzer instead of custom
• Adding unsupported keys like stopwords in request body