Testing analyzers (_analyze API) in Elasticsearch - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

What is the output tokens from this _analyze request?

Given the following Elasticsearch _analyze API request using the standard analyzer, what tokens will be produced?

Elasticsearch

{
  "analyzer": "standard",
  "text": "Quick Brown Foxes"
}

A ["quick", "brown", "foxes"]

B ["Quick", "Brown", "Foxes"]

C ["quick brown foxes"]

D ["quick", "brown", "fox"]

💡 Hint

The standard analyzer lowercases and splits on whitespace and punctuation.

The standard analyzer breaks text into tokens by whitespace and punctuation, then lowercases all tokens. So 'Quick Brown Foxes' becomes ['quick', 'brown', 'foxes'].

❓ Predict Output

intermediate

What tokens does the whitespace analyzer produce?

Using the _analyze API with the whitespace analyzer on the text "Hello, World! Welcome to Elasticsearch.", what tokens are returned?

Elasticsearch

{
  "analyzer": "whitespace",
  "text": "Hello, World! Welcome to Elasticsearch."
}

A ["hello", "world", "welcome", "to", "elasticsearch"]

B ["Hello", "World", "Welcome", "to", "Elasticsearch"]

C ["Hello,", "World!", "Welcome", "to", "Elasticsearch."]

D ["Hello,", "World", "Welcome", "to", "Elasticsearch"]

💡 Hint

Whitespace analyzer splits only on whitespace and does not lowercase or remove punctuation.

The whitespace analyzer splits text only on spaces and does not lowercase or remove punctuation, so tokens keep original casing and punctuation.

❓ Predict Output

advanced

What is the output tokens with a custom analyzer using lowercase and stop filters?

Given this _analyze request with a custom analyzer that uses the standard tokenizer, lowercase filter, and stop filter with stopwords ["the", "is"], what tokens are produced for the text "The quick brown fox is fast"?

Elasticsearch

{
  "tokenizer": "standard",
  "filter": ["lowercase", "stop"],
  "text": "The quick brown fox is fast"
}

A ["quick", "brown", "fox", "fast"]

B ["the", "quick", "brown", "fox", "is", "fast"]

C ["quick", "brown", "fox", "is", "fast"]

D ["The", "quick", "brown", "fox", "fast"]

💡 Hint

Stop filter removes common stopwords after lowercasing.

The standard tokenizer splits words, lowercase filter lowercases all tokens, and stop filter removes 'the' and 'is'. So tokens are ['quick', 'brown', 'fox', 'fast'].

❓ Predict Output

advanced

What error does this _analyze request produce?

What error will Elasticsearch return when running this _analyze request with an invalid tokenizer name?

Elasticsearch

{
  "tokenizer": "nonexistent_tokenizer",
  "text": "Test text"
}

A 500 Internal Server Error

B 400 Bad Request with error 'tokenizer [nonexistent_tokenizer] not found'

C 200 OK with tokens ["test", "text"]

D 404 Not Found

💡 Hint

Elasticsearch returns a 400 error when a requested tokenizer is missing.

If the tokenizer name is invalid, Elasticsearch returns a 400 Bad Request with a message indicating the tokenizer was not found.

🧠 Conceptual

expert

How many tokens are produced by this _analyze request with a pattern tokenizer?

Using the _analyze API with a pattern tokenizer that splits on commas and spaces (pattern: '[,\s]+') on the text "apple, banana orange,pear", how many tokens are produced?

Elasticsearch

{
  "tokenizer": {
    "type": "pattern",
    "pattern": "[,\s]+"
  },
  "text": "apple, banana orange,pear"
}

A 2

B 3

C 5

D 4

💡 Hint

Count tokens split by commas or spaces.

The pattern splits on commas or spaces, so the text splits into ['apple', 'banana', 'orange', 'pear'], which is 4 tokens.

Practice

1. What is the main purpose of using the /_analyze API in Elasticsearch?

easy

A. To create a new index

B. To update the index mapping settings

C. To see how input text is broken into tokens by an analyzer

D. To delete documents from an index

Solution

1. Step 1: Understand the role of /_analyze

   The /_analyze API shows how text is processed by an analyzer, including tokenization and filtering.

2. Step 2: Compare with other API functions

   Updating mappings, deleting documents, or creating indexes are done by other APIs, not /_analyze.

3. Final Answer:

   To see how input text is broken into tokens by an analyzer -> Option C

4. Quick Check:

   Analyze API = tokenizes text [OK]

Hint: Remember: /_analyze shows token breakdown [OK]

Common Mistakes:

• Confusing /_analyze with index creation APIs
• Thinking it modifies data instead of analyzing text
• Assuming it deletes or updates documents

2. Which of the following is the correct syntax to test the standard analyzer on the text "Hello World" using the /_analyze API?

easy

A. { "analyzer": "standard", "text": "Hello World" }

B. { "analyzer": "standard" "text": "Hello World" }

C. { "text": "Hello World", "analyzer": standard }

D. { analyzer: "standard", text: "Hello World" }

Solution

1. Step 1: Check JSON syntax for /_analyze request body

   The body must be valid JSON with keys as strings and values properly quoted. { "analyzer": "standard", "text": "Hello World" } uses correct JSON syntax.

2. Step 2: Identify errors in other options

   { "analyzer": "standard" "text": "Hello World" } misses a comma between keys. { "text": "Hello World", "analyzer": standard } lacks quotes around "standard". { analyzer: "standard", text: "Hello World" } uses unquoted keys, invalid in JSON.

3. Final Answer:

   { "analyzer": "standard", "text": "Hello World" } -> Option A

4. Quick Check:

   Valid JSON with quoted keys and values = { "analyzer": "standard", "text": "Hello World" } [OK]

Hint: Always use quotes around keys and string values in JSON [OK]

Common Mistakes:

• Missing commas between JSON fields
• Not quoting string values
• Using unquoted keys in JSON

3. Given this request to /_analyze API with the standard analyzer:

{ "analyzer": "standard", "text": "Quick brown fox" }

What is the expected list of tokens returned?

medium

A. ["quick brown", "fox"]

B. ["quick", "brown", "fox"]

C. ["Quick", "brown", "fox"]

D. ["quick", "brown fox"]

Solution

1. Step 1: Understand standard analyzer behavior

   The standard analyzer lowercases text and splits on whitespace and punctuation, so "Quick brown fox" becomes tokens "quick", "brown", "fox".

2. Step 2: Check options for token case and splitting

   ["quick", "brown", "fox"] shows all lowercase separate tokens. ["Quick", "brown", "fox"] keeps uppercase 'Quick' which standard analyzer lowercases. Options that combine words are incorrect.

3. Final Answer:

   ["quick", "brown", "fox"] -> Option B

4. Quick Check:

   Standard analyzer lowercases and splits words [OK]

Hint: Standard analyzer lowercases and splits on spaces [OK]

Common Mistakes:

• Expecting tokens to keep original case
• Thinking tokens combine multiple words
• Confusing analyzer output with raw text

4. You run this /_analyze request:

{ "analyzer": "whitespace", "text": "Hello, World!" }

But the output tokens include punctuation marks like "Hello," instead of just "Hello". What is the likely cause?

medium

A. Using the whitespace analyzer which only splits on spaces, not removing punctuation

B. The standard analyzer was used instead of whitespace

C. The text was not provided in the request body

D. The analyzer automatically removes punctuation by default

Solution

1. Step 1: Recall whitespace analyzer behavior

   The whitespace analyzer splits text only on spaces and does not remove punctuation, so tokens keep punctuation marks.

2. Step 2: Eliminate other options

   The standard analyzer was used instead of whitespace is incorrect because the standard analyzer removes punctuation. The text was not provided in the request body is invalid because text is provided. The analyzer automatically removes punctuation by default is false; whitespace analyzer does not remove punctuation.

3. Final Answer:

   Using the whitespace analyzer which only splits on spaces, not removing punctuation -> Option A

4. Quick Check:

   Whitespace analyzer keeps punctuation in tokens [OK]

Hint: Whitespace analyzer splits only on spaces, keeps punctuation [OK]

Common Mistakes:

• Expecting punctuation removal with whitespace analyzer
• Confusing analyzer names
• Omitting text in request body

5. You want to test a custom analyzer named "my_custom_analyzer" that uses a lowercase filter and a stopword filter removing "the" and "and". Which /_analyze request body will correctly show tokens for the text "The quick brown fox and the lazy dog"?

hard

A. { "analyzer": "standard", "filter": ["lowercase", "stop"], "text": "The quick brown fox and the lazy dog" }

B. { "analyzer": "my_custom_analyzer", "text": "The quick brown fox and the lazy dog", "stopwords": ["the", "and"] }

C. { "tokenizer": "standard", "filter": ["lowercase", "stop"], "text": "The quick brown fox and the lazy dog" }

D. { "analyzer": "my_custom_analyzer", "text": "The quick brown fox and the lazy dog" }

Solution

1. Step 1: Use the custom analyzer name in the /_analyze request

   To test a custom analyzer, specify its name under the "analyzer" key with the text to analyze. { "analyzer": "my_custom_analyzer", "text": "The quick brown fox and the lazy dog" } does this correctly.

2. Step 2: Understand why other options are incorrect

   { "analyzer": "my_custom_analyzer", "text": "The quick brown fox and the lazy dog", "stopwords": ["the", "and"] } incorrectly tries to add stopwords in the request body, which is not supported here. { "tokenizer": "standard", "filter": ["lowercase", "stop"], "text": "The quick brown fox and the lazy dog" } tries to define tokenizer and filters inline, which is invalid for /_analyze without full analyzer definition. { "analyzer": "standard", "filter": ["lowercase", "stop"], "text": "The quick brown fox and the lazy dog" } uses standard analyzer, not the custom one.

3. Final Answer:

   { "analyzer": "my_custom_analyzer", "text": "The quick brown fox and the lazy dog" } -> Option D

4. Quick Check:

   Use custom analyzer name directly in /_analyze [OK]

Hint: Pass custom analyzer name directly to /_analyze [OK]

Common Mistakes:

• Trying to redefine filters in /_analyze request
• Using standard analyzer instead of custom
• Adding unsupported keys like stopwords in request body