Analyzer components (tokenizer, filters) in Elasticsearch - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

What is the output of this Elasticsearch analyzer test?

Given the following analyzer configuration and input text, what is the list of tokens produced?

Elasticsearch

{
  "settings": {
    "analysis": {
      "analyzer": {
        "my_analyzer": {
          "tokenizer": "standard",
          "filter": ["lowercase", "stop"]
        }
      },
      "filter": {
        "stop": {
          "type": "stop",
          "stopwords": ["the", "is"]
        }
      }
    }
  }
}

Input text: "The quick brown fox is jumping"

A ["quick", "brown", "fox", "is", "jumping"]

B ["The", "quick", "brown", "fox", "jumping"]

C ["quick", "brown", "fox", "jumping"]

D ["the", "quick", "brown", "fox", "is", "jumping"]

💡 Hint

Remember that the stop filter removes specified stopwords after tokenizing and lowercasing.

The standard tokenizer splits the text into tokens. The lowercase filter converts all tokens to lowercase. The stop filter removes the words "the" and "is". So the tokens "the" and "is" are removed, leaving only "quick", "brown", "fox", and "jumping".

❓ Predict Output

intermediate

What tokens result from this custom analyzer with a pattern tokenizer?

Given this analyzer configuration and input, what tokens are produced?

Elasticsearch

{
  "settings": {
    "analysis": {
      "analyzer": {
        "pattern_analyzer": {
          "tokenizer": "pattern",
          "filter": ["lowercase"]
        }
      },
      "tokenizer": {
        "pattern": {
          "type": "pattern",
          "pattern": "\\W+"
        }
      }
    }
  }
}

Input text: "Hello, World! Welcome to Elasticsearch."

A ["hello", "world", "welcome", "to", "elasticsearch", ""]

B ["hello", "world", "welcome", "to", "elasticsearch"]

C ["Hello", "World", "Welcome", "to", "Elasticsearch", ""]

D ["Hello", "World", "Welcome", "to", "Elasticsearch"]

💡 Hint

The pattern tokenizer splits on non-word characters, and the lowercase filter converts tokens to lowercase.

The pattern tokenizer splits the text on any sequence of non-word characters (like spaces, commas, punctuation). This produces tokens without punctuation. The lowercase filter converts all tokens to lowercase. No empty tokens remain because the pattern splits cleanly.

🔧 Debug

advanced

Why does this analyzer configuration cause an error?

This analyzer configuration causes an error when indexing. What is the cause?

Elasticsearch

{
  "settings": {
    "analysis": {
      "analyzer": {
        "bad_analyzer": {
          "tokenizer": "standard",
          "filter": ["lowercase", "nonexistent_filter"]
        }
      }
    }
  }
}

A The filter "nonexistent_filter" is not defined, causing a configuration error.

B The standard tokenizer is deprecated and causes errors.

C The lowercase filter must be defined explicitly in the filter section.

D The analyzer must have at least two tokenizers defined.

💡 Hint

Check if all filters used are defined in the analysis settings.

Elasticsearch requires that all filters used in an analyzer be defined or built-in. "nonexistent_filter" is not defined anywhere, so Elasticsearch throws a configuration error.

❓ Predict Output

advanced

What is the output tokens of this analyzer with a synonym filter?

Given this analyzer configuration and input, what tokens are produced?

Elasticsearch

{
  "settings": {
    "analysis": {
      "analyzer": {
        "synonym_analyzer": {
          "tokenizer": "standard",
          "filter": ["lowercase", "my_synonym"]
        }
      },
      "filter": {
        "my_synonym": {
          "type": "synonym",
          "synonyms": ["quick,fast"]
        }
      }
    }
  }
}

Input text: "The quick fox"

A ["the", "quick", "fast", "fox"]

B ["the", "quick", "fox"]

C ["the", "fast", "fox"]

D ["quick", "fast", "fox"]

💡 Hint

Synonym filters add synonyms as extra tokens, not replace tokens.

The synonym filter adds "fast" as a synonym token for "quick". The lowercase filter converts all tokens to lowercase. The standard tokenizer splits the text into words. So tokens are "the", "quick", "fast", "fox".

🧠 Conceptual

expert

Which filter order produces this token output?

You want to produce tokens from the input "Running runs run" that are stemmed and lowercase, but stopwords are removed after stemming. Which filter order in the analyzer produces the tokens ["run", "run", "run"]?

A filter: ["lowercase", "stop", "stemmer"]

B filter: ["stemmer", "lowercase", "stop"]

C filter: ["stop", "lowercase", "stemmer"]

D filter: ["lowercase", "stemmer", "stop"]

💡 Hint

Think about when stopwords are removed relative to stemming and lowercasing.

The input tokens are first lowercased, then stemmed to "run", then stopwords are removed. Since "run" is not a stopword, all tokens remain. If stop was before stemmer, "running" and "runs" would not be stemmed before stopword removal, possibly causing different results.

Practice

1. What is the main role of a tokenizer in an Elasticsearch analyzer?

easy

A. To remove stop words from the text

B. To split text into smaller pieces called tokens

C. To convert tokens to lowercase

D. To index the tokens for searching

Solution

1. Step 1: Understand the tokenizer's function

   A tokenizer breaks the input text into smaller parts called tokens, which are the basic units for searching.

2. Step 2: Differentiate tokenizer from filters

   Filters modify tokens after tokenization, such as lowercasing or removing stop words, but tokenizers do the initial splitting.

3. Final Answer:

   To split text into smaller pieces called tokens -> Option B

4. Quick Check:

   Tokenizer = splits text [OK]

Hint: Tokenizer always splits text into tokens first [OK]

Common Mistakes:

• Confusing tokenizer with filters like lowercase or stop word removal
• Thinking tokenizer indexes tokens
• Assuming tokenizer modifies tokens after splitting

2. Which of the following is the correct syntax to define a standard tokenizer in an Elasticsearch analyzer settings?

easy

A. "tokenizer": standard

B. "tokenizer": ["standard"]

C. "tokenizer": { "type": "standard" }

D. "tokenizer": { "name": "standard" }

Solution

1. Step 1: Recall Elasticsearch tokenizer syntax

   Tokenizers are defined as objects with a type field specifying the tokenizer type.

2. Step 2: Compare options

   "tokenizer": { "type": "standard" } correctly uses an object with type key, while others use incorrect formats like strings or arrays.

3. Final Answer:

   "tokenizer": { "type": "standard" } -> Option C

4. Quick Check:

   Tokenizer syntax uses type key in object [OK]

Hint: Tokenizer must be an object with type key [OK]

Common Mistakes:

• Using string directly instead of object with type
• Using array syntax for tokenizer
• Using incorrect key names like name instead of type

3. Given this analyzer configuration snippet:

{
  "tokenizer": "whitespace",
  "filter": ["lowercase", "stop"]
}

What will be the output tokens for the input text: "The Quick Brown Fox"?

medium

A. ["quick", "brown", "fox"]

B. ["The", "Quick", "Brown", "Fox"]

C. ["quick", "brown"]

D. ["the", "quick", "brown", "fox"]

Solution

1. Step 1: Apply the whitespace tokenizer

   The whitespace tokenizer splits text by spaces, so tokens are ["The", "Quick", "Brown", "Fox"].

2. Step 2: Apply lowercase and stop filters

   The lowercase filter converts all tokens to lowercase: ["the", "quick", "brown", "fox"]. The stop filter removes common stop words like "the", leaving ["quick", "brown", "fox"].

3. Final Answer:

   ["quick", "brown", "fox"] -> Option A

4. Quick Check:

   Lowercase + stop filter removes "the" [OK]

Hint: Stop filter removes common words after lowercase [OK]

Common Mistakes:

• Ignoring stop word removal
• Not applying lowercase filter before stop filter
• Assuming tokenizer changes case

4. Identify the error in this analyzer filter configuration:

{
  "filter": [
    { "type": "lowercase" },
    { "type": "stop", "stopwords": "_english_" },
    { "type": "unknown_filter" }
  ]
}

medium

A. Filters cannot be defined as objects

B. The lowercase filter must be a string, not an object

C. The stop filter syntax is incorrect

D. The filter list contains an invalid filter type

Solution

1. Step 1: Review each filter type

   Lowercase and stop filters are valid and correctly defined. The stop filter uses a valid stopwords parameter.

2. Step 2: Identify invalid filter

   The filter with type "unknown_filter" is not a valid Elasticsearch filter type and will cause an error.

3. Final Answer:

   The filter list contains an invalid filter type -> Option D

4. Quick Check:

   Invalid filter type causes error [OK]

Hint: Check filter types are valid Elasticsearch filters [OK]

Common Mistakes:

• Assuming all filter types are valid
• Thinking lowercase filter must be string
• Ignoring unknown filter errors

5. You want to create a custom analyzer that splits text on whitespace, lowercases tokens, and removes English stop words. Which of the following analyzer settings is correct?

hard

A. { "tokenizer": { "type": "whitespace" }, "filter": ["lowercase", { "type": "stop", "stopwords": "_english_" }] }

B. { "tokenizer": "whitespace", "filter": ["lowercase", stop] }

C. { "tokenizer": "standard", "filter": ["lowercase", "stop"] }

D. { "tokenizer": { "type": "whitespace" }, "filter": ["stop", "lowercase"] }

Solution

1. Step 1: Confirm tokenizer type

   The tokenizer must be whitespace type, correctly defined as an object with type "whitespace".

2. Step 2: Confirm filter order and definition

   Filters should lowercase first, then remove stop words. { "tokenizer": { "type": "whitespace" }, "filter": ["lowercase", { "type": "stop", "stopwords": "_english_" }] } correctly defines lowercase as string and stop filter with explicit stopwords parameter.

3. Step 3: Check other options

   { "tokenizer": "whitespace", "filter": ["lowercase", "stop"] } uses tokenizer as string which is allowed but less explicit; { "tokenizer": "standard", "filter": ["lowercase", "stop"] } uses standard tokenizer which is incorrect; { "tokenizer": { "type": "whitespace" }, "filter": ["stop", "lowercase"] } reverses filter order which may affect results.

4. Final Answer:

   { "tokenizer": { "type": "whitespace" }, "filter": ["lowercase", { "type": "stop", "stopwords": "_english_" }] } -> Option A

5. Quick Check:

   Explicit tokenizer and correct filter order [OK]

Hint: Use object for tokenizer and correct filter order [OK]

Common Mistakes:

• Using wrong tokenizer type
• Reversing filter order
• Not specifying stopwords explicitly