Analyzer components (tokenizer, filters) in Elasticsearch - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

What is the output of this Elasticsearch analyzer test?

Given the following analyzer configuration and input text, what is the list of tokens produced?

Elasticsearch

{
  "settings": {
    "analysis": {
      "analyzer": {
        "my_analyzer": {
          "tokenizer": "standard",
          "filter": ["lowercase", "stop"]
        }
      },
      "filter": {
        "stop": {
          "type": "stop",
          "stopwords": ["the", "is"]
        }
      }
    }
  }
}

Input text: "The quick brown fox is jumping"

A ["quick", "brown", "fox", "is", "jumping"]

B ["The", "quick", "brown", "fox", "jumping"]

C ["quick", "brown", "fox", "jumping"]

D ["the", "quick", "brown", "fox", "is", "jumping"]

💡 Hint

Remember that the stop filter removes specified stopwords after tokenizing and lowercasing.

The standard tokenizer splits the text into tokens. The lowercase filter converts all tokens to lowercase. The stop filter removes the words "the" and "is". So the tokens "the" and "is" are removed, leaving only "quick", "brown", "fox", and "jumping".

❓ Predict Output

intermediate

What tokens result from this custom analyzer with a pattern tokenizer?

Given this analyzer configuration and input, what tokens are produced?

Elasticsearch

{
  "settings": {
    "analysis": {
      "analyzer": {
        "pattern_analyzer": {
          "tokenizer": "pattern",
          "filter": ["lowercase"]
        }
      },
      "tokenizer": {
        "pattern": {
          "type": "pattern",
          "pattern": "\\W+"
        }
      }
    }
  }
}

Input text: "Hello, World! Welcome to Elasticsearch."

A ["hello", "world", "welcome", "to", "elasticsearch", ""]

B ["hello", "world", "welcome", "to", "elasticsearch"]

C ["Hello", "World", "Welcome", "to", "Elasticsearch", ""]

D ["Hello", "World", "Welcome", "to", "Elasticsearch"]

💡 Hint

The pattern tokenizer splits on non-word characters, and the lowercase filter converts tokens to lowercase.

The pattern tokenizer splits the text on any sequence of non-word characters (like spaces, commas, punctuation). This produces tokens without punctuation. The lowercase filter converts all tokens to lowercase. No empty tokens remain because the pattern splits cleanly.

🔧 Debug

advanced

Why does this analyzer configuration cause an error?

This analyzer configuration causes an error when indexing. What is the cause?

Elasticsearch

{
  "settings": {
    "analysis": {
      "analyzer": {
        "bad_analyzer": {
          "tokenizer": "standard",
          "filter": ["lowercase", "nonexistent_filter"]
        }
      }
    }
  }
}

A The filter "nonexistent_filter" is not defined, causing a configuration error.

B The standard tokenizer is deprecated and causes errors.

C The lowercase filter must be defined explicitly in the filter section.

D The analyzer must have at least two tokenizers defined.

💡 Hint

Check if all filters used are defined in the analysis settings.

Elasticsearch requires that all filters used in an analyzer be defined or built-in. "nonexistent_filter" is not defined anywhere, so Elasticsearch throws a configuration error.

❓ Predict Output

advanced

What is the output tokens of this analyzer with a synonym filter?

Given this analyzer configuration and input, what tokens are produced?

Elasticsearch

{
  "settings": {
    "analysis": {
      "analyzer": {
        "synonym_analyzer": {
          "tokenizer": "standard",
          "filter": ["lowercase", "my_synonym"]
        }
      },
      "filter": {
        "my_synonym": {
          "type": "synonym",
          "synonyms": ["quick,fast"]
        }
      }
    }
  }
}

Input text: "The quick fox"

A ["the", "quick", "fast", "fox"]

B ["the", "quick", "fox"]

C ["the", "fast", "fox"]

D ["quick", "fast", "fox"]

💡 Hint

Synonym filters add synonyms as extra tokens, not replace tokens.

The synonym filter adds "fast" as a synonym token for "quick". The lowercase filter converts all tokens to lowercase. The standard tokenizer splits the text into words. So tokens are "the", "quick", "fast", "fox".

🧠 Conceptual

expert

Which filter order produces this token output?

You want to produce tokens from the input "Running runs run" that are stemmed and lowercase, but stopwords are removed after stemming. Which filter order in the analyzer produces the tokens ["run", "run", "run"]?

A filter: ["lowercase", "stop", "stemmer"]

B filter: ["stemmer", "lowercase", "stop"]

C filter: ["stop", "lowercase", "stemmer"]

D filter: ["lowercase", "stemmer", "stop"]

💡 Hint

Think about when stopwords are removed relative to stemming and lowercasing.

The input tokens are first lowercased, then stemmed to "run", then stopwords are removed. Since "run" is not a stopword, all tokens remain. If stop was before stemmer, "running" and "runs" would not be stemmed before stopword removal, possibly causing different results.