Custom analyzers in Elasticsearch - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

Output of a custom analyzer with lowercase and stop filters

Given the following Elasticsearch analyzer configuration and input text, what is the output tokens after analysis?

Elasticsearch

{
  "analysis": {
    "analyzer": {
      "my_custom_analyzer": {
        "type": "custom",
        "tokenizer": "standard",
        "filter": ["lowercase", "stop"]
      }
    },
    "filter": {
      "stop": {
        "type": "stop",
        "stopwords": ["the", "is"]
      }
    }
  }
}

Input text: "The Quick Brown Fox is Jumping"

A ["quick", "brown", "fox", "jumping"]

B ["the", "quick", "brown", "fox", "jumping"]

C ["quick", "brown", "fox", "is", "jumping"]

D ["The", "Quick", "Brown", "Fox", "Jumping"]

💡 Hint

Remember that the lowercase filter converts all tokens to lowercase and the stop filter removes specified stopwords.

The tokenizer splits the text into tokens. The lowercase filter converts all tokens to lowercase. The stop filter removes the words "the" and "is". So the output tokens are ["quick", "brown", "fox", "jumping"].

🧠 Conceptual

intermediate

Purpose of a custom analyzer in Elasticsearch

What is the main purpose of creating a custom analyzer in Elasticsearch?

A To store data in a custom format in Elasticsearch

B To create custom visualizations in Kibana

C To define a specific way to tokenize and filter text for indexing and searching

D To manage user access and permissions

💡 Hint

Think about how Elasticsearch processes text for search.

Custom analyzers let you control how text is broken into tokens and how those tokens are filtered before indexing and searching, improving search relevance.

🔧 Debug

advanced

Identify the error in this custom analyzer configuration

What error will occur when trying to create this analyzer in Elasticsearch?

Elasticsearch

{
  "analysis": {
    "analyzer": {
      "my_analyzer": {
        "type": "custom",
        "tokenizer": "standard",
        "filter": ["lowercase", "nonexistent_filter"]
      }
    }
  }
}

A Elasticsearch will raise an error because 'nonexistent_filter' is not defined

B The analyzer will work but ignore 'nonexistent_filter'

C The analyzer will use the default filter instead

D The analyzer will create a new filter named 'nonexistent_filter' automatically

💡 Hint

Check if all filters used in the analyzer are defined in the analysis settings.

Elasticsearch requires all filters used in an analyzer to be defined. Using an undefined filter causes an error.

📝 Syntax

advanced

Correct syntax for defining a custom analyzer with a synonym filter

Which option correctly defines a custom analyzer that uses a synonym filter named 'my_synonyms'?

A

{
  "analysis": {
    "analyzer": {
      "synonym_analyzer": {
        "type": "custom",
        "tokenizer": "standard",
        "filters": ["lowercase", "my_synonyms"]
      }
    },
    "filter": {
      "my_synonyms": {
        "type": "synonym",
        "synonyms": ["quick,fast", "jumps,leaps"]
      }
    }
  }
}

B

{
  "analysis": {
    "analyzer": {
      "synonym_analyzer": {
        "type": "custom",
        "tokenizer": "standard",
        "filter": ["lowercase", "my_synonyms"]
      }
    },
    "filter": {
      "my_synonyms": {
        "type": "synonym",
        "synonyms": ["quick,fast", "jumps,leaps"]
      }
    }
  }
}

C

{
  "analysis": {
    "analyzer": {
      "synonym_analyzer": {
        "type": "custom",
        "tokenizer": "standard",
        "filter": ["lowercase", "my_synonyms"]
      }
    },
    "filters": {
      "my_synonyms": {
        "type": "synonym",
        "synonyms": ["quick,fast", "jumps,leaps"]
      }
    }
  }
}

D

{
  "analysis": {
    "analyzer": {
      "synonym_analyzer": {
        "type": "custom",
        "tokenizer": "standard",
        "filter": ["lowercase", "my_synonyms"]
      }
    },
    "filter": {
      "my_synonyms": {
        "type": "synonym",
        "synonym": ["quick,fast", "jumps,leaps"]
      }
    }
  }
}

💡 Hint

Check the spelling of keys: 'filter' vs 'filters' and 'synonyms' vs 'synonym'.

The correct key for filters is 'filter' (singular) inside the analyzer. The filter definitions must be under 'filter' (not 'filters'). The synonym filter requires the key 'synonyms' (plural) for the list of synonyms.

🚀 Application

expert

Number of tokens produced by a custom analyzer with pattern tokenizer and lowercase filter

Given this custom analyzer configuration, how many tokens will be produced when analyzing the text "Hello, World! Welcome to Elasticsearch."?

Elasticsearch

{
  "analysis": {
    "analyzer": {
      "pattern_analyzer": {
        "type": "custom",
        "tokenizer": "pattern",
        "filter": ["lowercase"]
      }
    },
    "tokenizer": {
      "pattern": {
        "type": "pattern",
        "pattern": "\\W+"
      }
    }
  }
}

A 7

B 6

C 4

D 5

💡 Hint

The pattern tokenizer splits text by non-word characters (\W+). Count the resulting tokens after splitting.

The pattern '\\W+' splits on any sequence of non-word characters. The text splits into: ['Hello', 'World', 'Welcome', 'to', 'Elasticsearch']. After lowercase filter, tokens remain the same count. So total tokens = 5.

Practice

1. What is the main purpose of a custom analyzer in Elasticsearch?

easy

A. To control how text is broken down and processed before indexing

B. To store documents in the database

C. To create visualizations of search results

D. To manage user access permissions

Solution

1. Step 1: Understand the role of analyzers

   Analyzers in Elasticsearch process text by breaking it into tokens and applying filters.

2. Step 2: Identify the purpose of custom analyzers

   Custom analyzers let you define specific tokenizers and filters to control text processing.

3. Final Answer:

   To control how text is broken down and processed before indexing -> Option A

4. Quick Check:

   Custom analyzer = text processing control [OK]

Hint: Custom analyzers shape text before search indexing [OK]

Common Mistakes:

• Confusing analyzers with data storage
• Thinking analyzers create visualizations
• Mixing analyzers with security settings

2. Which of the following is the correct way to define a custom analyzer with a lowercase filter in Elasticsearch settings?

easy

A. "analyzer": {"my_analyzer": {"type": "custom", "tokenizer": "standard", "filter": ["lowercase"]}}

B. "analyzer": {"my_analyzer": {"type": "standard", "tokenizer": "lowercase", "filter": ["custom"]}}

C. "analyzer": {"my_analyzer": {"type": "custom", "tokenizer": "lowercase", "filter": ["standard"]}}

D. "analyzer": {"my_analyzer": {"type": "filter", "tokenizer": "standard", "filter": ["lowercase"]}}

Solution

1. Step 1: Review custom analyzer syntax

   A custom analyzer requires "type": "custom", a tokenizer, and filters as an array.

2. Step 2: Match correct tokenizer and filter

   "standard" tokenizer and "lowercase" filter are valid and correctly placed in "analyzer": {"my_analyzer": {"type": "custom", "tokenizer": "standard", "filter": ["lowercase"]}}.

3. Final Answer:

   "analyzer": {"my_analyzer": {"type": "custom", "tokenizer": "standard", "filter": ["lowercase"]}} -> Option A

4. Quick Check:

   Custom analyzer syntax = "analyzer": {"my_analyzer": {"type": "custom", "tokenizer": "standard", "filter": ["lowercase"]}} [OK]

Hint: Custom analyzer needs type, tokenizer, and filter array [OK]

Common Mistakes:

• Using wrong type like 'filter' instead of 'custom'
• Swapping tokenizer and filter names
• Missing array brackets for filters

3. Given this custom analyzer configuration:

{
  "settings": {
    "analysis": {
      "analyzer": {
        "my_analyzer": {
          "type": "custom",
          "tokenizer": "standard",
          "filter": ["lowercase", "stop"]
        }
      },
      "filter": {
        "stop": {
          "type": "stop",
          "stopwords": ["the", "is"]
        }
      }
    }
  }
}

What is the output tokens when analyzing the text "The cat is on the mat" with my_analyzer?

medium

A. ["The", "cat", "is", "on", "the", "mat"]

B. ["cat", "is", "on", "mat"]

C. ["the", "cat", "is", "on", "the", "mat"]

D. ["cat", "on", "mat"]

Solution

1. Step 1: Apply lowercase filter

   All words become lowercase: "the", "cat", "is", "on", "the", "mat".

2. Step 2: Apply stop filter with stopwords ["the", "is"]

   Remove "the" and "is" from tokens, leaving "cat", "on", "mat".

3. Final Answer:

   ["cat", "on", "mat"] -> Option D

4. Quick Check:

   Lowercase + stop filter removes 'the' and 'is' [OK]

Hint: Lowercase then remove stopwords from tokens [OK]

Common Mistakes:

• Ignoring lowercase filter effect
• Not removing stopwords correctly
• Confusing original text with tokens

4. Identify the error in this custom analyzer definition:

{
  "analysis": {
    "analyzer": {
      "my_analyzer": {
        "type": "custom",
        "tokenizer": "standard",
        "filter": "lowercase"
      }
    }
  }
}

medium

A. Missing stopwords definition

B. The tokenizer name is invalid

C. The filter should be an array, not a string

D. The analyzer type should be "standard"

Solution

1. Step 1: Check filter syntax in custom analyzer

   Filters must be provided as an array, e.g., ["lowercase"], not a string.

2. Step 2: Verify other parts

   Tokenizer "standard" and analyzer type "custom" are correct; stopwords are optional.

3. Final Answer:

   The filter should be an array, not a string -> Option C

4. Quick Check:

   Filter must be array syntax [OK]

Hint: Filters always use array brackets [] in custom analyzers [OK]

Common Mistakes:

• Using string instead of array for filters
• Changing tokenizer to invalid types
• Assuming stopwords are mandatory

5. You want to create a custom analyzer that removes English stopwords, lowercases text, and also applies a stemmer filter named english_stemmer. Which of the following is the correct way to define this analyzer in Elasticsearch settings?

hard

A. {
"analysis": {
"analyzer": {
"custom_english": {
"type": "standard",
"tokenizer": "standard",
"filter": ["lowercase", "stop", "english_stemmer"]
}
},
"filter": {
"stop": {
"type": "stop",
"stopwords": "_english_"
},
"english_stemmer": {
"type": "stemmer",
"language": "english"
}
}
}
}

B. {
"analysis": {
"analyzer": {
"custom_english": {
"type": "custom",
"tokenizer": "standard",
"filter": ["lowercase", "stop", "english_stemmer"]
}
},
"filter": {
"stop": {
"type": "stop",
"stopwords": "_english_"
},
"english_stemmer": {
"type": "stemmer",
"language": "english"
}
}
}
}

C. {
"analysis": {
"analyzer": {
"custom_english": {
"type": "custom",
"tokenizer": "standard",
"filter": ["stop", "lowercase", "english_stemmer"]
}
},
"filter": {
"stop": {
"type": "stop",
"stopwords": "_english_"
},
"english_stemmer": {
"type": "stemmer",
"language": "english"
}
}
}
}

D. {
"analysis": {
"analyzer": {
"custom_english": {
"type": "custom",
"tokenizer": "standard",
"filter": ["lowercase", "english_stemmer", "stop"]
}
},
"filter": {
"stop": {
"type": "stop",
"stopwords": "_english_"
},
"english_stemmer": {
"type": "stemmer",
"language": "english"
}
}
}
}

Solution

1. Step 1: Confirm analyzer type and tokenizer

   The analyzer must be "custom" with "standard" tokenizer to apply filters correctly.

2. Step 2: Check filter order and definitions

   Filters should be ["lowercase", "stop", "english_stemmer"] in that order to lowercase first, remove stopwords next, then stem.

3. Step 3: Verify filter definitions

   Stop filter uses "_english_" stopwords, and stemmer filter is correctly defined with language "english".

4. Final Answer:

   { "analysis": { "analyzer": { "custom_english": { "type": "custom", "tokenizer": "standard", "filter": ["lowercase", "stop", "english_stemmer"] } }, "filter": { "stop": { "type": "stop", "stopwords": "_english_" }, "english_stemmer": { "type": "stemmer", "language": "english" } } } } -> Option B

5. Quick Check:

   Custom analyzer with correct filter order and definitions = { "analysis": { "analyzer": { "custom_english": { "type": "custom", "tokenizer": "standard", "filter": ["lowercase", "stop", "english_stemmer"] } }, "filter": { "stop": { "type": "stop", "stopwords": "_english_" }, "english_stemmer": { "type": "stemmer", "language": "english" } } } } [OK]

Hint: Filter order matters: lowercase, stop, then stemmer [OK]

Common Mistakes:

• Using wrong analyzer type like 'standard'
• Incorrect filter order causing wrong results
• Missing or wrong stopwords definition