Tokenizers (standard, whitespace, pattern) in Elasticsearch - Mini Project: Build & Apply

Create and Test Tokenizers in Elasticsearch

📖 Scenario: You are setting up an Elasticsearch index for a small library catalog. You want to understand how different tokenizers break down text into searchable words.

🎯 Goal: Build an Elasticsearch index with three different tokenizers: standard, whitespace, and pattern. Then test each tokenizer with the same sample text to see how they split the text into tokens.

📋 What You'll Learn

Create an index called library with a custom analyzer using the standard tokenizer

Add a custom analyzer using the whitespace tokenizer

Add a custom analyzer using the pattern tokenizer with pattern \W+

Test each tokenizer by analyzing the text "Elasticsearch is great, isn't it?"

Print the tokens produced by each tokenizer

💡 Why This Matters

🌍 Real World

Tokenizers help break down text into searchable pieces in search engines like Elasticsearch, improving search accuracy.

💼 Career

Understanding tokenizers is important for roles in search engineering, data indexing, and backend development involving text search.

1

Create the Elasticsearch index with the standard tokenizer analyzer

Write the JSON to create an Elasticsearch index called library with a custom analyzer named standard_analyzer that uses the standard tokenizer.

Elasticsearch

{
  "settings": {
    "analysis": {
      "analyzer": {
        "standard_analyzer": {
          "tokenizer": "standard"
        }
      }
    }
  }
}
# Your code here to create the index with this settings

Hint

Use the PUT method to create the index and define the standard_analyzer inside settings.analysis.analyzer.

2

Add whitespace and pattern tokenizer analyzers to the index settings

Add two more custom analyzers to the library index settings: whitespace_analyzer using the whitespace tokenizer, and pattern_analyzer using the pattern tokenizer with the pattern "\\W+".

Elasticsearch

PUT /library
{
  "settings": {
    "analysis": {
      "analyzer": {
        "standard_analyzer": {
          "tokenizer": "standard"
        },
        "whitespace_analyzer": {
          "tokenizer": "whitespace"
        },
        "pattern_analyzer": {
          "tokenizer": "pattern",
          "filter": [],
          "tokenizer": {
            "type": "pattern",
            "pattern": "\\W+"
          }
        }
      }
    }
  }
}
# Your code here to fix the pattern_analyzer tokenizer definition

Hint

Define the pattern_tokenizer under analysis.tokenizer with type pattern and the given pattern, then use it in pattern_analyzer.

3

Analyze the sample text with each tokenizer

Write three separate _analyze requests to test the text "Elasticsearch is great, isn't it?" using the analyzers standard_analyzer, whitespace_analyzer, and pattern_analyzer.

Elasticsearch

POST /library/_analyze
{
  "analyzer": "standard_analyzer",
  "text": "Elasticsearch is great, isn't it?"
}

# Your code here to analyze with whitespace_analyzer and pattern_analyzer

Hint

Use the _analyze API with the analyzer field set to each analyzer name and the same text.

4

Print the tokens from each analyzer's output

Print the tokens produced by each analyzer from the previous step in this exact format:
Standard tokens: [list]
Whitespace tokens: [list]
Pattern tokens: [list]
Replace [list] with the tokens as a Python list of strings.

Elasticsearch

# Print the tokens from each analyzer's output here
# Your code here

Hint

Use print statements with the exact token lists shown.

Practice

1. Which Elasticsearch tokenizer splits text by spaces only, keeping punctuation as part of tokens?

easy

A. standard tokenizer

B. whitespace tokenizer

C. pattern tokenizer

D. keyword tokenizer

Solution

1. Step 1: Understand whitespace tokenizer behavior

   The whitespace tokenizer splits text only at spaces and keeps punctuation inside tokens.

2. Step 2: Compare with other tokenizers

   The standard tokenizer removes punctuation, and the pattern tokenizer splits by regex patterns, so they behave differently.

3. Final Answer:

   whitespace tokenizer -> Option B

4. Quick Check:

   Spaces split tokens only = whitespace tokenizer [OK]

Hint: Whitespace tokenizer splits only on spaces, keeps punctuation [OK]

Common Mistakes:

• Confusing standard tokenizer with whitespace tokenizer
• Thinking pattern tokenizer splits only on spaces
• Assuming keyword tokenizer splits text

2. Which of the following is the correct JSON syntax to define a pattern tokenizer that splits text on commas?

easy

A. { "tokenizer": "pattern", "pattern": ',' }

B. { 'tokenizer': 'pattern', 'pattern': ',' }

C. { "tokenizer": "pattern", pattern: "," }

D. { "tokenizer": "pattern", "pattern": "," }

Solution

1. Step 1: Identify correct JSON string escaping

   In JSON, double quotes inside strings must be escaped with a backslash, so the comma pattern must be inside escaped quotes.

2. Step 2: Check each option for proper escaping

   { "tokenizer": "pattern", "pattern": "," } correctly shows the pattern as a string with comma inside quotes.

3. Final Answer:

   { "tokenizer": "pattern", "pattern": "," } -> Option D

4. Quick Check:

   Proper JSON escaping for pattern string = { "tokenizer": "pattern", "pattern": "," } [OK]

Hint: Escape quotes inside JSON strings with backslash [OK]

Common Mistakes:

• Not escaping quotes inside JSON strings
• Using single quotes instead of double quotes
• Missing commas or braces in JSON

3. Given the text "Hello, world! Welcome to Elasticsearch.", what tokens does the standard tokenizer produce?

medium

A. ["Hello", "world!", "Welcome", "to", "Elasticsearch"]

B. ["Hello,", "world!", "Welcome", "to", "Elasticsearch."]

C. ["Hello", "world", "Welcome", "to", "Elasticsearch"]

D. ["Hello", "world", "Welcome", "to", "Elasticsearch."]

Solution

1. Step 1: Understand standard tokenizer behavior

   The standard tokenizer splits text by word boundaries and removes punctuation from tokens.

2. Step 2: Apply tokenizer to given text

   It splits "Hello," into "Hello", "world!" into "world", and removes punctuation marks like commas and exclamation points.

3. Final Answer:

   ["Hello", "world", "Welcome", "to", "Elasticsearch"] -> Option C

4. Quick Check:

   Standard tokenizer removes punctuation = ["Hello", "world", "Welcome", "to", "Elasticsearch"] [OK]

Hint: Standard tokenizer removes punctuation, returns clean words [OK]

Common Mistakes:

• Including punctuation in tokens with standard tokenizer
• Confusing output with whitespace tokenizer
• Assuming pattern tokenizer output without pattern

4. You defined a pattern tokenizer with pattern \W+ but it does not split the text as expected. What is the likely issue?

medium

A. The pattern is not properly escaped in JSON

B. Pattern tokenizer cannot use regex

C. Whitespace tokenizer should be used instead

D. Standard tokenizer must be enabled first

Solution

1. Step 1: Check pattern escaping in JSON

   Regex patterns with backslashes must be double escaped in JSON strings, so \W+ should be written as \\W+.

2. Step 2: Understand tokenizer requirements

   If pattern is not escaped properly, the tokenizer won't interpret it as intended, causing wrong splits.

3. Final Answer:

   The pattern is not properly escaped in JSON -> Option A

4. Quick Check:

   Regex backslash must be double escaped in JSON [OK]

Hint: Double escape backslashes in regex patterns for JSON [OK]

Common Mistakes:

• Using single backslash in JSON regex pattern
• Confusing tokenizer types
• Assuming pattern tokenizer needs enabling

5. You want to index a text field so that tokens are split on commas and semicolons only, keeping spaces and punctuation inside tokens. Which tokenizer configuration should you use?

hard

A. Use a pattern tokenizer with pattern [,;]

B. Use the standard tokenizer

C. Use the whitespace tokenizer

D. Use a pattern tokenizer with pattern \s+

Solution

1. Step 1: Identify tokenizer that splits on specific characters

   The pattern tokenizer allows splitting text by a custom regex pattern, so you can specify commas and semicolons.

2. Step 2: Confirm pattern matches commas and semicolons only

   The pattern [,;] matches commas or semicolons, splitting tokens at those characters but keeping spaces and other punctuation inside tokens.

3. Final Answer:

   Use a pattern tokenizer with pattern [,;] -> Option A

4. Quick Check:

   Pattern tokenizer with custom split chars = Use a pattern tokenizer with pattern [,;] [OK]

Hint: Pattern tokenizer splits by your regex pattern [OK]

Common Mistakes:

• Using whitespace tokenizer which splits on spaces
• Using standard tokenizer which removes punctuation
• Using wrong regex pattern for splitting