Tokenizers (standard, whitespace, pattern) in Elasticsearch - Step-by-Step Execution

Concept Flow - Tokenizers (standard, whitespace, pattern)

Input Text

↓

Choose Tokenizer Type

↓

Standard

↓

Tokens Output

↓

Used in Analysis

Text goes into a tokenizer, which splits it into tokens based on rules: standard splits on punctuation, whitespace splits on spaces, pattern splits by a regex.

Execution Sample

Elasticsearch

POST _analyze
{
  "tokenizer": "whitespace",
  "text": "Hello, world! This is Elasticsearch."
}

This example uses the whitespace tokenizer to split text into tokens separated by spaces.

Execution Table

Step	Input Text	Tokenizer Type	Tokenization Rule	Tokens Produced
1	Hello, world! This is Elasticsearch.	whitespace	Split by spaces	["Hello,", "world!", "This", "is", "Elasticsearch."]
2	Hello, world! This is Elasticsearch.	standard	Split by punctuation and spaces	["Hello", "world", "This", "is", "Elasticsearch"]
3	Hello, world! This is Elasticsearch.	pattern	Split by regex '\\W+' (non-word chars)	["Hello", "world", "This", "is", "Elasticsearch"]
4	N/A	N/A	N/A	End of tokenization

💡 Tokenization ends after splitting input text into tokens based on chosen tokenizer rules.

Variable Tracker

Variable	Start	After Step 1	After Step 2	After Step 3	Final
input_text	"Hello, world! This is Elasticsearch."	"Hello, world! This is Elasticsearch."	"Hello, world! This is Elasticsearch."	"Hello, world! This is Elasticsearch."	"Hello, world! This is Elasticsearch."
tokens_whitespace	[]	["Hello,", "world!", "This", "is", "Elasticsearch."]	["Hello,", "world!", "This", "is", "Elasticsearch."]	["Hello,", "world!", "This", "is", "Elasticsearch."]	["Hello,", "world!", "This", "is", "Elasticsearch."]
tokens_standard	[]	[]	["Hello", "world", "This", "is", "Elasticsearch"]	["Hello", "world", "This", "is", "Elasticsearch"]	["Hello", "world", "This", "is", "Elasticsearch"]
tokens_pattern	[]	[]	[]	["Hello", "world", "This", "is", "Elasticsearch"]	["Hello", "world", "This", "is", "Elasticsearch"]

Key Moments - 3 Insights

Why does the standard tokenizer produce tokens without punctuation while the whitespace tokenizer does not?

The standard tokenizer splits on punctuation and discards it as part of its process (see Step 2 in execution_table), while the whitespace tokenizer only splits by spaces and keeps punctuation attached to words (Step 1).

Why does the pattern tokenizer split differently than whitespace tokenizer?

Pattern tokenizer uses a regex to split on non-word characters (Step 3), so it removes punctuation as separators, unlike whitespace tokenizer which splits only on spaces (Step 1).

Can tokens include punctuation with whitespace tokenizer?

Yes, whitespace tokenizer keeps punctuation attached to words because it only splits on spaces (Step 1 tokens include commas and exclamation marks).

Visual Quiz - 3 Questions

Look at the execution_table, what tokens does the standard tokenizer produce at Step 2?

A ["Hello,", "world!", "This", "is", "Elasticsearch."]

B ["Hello", "world", "This", "is", "Elasticsearch"]

C ["hello", "world", "this", "is", "elasticsearch"]

D ["Hello world This is Elasticsearch"]

💡 Hint

Check the Tokens Produced column at Step 2 in execution_table.

At which step does the tokenizer split text by regex pattern?

A Step 1

B Step 2

C Step 3

D Step 4

💡 Hint

Look for 'pattern' tokenizer and regex '\\W+' in execution_table.

If the input text had no spaces, which tokenizer would produce the fewest tokens?

A Whitespace tokenizer

B Standard tokenizer

C Pattern tokenizer

D All produce the same tokens

💡 Hint

Whitespace tokenizer splits only on spaces, so no spaces means one token (see variable_tracker for splitting rules).

Concept Snapshot

Tokenizers split text into tokens for search analysis.
Standard tokenizer splits on punctuation and spaces, discarding punctuation.
Whitespace tokenizer splits only on spaces, keeps punctuation.
Pattern tokenizer splits by regex pattern (e.g., non-word chars).
Choose tokenizer based on how you want to break text.

Full Transcript

Tokenizers in Elasticsearch break text into smaller pieces called tokens. The standard tokenizer splits text by punctuation and spaces. The whitespace tokenizer splits text only by spaces and keeps punctuation attached to words. The pattern tokenizer splits text using a regular expression pattern, such as splitting on non-word characters. For example, given the text 'Hello, world! This is Elasticsearch.', the whitespace tokenizer produces tokens including punctuation like 'Hello,' and 'world!'. The standard tokenizer produces tokens without punctuation like 'Hello' and 'world'. The pattern tokenizer splits by regex and removes punctuation as separators. Understanding these differences helps choose the right tokenizer for your search needs.

Practice

1. Which Elasticsearch tokenizer splits text by spaces only, keeping punctuation as part of tokens?

easy

A. standard tokenizer

B. whitespace tokenizer

C. pattern tokenizer

D. keyword tokenizer

Solution

1. Step 1: Understand whitespace tokenizer behavior

   The whitespace tokenizer splits text only at spaces and keeps punctuation inside tokens.

2. Step 2: Compare with other tokenizers

   The standard tokenizer removes punctuation, and the pattern tokenizer splits by regex patterns, so they behave differently.

3. Final Answer:

   whitespace tokenizer -> Option B

4. Quick Check:

   Spaces split tokens only = whitespace tokenizer [OK]

Hint: Whitespace tokenizer splits only on spaces, keeps punctuation [OK]

Common Mistakes:

• Confusing standard tokenizer with whitespace tokenizer
• Thinking pattern tokenizer splits only on spaces
• Assuming keyword tokenizer splits text

2. Which of the following is the correct JSON syntax to define a pattern tokenizer that splits text on commas?

easy

A. { "tokenizer": "pattern", "pattern": ',' }

B. { 'tokenizer': 'pattern', 'pattern': ',' }

C. { "tokenizer": "pattern", pattern: "," }

D. { "tokenizer": "pattern", "pattern": "," }

Solution

1. Step 1: Identify correct JSON string escaping

   In JSON, double quotes inside strings must be escaped with a backslash, so the comma pattern must be inside escaped quotes.

2. Step 2: Check each option for proper escaping

   { "tokenizer": "pattern", "pattern": "," } correctly shows the pattern as a string with comma inside quotes.

3. Final Answer:

   { "tokenizer": "pattern", "pattern": "," } -> Option D

4. Quick Check:

   Proper JSON escaping for pattern string = { "tokenizer": "pattern", "pattern": "," } [OK]

Hint: Escape quotes inside JSON strings with backslash [OK]

Common Mistakes:

• Not escaping quotes inside JSON strings
• Using single quotes instead of double quotes
• Missing commas or braces in JSON

3. Given the text "Hello, world! Welcome to Elasticsearch.", what tokens does the standard tokenizer produce?

medium

A. ["Hello", "world!", "Welcome", "to", "Elasticsearch"]

B. ["Hello,", "world!", "Welcome", "to", "Elasticsearch."]

C. ["Hello", "world", "Welcome", "to", "Elasticsearch"]

D. ["Hello", "world", "Welcome", "to", "Elasticsearch."]

Solution

1. Step 1: Understand standard tokenizer behavior

   The standard tokenizer splits text by word boundaries and removes punctuation from tokens.

2. Step 2: Apply tokenizer to given text

   It splits "Hello," into "Hello", "world!" into "world", and removes punctuation marks like commas and exclamation points.

3. Final Answer:

   ["Hello", "world", "Welcome", "to", "Elasticsearch"] -> Option C

4. Quick Check:

   Standard tokenizer removes punctuation = ["Hello", "world", "Welcome", "to", "Elasticsearch"] [OK]

Hint: Standard tokenizer removes punctuation, returns clean words [OK]

Common Mistakes:

• Including punctuation in tokens with standard tokenizer
• Confusing output with whitespace tokenizer
• Assuming pattern tokenizer output without pattern

4. You defined a pattern tokenizer with pattern \W+ but it does not split the text as expected. What is the likely issue?

medium

A. The pattern is not properly escaped in JSON

B. Pattern tokenizer cannot use regex

C. Whitespace tokenizer should be used instead

D. Standard tokenizer must be enabled first

Solution

1. Step 1: Check pattern escaping in JSON

   Regex patterns with backslashes must be double escaped in JSON strings, so \W+ should be written as \\W+.

2. Step 2: Understand tokenizer requirements

   If pattern is not escaped properly, the tokenizer won't interpret it as intended, causing wrong splits.

3. Final Answer:

   The pattern is not properly escaped in JSON -> Option A

4. Quick Check:

   Regex backslash must be double escaped in JSON [OK]

Hint: Double escape backslashes in regex patterns for JSON [OK]

Common Mistakes:

• Using single backslash in JSON regex pattern
• Confusing tokenizer types
• Assuming pattern tokenizer needs enabling

5. You want to index a text field so that tokens are split on commas and semicolons only, keeping spaces and punctuation inside tokens. Which tokenizer configuration should you use?

hard

A. Use a pattern tokenizer with pattern [,;]

B. Use the standard tokenizer

C. Use the whitespace tokenizer

D. Use a pattern tokenizer with pattern \s+

Solution

1. Step 1: Identify tokenizer that splits on specific characters

   The pattern tokenizer allows splitting text by a custom regex pattern, so you can specify commas and semicolons.

2. Step 2: Confirm pattern matches commas and semicolons only

   The pattern [,;] matches commas or semicolons, splitting tokens at those characters but keeping spaces and other punctuation inside tokens.

3. Final Answer:

   Use a pattern tokenizer with pattern [,;] -> Option A

4. Quick Check:

   Pattern tokenizer with custom split chars = Use a pattern tokenizer with pattern [,;] [OK]

Hint: Pattern tokenizer splits by your regex pattern [OK]

Common Mistakes:

• Using whitespace tokenizer which splits on spaces
• Using standard tokenizer which removes punctuation
• Using wrong regex pattern for splitting