Tokenizers (standard, whitespace, pattern) in Elasticsearch - Interactive Code Practice

Practice - 5 Tasks

1fill in blank

easy

Complete the code to define a standard tokenizer in Elasticsearch.

Elasticsearch

{
  "settings": {
    "analysis": {
      "tokenizer": {
        "my_tokenizer": {
          "type": "[1]"
        }
      }
    }
  }
}

Drag options to blanks, or click blank then click option'

Akeyword

Bstandard

Cpattern

Dwhitespace

Attempts:

3 left

💡 Hint

Common Mistakes

Using 'whitespace' instead of 'standard' when you want word boundaries.

Confusing 'pattern' tokenizer with 'standard' tokenizer.

The standard tokenizer is the default tokenizer in Elasticsearch that breaks text into terms on word boundaries.

2fill in blank

medium

Complete the code to define a whitespace tokenizer in Elasticsearch.

Elasticsearch

{
  "settings": {
    "analysis": {
      "tokenizer": {
        "my_ws_tokenizer": {
          "type": "[1]"
        }
      }
    }
  }
}

Drag options to blanks, or click blank then click option'

Awhitespace

Bpattern

Cstandard

Dkeyword

Attempts:

3 left

💡 Hint

Common Mistakes

Using 'standard' tokenizer when whitespace tokenizer is needed.

Confusing 'pattern' tokenizer with 'whitespace' tokenizer.

The whitespace tokenizer splits text only on whitespace characters, keeping punctuation attached to words.

3fill in blank

hard

Fix the error in the pattern tokenizer definition by completing the pattern field.

Elasticsearch

{
  "settings": {
    "analysis": {
      "tokenizer": {
        "my_pattern_tokenizer": {
          "type": "pattern",
          "pattern": "[1]"
        }
      }
    }
  }
}

Drag options to blanks, or click blank then click option'

A\\d+

B\\w+

C\\s+

D\\S+

Attempts:

3 left

💡 Hint

Common Mistakes

Using '\w+' which matches word characters, not spaces.

Using '\d+' which matches digits, not spaces.

The pattern tokenizer uses a regular expression to split text. The pattern \s+ matches one or more whitespace characters, splitting text on spaces.

4fill in blank

hard

Fill both blanks to define a pattern tokenizer that splits on commas and spaces.

Elasticsearch

{
  "settings": {
    "analysis": {
      "tokenizer": {
        "comma_space_tokenizer": {
          "type": "[1]",
          "pattern": "[2]"
        }
      }
    }
  }
}

Drag options to blanks, or click blank then click option'

Apattern

B\\s*,\\s*

C\\s+

Dwhitespace

Attempts:

3 left

💡 Hint

Common Mistakes

Using 'whitespace' type with a pattern field.

Using a pattern that does not match commas.

The tokenizer type must be pattern to use a regex pattern. The pattern \s*,\s* splits text on commas surrounded by optional spaces.

5fill in blank

hard

Fill all three blanks to define a custom analyzer using the standard tokenizer and lowercase filter.

Elasticsearch

{
  "settings": {
    "analysis": {
      "analyzer": {
        "my_custom_analyzer": {
          "type": "custom",
          "tokenizer": "[1]",
          "filter": ["[2]", "[3]"]
        }
      }
    }
  }
}

Drag options to blanks, or click blank then click option'

Astandard

Blowercase

Cstop

Dwhitespace

Attempts:

3 left

💡 Hint

Common Mistakes

Using 'whitespace' tokenizer instead of 'standard'.

Omitting the 'stop' filter when filtering stop words is needed.

The custom analyzer uses the standard tokenizer. The filters include lowercase to convert tokens to lowercase and stop to remove common stop words.

Practice

1. Which Elasticsearch tokenizer splits text by spaces only, keeping punctuation as part of tokens?

easy

A. standard tokenizer

B. whitespace tokenizer

C. pattern tokenizer

D. keyword tokenizer

Solution

1. Step 1: Understand whitespace tokenizer behavior

   The whitespace tokenizer splits text only at spaces and keeps punctuation inside tokens.

2. Step 2: Compare with other tokenizers

   The standard tokenizer removes punctuation, and the pattern tokenizer splits by regex patterns, so they behave differently.

3. Final Answer:

   whitespace tokenizer -> Option B

4. Quick Check:

   Spaces split tokens only = whitespace tokenizer [OK]

Hint: Whitespace tokenizer splits only on spaces, keeps punctuation [OK]

Common Mistakes:

• Confusing standard tokenizer with whitespace tokenizer
• Thinking pattern tokenizer splits only on spaces
• Assuming keyword tokenizer splits text

2. Which of the following is the correct JSON syntax to define a pattern tokenizer that splits text on commas?

easy

A. { "tokenizer": "pattern", "pattern": ',' }

B. { 'tokenizer': 'pattern', 'pattern': ',' }

C. { "tokenizer": "pattern", pattern: "," }

D. { "tokenizer": "pattern", "pattern": "," }

Solution

1. Step 1: Identify correct JSON string escaping

   In JSON, double quotes inside strings must be escaped with a backslash, so the comma pattern must be inside escaped quotes.

2. Step 2: Check each option for proper escaping

   { "tokenizer": "pattern", "pattern": "," } correctly shows the pattern as a string with comma inside quotes.

3. Final Answer:

   { "tokenizer": "pattern", "pattern": "," } -> Option D

4. Quick Check:

   Proper JSON escaping for pattern string = { "tokenizer": "pattern", "pattern": "," } [OK]

Hint: Escape quotes inside JSON strings with backslash [OK]

Common Mistakes:

• Not escaping quotes inside JSON strings
• Using single quotes instead of double quotes
• Missing commas or braces in JSON

3. Given the text "Hello, world! Welcome to Elasticsearch.", what tokens does the standard tokenizer produce?

medium

A. ["Hello", "world!", "Welcome", "to", "Elasticsearch"]

B. ["Hello,", "world!", "Welcome", "to", "Elasticsearch."]

C. ["Hello", "world", "Welcome", "to", "Elasticsearch"]

D. ["Hello", "world", "Welcome", "to", "Elasticsearch."]

Solution

1. Step 1: Understand standard tokenizer behavior

   The standard tokenizer splits text by word boundaries and removes punctuation from tokens.

2. Step 2: Apply tokenizer to given text

   It splits "Hello," into "Hello", "world!" into "world", and removes punctuation marks like commas and exclamation points.

3. Final Answer:

   ["Hello", "world", "Welcome", "to", "Elasticsearch"] -> Option C

4. Quick Check:

   Standard tokenizer removes punctuation = ["Hello", "world", "Welcome", "to", "Elasticsearch"] [OK]

Hint: Standard tokenizer removes punctuation, returns clean words [OK]

Common Mistakes:

• Including punctuation in tokens with standard tokenizer
• Confusing output with whitespace tokenizer
• Assuming pattern tokenizer output without pattern

4. You defined a pattern tokenizer with pattern \W+ but it does not split the text as expected. What is the likely issue?

medium

A. The pattern is not properly escaped in JSON

B. Pattern tokenizer cannot use regex

C. Whitespace tokenizer should be used instead

D. Standard tokenizer must be enabled first

Solution

1. Step 1: Check pattern escaping in JSON

   Regex patterns with backslashes must be double escaped in JSON strings, so \W+ should be written as \\W+.

2. Step 2: Understand tokenizer requirements

   If pattern is not escaped properly, the tokenizer won't interpret it as intended, causing wrong splits.

3. Final Answer:

   The pattern is not properly escaped in JSON -> Option A

4. Quick Check:

   Regex backslash must be double escaped in JSON [OK]

Hint: Double escape backslashes in regex patterns for JSON [OK]

Common Mistakes:

• Using single backslash in JSON regex pattern
• Confusing tokenizer types
• Assuming pattern tokenizer needs enabling

5. You want to index a text field so that tokens are split on commas and semicolons only, keeping spaces and punctuation inside tokens. Which tokenizer configuration should you use?

hard

A. Use a pattern tokenizer with pattern [,;]

B. Use the standard tokenizer

C. Use the whitespace tokenizer

D. Use a pattern tokenizer with pattern \s+

Solution

1. Step 1: Identify tokenizer that splits on specific characters

   The pattern tokenizer allows splitting text by a custom regex pattern, so you can specify commas and semicolons.

2. Step 2: Confirm pattern matches commas and semicolons only

   The pattern [,;] matches commas or semicolons, splitting tokens at those characters but keeping spaces and other punctuation inside tokens.

3. Final Answer:

   Use a pattern tokenizer with pattern [,;] -> Option A

4. Quick Check:

   Pattern tokenizer with custom split chars = Use a pattern tokenizer with pattern [,;] [OK]

Hint: Pattern tokenizer splits by your regex pattern [OK]

Common Mistakes:

• Using whitespace tokenizer which splits on spaces
• Using standard tokenizer which removes punctuation
• Using wrong regex pattern for splitting