Character filters in Elasticsearch - Step-by-Step Execution

Concept Flow - Character filters

Input Text

↓

Apply Character Filters

↓

Modified Text

↓

Tokenizer

↓

Tokens for Analysis

Character filters take the input text and change characters before tokenizing, preparing text for analysis.

Execution Sample

Elasticsearch

{
  "settings": {
    "analysis": {
      "char_filter": {
        "my_filter": {
          "type": "mapping",
          "mappings": ["&=>and", "@=>at"]
        }
      },
      "analyzer": {
        "my_analyzer": {
          "type": "custom",
          "char_filter": ["my_filter"],
          "tokenizer": "whitespace"
        }
      }
    }
  }
}

This config replaces '&' with 'and' and '@' with 'at' before splitting text by spaces.

Execution Table

Step	Input Text	Character Filter Applied	Modified Text	Tokenizer Output
1	rock & roll @ night	Replace '&' with 'and'	rock and roll @ night	—
2	rock and roll @ night	Replace '@' with 'at'	rock and roll at night	—
3	rock and roll at night	No more filters	rock and roll at night	Tokens: ['rock', 'and', 'roll', 'at', 'night']
4	End	All filters applied	Final text ready for tokenizing	Tokenization complete

💡 All character filters applied; text is ready for tokenization.

Variable Tracker

Variable	Start	After Step 1	After Step 2	After Step 3	Final
input_text	rock & roll @ night	rock and roll @ night	rock and roll at night	rock and roll at night	rock and roll at night
modified_text	rock & roll @ night	rock and roll @ night	rock and roll at night	rock and roll at night	rock and roll at night
tokens	[]	[]	[]	['rock', 'and', 'roll', 'at', 'night']	['rock', 'and', 'roll', 'at', 'night']

Key Moments - 3 Insights

Why does the text change before tokenization?

Character filters modify the raw text first (see execution_table steps 1 and 2) so tokenization works on the cleaned text.

Are character filters applied after tokenization?

No, character filters run before tokenization (execution_table step 3), changing characters in the original text.

What happens if no character filters are defined?

The original text goes straight to tokenization without changes, so tokens reflect the raw input.

Visual Quiz - 3 Questions

Look at the execution_table at step 2, what is the modified text after replacing '@'?

A rock & roll at night

B rock and roll @ night

C rock and roll at night

D rock & roll @ night

💡 Hint

Check the 'Modified Text' column at step 2 in the execution_table.

At which step does tokenization produce the final tokens?

A Step 2

B Step 3

C Step 1

D Step 4

💡 Hint

Look at the 'Tokenizer Output' column in the execution_table.

If we remove the character filter replacing '&', what would be the token for '&'?

A '&'

B No token

C 'and'

D 'at'

💡 Hint

Refer to variable_tracker for 'tokens' after step 3 and consider what happens without the filter.

Concept Snapshot

Character filters modify input text before tokenizing.
They replace or remove characters.
Configured in analysis settings.
Run before tokenizer.
Help clean or normalize text.
Example: replace '&' with 'and'.

Full Transcript

Character filters in Elasticsearch change the input text before it is split into tokens. For example, they can replace symbols like '&' with words like 'and'. This happens before the tokenizer runs. The process starts with the original text, then each character filter applies its changes in order. After all filters run, the tokenizer splits the cleaned text into tokens. This helps search work better by normalizing text. If no character filters are used, the tokenizer works on the raw text. The example shows replacing '&' with 'and' and '@' with 'at', then splitting by spaces to get tokens.

Practice

1. What is the main purpose of a character_filter in Elasticsearch analyzers?

easy

A. To split text into tokens after analysis

B. To modify text before tokenizing by replacing or removing characters

C. To store documents in the index

D. To sort search results by relevance

Solution

1. Step 1: Understand character_filter role

   Character filters act on raw text before tokenization to clean or replace characters.

2. Step 2: Differentiate from other components

   Tokenizers split text into tokens, and other parts handle indexing or sorting, not character filters.

3. Final Answer:

   To modify text before tokenizing by replacing or removing characters -> Option B

4. Quick Check:

   Character filters = modify text pre-tokenizing [OK]

Hint: Character filters act before tokenizing text [OK]

Common Mistakes:

• Confusing character filters with tokenizers
• Thinking character filters sort or store data
• Assuming character filters work after tokenizing

2. Which of the following is the correct syntax to define a mapping character filter in Elasticsearch?

easy

A. "char_filter": {"my_mapping": {"type": "mapping", "mappings": ["ph=>f", "qu=>kw"]}}

B. "char_filter": {"my_mapping": {"type": "pattern_replace", "pattern": "ph", "replacement": "f"}}

C. "char_filter": {"my_mapping": {"type": "mapping", "pattern": "ph", "replacement": "f"}}

D. "char_filter": {"my_mapping": {"type": "replace", "mappings": ["ph=>f"]}}

Solution

1. Step 1: Identify correct type and keys for mapping filter

   The mapping filter uses "type": "mapping" and a "mappings" array with string replacements.

2. Step 2: Check syntax correctness

   "char_filter": {"my_mapping": {"type": "mapping", "mappings": ["ph=>f", "qu=>kw"]}} correctly uses "type": "mapping" and "mappings" with array of strings like "ph=>f".

3. Final Answer:

   "char_filter": {"my_mapping": {"type": "mapping", "mappings": ["ph=>f", "qu=>kw"]}} -> Option A

4. Quick Check:

   Mapping filter syntax = type mapping + mappings array [OK]

Hint: Mapping filters use type 'mapping' and 'mappings' array [OK]

Common Mistakes:

• Using pattern_replace keys for mapping filter
• Wrong type name like 'replace' instead of 'mapping'
• Using 'pattern' and 'replacement' keys in mapping filter

3. Given this character filter configuration:

{
  "char_filter": {
    "my_filter": {
      "type": "pattern_replace",
      "pattern": "\\d",
      "replacement": "#"
    }
  }
}

What will be the output text after applying my_filter to the input "abc123def"?

medium

A. "abc1#3def"

B. "abc123def"

C. "abc###def"

D. "abc#23def"

Solution

1. Step 1: Understand pattern_replace with pattern '\\d'

   The pattern '\\d' matches any digit character (0-9). The replacement is '#'.

2. Step 2: Apply replacement to input text

   All digits '1', '2', '3' in "abc123def" are replaced by '#', resulting in "abc###def".

3. Final Answer:

   "abc###def" -> Option C

4. Quick Check:

   Digits replaced by '#' = "abc###def" [OK]

Hint: Pattern '\d' matches all digits, replaced by '#' [OK]

Common Mistakes:

• Replacing only first digit instead of all
• Not escaping backslash in pattern
• Confusing pattern_replace with mapping filter

4. Identify the error in this character filter definition:

{
  "char_filter": {
    "bad_filter": {
      "type": "mapping",
      "mapping": ["ph=>f"]
    }
  }
}

medium

A. Character filters cannot use arrays

B. The type 'mapping' is invalid

C. The replacement string 'ph=>f' is incorrect format

D. The key should be 'mappings' not 'mapping'

Solution

1. Step 1: Check correct key for mapping filter

   The mapping filter requires the key 'mappings' (plural) for the array of replacements.

2. Step 2: Verify other parts

   Type 'mapping' is valid, and the replacement string format "ph=>f" is correct.

3. Final Answer:

   The key should be 'mappings' not 'mapping' -> Option D

4. Quick Check:

   Key name must be 'mappings' for mapping filter [OK]

Hint: Use 'mappings' key (plural) for mapping filters [OK]

Common Mistakes:

• Using 'mapping' instead of 'mappings'
• Mistaking type name
• Wrong replacement string format

5. You want to create a character filter that replaces all occurrences of 'cat' with 'dog' and removes digits from the text before tokenizing. Which configuration correctly combines these requirements?

hard

A. {
"char_filter": {
"replace_cat": {
"type": "mapping",
"mappings": ["cat=>dog"]
},
"remove_digits": {
"type": "pattern_replace",
"pattern": "[0-9]",
"replacement": ""
}
}
}

B. {
"char_filter": {
"replace_cat": {
"type": "pattern_replace",
"pattern": "cat",
"replacement": "dog"
},
"remove_digits": {
"type": "mapping",
"mapping": ["0=>", "1=>", "2=>", "3=>", "4=>", "5=>", "6=>", "7=>", "8=>", "9=>"]
}
}
}

C. {
"char_filter": {
"replace_cat": {
"type": "mapping",
"mapping": ["cat=>dog"]
},
"remove_digits": {
"type": "pattern_replace",
"pattern": "\\d",
"replacement": ""
}
}
}

D. {
"char_filter": {
"replace_cat": {
"type": "pattern_replace",
"pattern": "cat",
"replacement": "dog"
},
"remove_digits": {
"type": "pattern_replace",
"pattern": "d",
"replacement": ""
}
}
}

Solution

1. Step 1: Check replacement of 'cat' with 'dog'

   Using a mapping filter with "cat=>dog" is correct and simpler for exact string replacement.

2. Step 2: Check digit removal filter

   Removing digits with pattern_replace using pattern "[0-9]" and empty replacement is valid and concise.

3. Step 3: Verify combined configuration

   { "char_filter": { "replace_cat": { "type": "mapping", "mappings": ["cat=>dog"] }, "remove_digits": { "type": "pattern_replace", "pattern": "[0-9]", "replacement": "" } } } correctly combines a mapping filter for 'cat' and a pattern_replace filter for digits removal.

4. Final Answer:

   { "char_filter": { "replace_cat": { "type": "mapping", "mappings": ["cat=>dog"] }, "remove_digits": { "type": "pattern_replace", "pattern": "[0-9]", "replacement": "" } } } -> Option A

5. Quick Check:

   Mapping for 'cat', pattern_replace for digits removal [OK]

Hint: Use mapping for exact words, pattern_replace for patterns [OK]

Common Mistakes:

• Using pattern_replace for simple word mapping unnecessarily
• Using mapping filter to remove digits (inefficient)
• Incorrect regex patterns for digits