Why text analysis enables smart search in Elasticsearch - See It in Action

Why text analysis enables smart search

📖 Scenario: You are building a simple search feature for a small online bookstore. You want users to find books easily by typing words related to the book titles or descriptions.To do this, you will use Elasticsearch's text analysis features to break down the text into searchable parts and make the search smarter.

🎯 Goal: Build a basic Elasticsearch index with text analysis enabled, add some book data, and perform a search that shows how text analysis helps find relevant books.

📋 What You'll Learn

Create an Elasticsearch index with a custom analyzer that uses a lowercase filter

Add three book documents with titles and descriptions

Use a match query to search for a word in the book titles or descriptions

Print the search results showing matched books

💡 Why This Matters

🌍 Real World

Text analysis is key in search engines to break down and understand user queries and documents, making search results more relevant and user-friendly.

💼 Career

Knowing how to configure text analysis in Elasticsearch is important for roles in backend development, data engineering, and search engine optimization.

1

Create the Elasticsearch index with text analysis

Create an Elasticsearch index called books with a custom analyzer named my_analyzer that uses the standard tokenizer and the lowercase filter.

Elasticsearch

# Your code here to create the index with analyzer

Hint

Use the PUT method to create the index with settings.analysis.analyzer and mappings.properties.

2

Add book documents to the index

Add three book documents to the books index with these exact details:
1. title: "The Great Gatsby", description: "A classic novel set in the 1920s."
2. title: "Great Expectations", description: "A story about personal growth and ambition."
3. title: "The Art of War", description: "An ancient Chinese military treatise."

Elasticsearch

PUT /books
{
  "settings": {
    "analysis": {
      "analyzer": {
        "my_analyzer": {
          "type": "custom",
          "tokenizer": "standard",
          "filter": ["lowercase"]
        }
      }
    }
  },
  "mappings": {
    "properties": {
      "title": {
        "type": "text",
        "analyzer": "my_analyzer"
      },
      "description": {
        "type": "text",
        "analyzer": "my_analyzer"
      }
    }
  }
}

POST /books/_doc/1
{
  "title": "The Great Gatsby",
  "description": "A classic novel set in the 1920s."
}

POST /books/_doc/2
{
  "title": "Great Expectations",
  "description": "A story about personal growth and ambition."
}

POST /books/_doc/3
{
  "title": "The Art of War",
  "description": "An ancient Chinese military treatise."
}

# Your code here to add documents

Hint

Use POST /books/_doc/1, _doc/2, and _doc/3 to add each book with the exact titles and descriptions.

3

Search for books using text analysis

Write a search query using GET /books/_search with a match query on the title field to find books matching the word great.

Elasticsearch

PUT /books
{
  "settings": {
    "analysis": {
      "analyzer": {
        "my_analyzer": {
          "type": "custom",
          "tokenizer": "standard",
          "filter": ["lowercase"]
        }
      }
    }
  },
  "mappings": {
    "properties": {
      "title": {
        "type": "text",
        "analyzer": "my_analyzer"
      },
      "description": {
        "type": "text",
        "analyzer": "my_analyzer"
      }
    }
  }
}

POST /books/_doc/1
{
  "title": "The Great Gatsby",
  "description": "A classic novel set in the 1920s."
}

POST /books/_doc/2
{
  "title": "Great Expectations",
  "description": "A story about personal growth and ambition."
}

POST /books/_doc/3
{
  "title": "The Art of War",
  "description": "An ancient Chinese military treatise."
}

GET /books/_search
{
  "query": {
    "match": {
      "title": "great"
    }
  }
}

# Your code here to search for "great" in titles

Hint

Use a match query on the title field with the word great.

4

Display the search results

Print the titles of the books returned by the search query for great to show how text analysis helps find both "The Great Gatsby" and "Great Expectations".

Elasticsearch

PUT /books
{
  "settings": {
    "analysis": {
      "analyzer": {
        "my_analyzer": {
          "type": "custom",
          "tokenizer": "standard",
          "filter": ["lowercase"]
        }
      }
    }
  },
  "mappings": {
    "properties": {
      "title": {
        "type": "text",
        "analyzer": "my_analyzer"
      },
      "description": {
        "type": "text",
        "analyzer": "my_analyzer"
      }
    }
  }
}

POST /books/_doc/1
{
  "title": "The Great Gatsby",
  "description": "A classic novel set in the 1920s."
}

POST /books/_doc/2
{
  "title": "Great Expectations",
  "description": "A story about personal growth and ambition."
}

POST /books/_doc/3
{
  "title": "The Art of War",
  "description": "An ancient Chinese military treatise."
}

GET /books/_search
{
  "query": {
    "match": {
      "title": "great"
    }
  }
}

# Your code here to print titles of matched books

Hint

Show the titles of the books found by the search to demonstrate the effect of text analysis.

Practice

1. What is the main purpose of text analysis in Elasticsearch search?

easy

A. To create visual charts from search results

B. To store data in a compressed format

C. To encrypt the search queries

D. To break text into smaller parts for better matching

Solution

1. Step 1: Understand text analysis role

   Text analysis breaks down text into smaller pieces called tokens to help Elasticsearch match search terms better.

2. Step 2: Compare options

   Only To break text into smaller parts for better matching describes breaking text into parts for better matching, which is the core of text analysis.

3. Final Answer:

   To break text into smaller parts for better matching -> Option D

4. Quick Check:

   Text analysis = breaking text for better search [OK]

Hint: Text analysis splits text to improve search matches [OK]

Common Mistakes:

• Confusing text analysis with data storage
• Thinking it encrypts queries
• Assuming it creates charts

2. Which of the following is the correct way to specify an analyzer in an Elasticsearch mapping?

easy

A. { "content": { "type": "keyword", "analyzer": "standard" } }

B. { "properties": { "content": { "type": "text", "analyzer": "standard" } } }

C. { "properties": { "content": { "type": "text" } }, "analyzer": "standard" }

D. { "analyzer": "standard", "type": "text" }

Solution

1. Step 1: Review Elasticsearch mapping syntax

   Analyzers are set inside the field definition under properties, specifying type and analyzer.

2. Step 2: Identify correct JSON structure

   { "properties": { "content": { "type": "text", "analyzer": "standard" } } } correctly places analyzer inside the content field under properties with type text.

3. Final Answer:

   { "properties": { "content": { "type": "text", "analyzer": "standard" } } } -> Option B

4. Quick Check:

   Analyzer inside field properties = { "properties": { "content": { "type": "text", "analyzer": "standard" } } } [OK]

Hint: Analyzer goes inside field under properties in mapping [OK]

Common Mistakes:

• Placing analyzer outside properties
• Using keyword type with analyzer
• Incorrect JSON nesting

3. Given this Elasticsearch analyzer configuration:

{ "tokenizer": "standard", "filter": ["lowercase", "stop"] }

What happens when you index the text "The Quick Brown Fox"?

medium

A. ["quick", "brown", "fox"]

B. ["the", "quick", "brown", "fox"]

C. ["The", "Quick", "Brown", "Fox"]

D. ["quick", "brown"]

Solution

1. Step 1: Apply tokenizer and filters

   The standard tokenizer splits text into words: ["The", "Quick", "Brown", "Fox"]. Then lowercase filter makes all lowercase: ["the", "quick", "brown", "fox"]. The stop filter removes common stop words like "the".

2. Step 2: Final tokens after filters

   After removing "the", tokens are ["quick", "brown", "fox"].

3. Final Answer:

   ["quick", "brown", "fox"] -> Option A

4. Quick Check:

   Stop words removed, lowercase applied = ["quick", "brown", "fox"] [OK]

Hint: Stop filter removes common words like 'the' [OK]

Common Mistakes:

• Forgetting stop word removal
• Ignoring lowercase filter effect
• Assuming original casing remains

4. You wrote this analyzer in Elasticsearch:

{ "tokenizer": "standard", "filter": ["lowercase", "stop", "stemmer"] }

But your search results are missing relevant documents. What is a likely cause?

medium

A. The stemmer filter changed words too much, causing mismatches

B. The tokenizer is incorrect and causes syntax errors

C. The lowercase filter is missing, so case mismatches occur

D. Stop filter is disabled, so too many words are indexed

Solution

1. Step 1: Understand stemmer effect

   The stemmer filter reduces words to their root form, but sometimes it changes words too aggressively, causing search terms not to match indexed terms.

2. Step 2: Analyze impact on search results

   If stemmer changes words too much, relevant documents may be missed because the search terms don't match the stemmed tokens.

3. Final Answer:

   The stemmer filter changed words too much, causing mismatches -> Option A

4. Quick Check:

   Over-stemming causes missed matches = The stemmer filter changed words too much, causing mismatches [OK]

Hint: Over-stemming can hide relevant results [OK]

Common Mistakes:

• Blaming tokenizer for syntax errors
• Ignoring stemmer impact
• Assuming lowercase filter is missing

5. You want to create a smart search that finds documents even if users type plural or different forms of words. Which analyzer setup best achieves this?

hard

A. Use a whitespace tokenizer with stop filter only

B. Use a keyword tokenizer with no filters

C. Use a standard tokenizer with lowercase and stemmer filters

D. Use a pattern tokenizer with uppercase filter

Solution

1. Step 1: Identify analyzer needs for smart search

   To match different word forms like plurals, stemming is needed. Lowercase helps ignore case differences. Standard tokenizer splits text properly.

2. Step 2: Evaluate options

   Use a standard tokenizer with lowercase and stemmer filters includes standard tokenizer, lowercase, and stemmer filters, which together enable matching different word forms smartly.

3. Final Answer:

   Use a standard tokenizer with lowercase and stemmer filters -> Option C

4. Quick Check:

   Stemming + lowercase + standard tokenizer = smart search [OK]

Hint: Combine stemmer and lowercase for flexible word matching [OK]

Common Mistakes:

• Using keyword tokenizer which disables token splitting
• Relying only on stop filter
• Using uppercase filter which doesn't help word forms