Full-text search engine concept in Elasticsearch - Step-by-Step Execution

Concept Flow - Full-text search engine concept

User enters search query

↓

Query parsed and analyzed

↓

Search engine looks up inverted index

↓

Retrieve matching documents

↓

Rank documents by relevance

↓

Return results to user

This flow shows how a full-text search engine takes a user's query, processes it, finds matching documents using an index, ranks them, and returns results.

Execution Sample

Elasticsearch

GET /library/_search
{
  "query": {
    "match": { "content": "apple" }
  }
}

This query searches the 'library' index for documents containing the word 'apple' in the 'content' field.

Execution Table

Step	Action	Input/State	Output/Result
1	Receive query	{"match": {"content": "apple"}}	Query ready for analysis
2	Analyze query	Tokenize 'apple'	Tokens: ['apple']
3	Lookup inverted index	Token 'apple'	Document IDs with 'apple' found
4	Retrieve documents	Document IDs	Documents with 'apple' content fetched
5	Rank documents	Documents fetched	Documents ordered by relevance score
6	Return results	Ranked documents	Search results sent to user

💡 All matching documents processed and results returned to user

Variable Tracker

Variable	Start	After Step 2	After Step 3	After Step 4	After Step 5	Final
query	null	{"match": {"content": "apple"}}	{"match": {"content": "apple"}}	{"match": {"content": "apple"}}	{"match": {"content": "apple"}}	{"match": {"content": "apple"}}
tokens	[]	['apple']	['apple']	['apple']	['apple']	['apple']
doc_ids	[]	[]	[1, 3, 7]	[1, 3, 7]	[1, 3, 7]	[1, 3, 7]
documents	[]	[]	[]	Doc1, Doc3, Doc7	Doc1, Doc3, Doc7	Doc1, Doc3, Doc7
ranked_docs	[]	[]	[]	[]	Doc3, Doc1, Doc7	Doc3, Doc1, Doc7

Key Moments - 3 Insights

Why does the search engine tokenize the query word 'apple'?

Tokenizing breaks the query into searchable pieces. Here, 'apple' is a single token, so the engine can look it up in the index (see execution_table step 2).

How does the engine find documents containing 'apple' so fast?

It uses an inverted index that maps tokens to document IDs. This lets it quickly find all documents with 'apple' without scanning everything (see execution_table step 3).

Why are documents ranked before returning results?

Ranking orders documents by how well they match the query, so the most relevant appear first (see execution_table step 5).

Visual Quiz - 3 Questions

Look at the execution_table at step 3, what does the search engine output?

A A list of document IDs containing the token 'apple'

B The full text of documents containing 'apple'

C The user's original query

D The ranked list of documents

💡 Hint

Check the 'Output/Result' column at step 3 in the execution_table

At which step does the search engine rank documents by relevance?

A Step 2

B Step 4

C Step 5

D Step 6

💡 Hint

Look for 'Rank documents' action in the execution_table

If the query was 'apple pie', how would the tokens variable change after step 2?

A ['apple pie']

B ['apple', 'pie']

C ['apple']

D []

💡 Hint

Refer to variable_tracker tokens after step 2 and think about tokenizing multiple words

Concept Snapshot

Full-text search engine steps:
1. User inputs query
2. Query is tokenized into words
3. Tokens are looked up in an inverted index
4. Matching documents are retrieved
5. Documents are ranked by relevance
6. Results are returned to user

Full Transcript

A full-text search engine works by taking a user's search query and breaking it into smaller parts called tokens. It then looks up these tokens in a special index that quickly tells which documents contain them. The engine fetches these documents and ranks them by how well they match the query. Finally, it sends the ranked list back to the user. This process makes searching fast and relevant.

Practice

1. What is the main purpose of a full-text search engine like Elasticsearch?

easy

A. To create user interfaces for websites

B. To store images and videos efficiently

C. To quickly find words or phrases inside large amounts of text

D. To perform mathematical calculations on numbers

Solution

1. Step 1: Understand full-text search

   Full-text search engines are designed to search text data quickly and efficiently.

2. Step 2: Identify the main use case

   They help find words or phrases inside large text collections, unlike storing media or calculations.

3. Final Answer:

   To quickly find words or phrases inside large amounts of text -> Option C

4. Quick Check:

   Full-text search = find words fast [OK]

Hint: Full-text search = fast text lookup [OK]

Common Mistakes:

• Confusing full-text search with media storage
• Thinking it does math calculations
• Assuming it builds website interfaces

2. Which Elasticsearch query is commonly used to search text fields for matching words?

easy

A. term

B. match

C. range

D. exists

Solution

1. Step 1: Review query types

   Elasticsearch has different queries: term for exact matches, range for numbers, exists for presence.

2. Step 2: Identify text search query

   The match query is designed to find words inside text fields using full-text search.

3. Final Answer:

   match -> Option B

4. Quick Check:

   Text search query = match [OK]

Hint: Use match query for text search in Elasticsearch [OK]

Common Mistakes:

• Using term query for full-text search
• Confusing range query with text search
• Using exists query to find words

3. Given this Elasticsearch query snippet, what does it do?

{
  "query": {
    "match": { "description": "fast car" }
  }
}

medium

A. Searches documents where the description field exists

B. Searches documents where the description contains the exact phrase "fast car"

C. Searches documents where the description contains both words "fast" and "car" in order

D. Searches documents where the description contains either "fast" or "car"

Solution

1. Step 1: Understand match query behavior

   The match query analyzes the input text and searches for documents containing any of the words.

2. Step 2: Analyze the query text "fast car"

   It looks for documents containing "fast" or "car", not necessarily as an exact phrase or order.

3. Final Answer:

   Searches documents where the description contains either "fast" or "car" -> Option D

4. Quick Check:

   Match query finds any matching words [OK]

Hint: Match query finds any word in text [OK]

Common Mistakes:

• Assuming match query searches exact phrase
• Thinking it requires words in order
• Confusing with exists query

4. You wrote this Elasticsearch query but it returns no results:

{
  "query": {
    "match": { "title": "" }
  }
}

What is the likely problem?

medium

A. The match query cannot search empty strings

B. The field name "title" is invalid syntax

C. Elasticsearch requires a range query for text fields

D. The query must use term instead of match

Solution

1. Step 1: Check the query input

   The match query is given an empty string "" to search, which means no words to match.

2. Step 2: Understand match query behavior with empty input

   Match queries need some text to find matches; empty strings return no results.

3. Final Answer:

   The match query cannot search empty strings -> Option A

4. Quick Check:

   Empty match query = no results [OK]

Hint: Match query needs non-empty text [OK]

Common Mistakes:

• Thinking field name syntax is wrong
• Using range query for text fields
• Confusing term and match query roles

5. You want to find documents where the text field "content" contains the word "apple" but not the word "fruit". Which Elasticsearch query structure should you use?

hard

A. { "query": { "bool": { "must": { "match": { "content": "apple" } }, "must_not": { "match": { "content": "fruit" } } } } }

B. { "query": { "match": { "content": "apple fruit" } } }

C. { "query": { "term": { "content": "apple" } } }

D. { "query": { "range": { "content": { "gte": "apple", "lte": "fruit" } } } }

Solution

1. Step 1: Understand the requirement

   You want documents containing "apple" but excluding those containing "fruit" in the same field.

2. Step 2: Use bool query with must and must_not

   The bool query allows combining conditions: must for required matches, must_not to exclude matches.

3. Step 3: Analyze options

   { "query": { "bool": { "must": { "match": { "content": "apple" } }, "must_not": { "match": { "content": "fruit" } } } } } uses bool with must match "apple" and must_not match "fruit", which fits the requirement.

4. Final Answer:

   { "query": { "bool": { "must": { "match": { "content": "apple" } }, "must_not": { "match": { "content": "fruit" } } } } } -> Option A

5. Quick Check:

   Bool must + must_not = include apple, exclude fruit [OK]

Hint: Use bool must and must_not for include/exclude text [OK]

Common Mistakes:

• Using match with both words includes fruit
• Using term query ignores text analysis
• Using range query on text fields is invalid