Full-text search engine concept in Elasticsearch - Step-by-Step Execution

Concept Flow - Full-text search engine concept

User enters search query

↓

Query parsed and analyzed

↓

Search engine looks up inverted index

↓

Retrieve matching documents

↓

Rank documents by relevance

↓

Return results to user

This flow shows how a full-text search engine takes a user's query, processes it, finds matching documents using an index, ranks them, and returns results.

Execution Sample

Elasticsearch

GET /library/_search
{
  "query": {
    "match": { "content": "apple" }
  }
}

This query searches the 'library' index for documents containing the word 'apple' in the 'content' field.

Execution Table

Step	Action	Input/State	Output/Result
1	Receive query	{"match": {"content": "apple"}}	Query ready for analysis
2	Analyze query	Tokenize 'apple'	Tokens: ['apple']
3	Lookup inverted index	Token 'apple'	Document IDs with 'apple' found
4	Retrieve documents	Document IDs	Documents with 'apple' content fetched
5	Rank documents	Documents fetched	Documents ordered by relevance score
6	Return results	Ranked documents	Search results sent to user

💡 All matching documents processed and results returned to user

Variable Tracker

Variable	Start	After Step 2	After Step 3	After Step 4	After Step 5	Final
query	null	{"match": {"content": "apple"}}	{"match": {"content": "apple"}}	{"match": {"content": "apple"}}	{"match": {"content": "apple"}}	{"match": {"content": "apple"}}
tokens	[]	['apple']	['apple']	['apple']	['apple']	['apple']
doc_ids	[]	[]	[1, 3, 7]	[1, 3, 7]	[1, 3, 7]	[1, 3, 7]
documents	[]	[]	[]	Doc1, Doc3, Doc7	Doc1, Doc3, Doc7	Doc1, Doc3, Doc7
ranked_docs	[]	[]	[]	[]	Doc3, Doc1, Doc7	Doc3, Doc1, Doc7

Key Moments - 3 Insights

Why does the search engine tokenize the query word 'apple'?

Tokenizing breaks the query into searchable pieces. Here, 'apple' is a single token, so the engine can look it up in the index (see execution_table step 2).

How does the engine find documents containing 'apple' so fast?

It uses an inverted index that maps tokens to document IDs. This lets it quickly find all documents with 'apple' without scanning everything (see execution_table step 3).

Why are documents ranked before returning results?

Ranking orders documents by how well they match the query, so the most relevant appear first (see execution_table step 5).

Visual Quiz - 3 Questions

Look at the execution_table at step 3, what does the search engine output?

A A list of document IDs containing the token 'apple'

B The full text of documents containing 'apple'

C The user's original query

D The ranked list of documents

💡 Hint

Check the 'Output/Result' column at step 3 in the execution_table

At which step does the search engine rank documents by relevance?

A Step 2

B Step 4

C Step 5

D Step 6

💡 Hint

Look for 'Rank documents' action in the execution_table

If the query was 'apple pie', how would the tokens variable change after step 2?

A ['apple pie']

B ['apple', 'pie']

C ['apple']

D []

💡 Hint

Refer to variable_tracker tokens after step 2 and think about tokenizing multiple words

Concept Snapshot

Full-text search engine steps:
1. User inputs query
2. Query is tokenized into words
3. Tokens are looked up in an inverted index
4. Matching documents are retrieved
5. Documents are ranked by relevance
6. Results are returned to user

Full Transcript

A full-text search engine works by taking a user's search query and breaking it into smaller parts called tokens. It then looks up these tokens in a special index that quickly tells which documents contain them. The engine fetches these documents and ranks them by how well they match the query. Finally, it sends the ranked list back to the user. This process makes searching fast and relevant.