TF-IDF and BM25 scoring in Elasticsearch - Step-by-Step Execution

Concept Flow - TF-IDF and BM25 scoring

Input Query

↓

Document Collection

↓

Calculate Term Frequency (TF)

↓

Calculate Inverse Document Frequency (IDF)

↓

Compute TF-IDF Score

↓

Apply BM25 Formula (TF, IDF, doc length)

↓

Rank Documents by Score

↓

Return Top Results

The flow shows how Elasticsearch scores documents by first calculating term frequency and inverse document frequency, then combining them using TF-IDF or BM25 formulas to rank documents.

Execution Sample

Elasticsearch

GET /my_index/_search
{
  "query": {
    "match": { "text": "apple banana" }
  }
}

This query searches documents containing 'apple' and 'banana' and scores them using BM25 by default.

Execution Table

Step	Action	Term	TF	IDF	BM25 Score	Explanation
1	Calculate TF for 'apple' in Doc1	apple	3	2.0	0	Count how many times 'apple' appears in Doc1
2	Calculate TF for 'banana' in Doc1	banana	1	1.5	0	Count how many times 'banana' appears in Doc1
3	Calculate IDF for 'apple'	apple	-	2.0	0	Inverse document frequency for 'apple'
4	Calculate IDF for 'banana'	banana	-	1.5	0	Inverse document frequency for 'banana'
5	Compute BM25 score for 'apple' in Doc1	apple	3	2.0	2.5	Apply BM25 formula with TF=3, IDF=2.0, doc length normalization
6	Compute BM25 score for 'banana' in Doc1	banana	1	1.5	1.2	Apply BM25 formula with TF=1, IDF=1.5, doc length normalization
7	Sum BM25 scores for Doc1	-	-	-	3.7	Total BM25 score for Doc1 is sum of term scores
8	Repeat steps 1-7 for Doc2	-	-	-	2.1	Calculate scores for another document
9	Rank documents by BM25 score	-	-	-	-	Doc1 (3.7) ranks higher than Doc2 (2.1)
10	Return top ranked documents	-	-	-	-	Results returned to user sorted by score

💡 All documents scored and ranked; top results returned.

Variable Tracker

Variable	Start	After Step 1	After Step 5	After Step 7	Final
TF_apple_Doc1	0	3	3	3	3
TF_banana_Doc1	0	1	1	1	1
IDF_apple	-	-	2.0	2.0	2.0
IDF_banana	-	-	1.5	1.5	1.5
BM25_apple_Doc1	0	0	2.5	2.5	2.5
BM25_banana_Doc1	0	0	0	1.2	1.2
BM25_total_Doc1	0	0	0	3.7	3.7

Key Moments - 3 Insights

Why does BM25 score consider document length while TF-IDF does not?

BM25 adjusts term frequency by document length to avoid bias toward longer documents, as shown in steps 5 and 6 where BM25 formula includes normalization, unlike simple TF-IDF.

Why is IDF important in scoring?

IDF reduces the weight of common terms across documents, making rare terms more influential, as seen in steps 3 and 4 where IDF values differ per term.

How are scores combined for multiple terms?

Scores for each term are calculated separately then summed to get the document's total score, demonstrated in step 7.

Visual Quiz - 3 Questions

Look at the execution table, what is the BM25 score for 'apple' in Doc1 at step 5?

A 1.5

B 3

C 2.5

D 0

💡 Hint

Check the BM25 Score column at step 5 in the execution_table.

At which step does the total BM25 score for Doc1 get calculated?

A Step 6

B Step 7

C Step 9

D Step 3

💡 Hint

Look for the row where BM25_total_Doc1 is updated in the variable_tracker and execution_table.

If the term frequency of 'banana' in Doc1 increased to 4, how would the BM25 score at step 6 change?

A It would increase

B It would decrease

C It would stay the same

D It would become zero

💡 Hint

Higher term frequency increases BM25 score as seen in steps 1, 2, 5, and 6.

Concept Snapshot

TF-IDF and BM25 scoring in Elasticsearch:
- TF counts term appearances in a doc
- IDF measures term rarity across docs
- TF-IDF multiplies TF by IDF
- BM25 improves TF-IDF by normalizing for doc length
- Elasticsearch uses BM25 by default to rank search results

Full Transcript

This visual execution shows how Elasticsearch scores documents using TF-IDF and BM25. First, it counts how often each search term appears in a document (TF). Then, it calculates how rare each term is across all documents (IDF). BM25 scoring combines these with adjustments for document length to avoid favoring longer documents. Each term's BM25 score is computed and summed to get the document's total score. Documents are then ranked by these scores to return the most relevant results. Key points include the importance of IDF to reduce common term weight, BM25's length normalization, and summing scores for multiple terms. The execution table traces these calculations step-by-step for clarity.