TF-IDF and BM25 scoring in Elasticsearch - Deep Dive

Overview - TF-IDF and BM25 scoring

What is it?

TF-IDF and BM25 are methods used to rank documents by how relevant they are to a search query. TF-IDF stands for Term Frequency-Inverse Document Frequency, which measures how important a word is in a document compared to all documents. BM25 is a more advanced scoring method that improves on TF-IDF by considering document length and term saturation. Both help search engines like Elasticsearch find the best matches for your search words.

Why it matters

Without these scoring methods, search engines would treat all words and documents equally, making search results less useful and less relevant. Imagine searching for a recipe and getting random pages instead of the best matches. TF-IDF and BM25 solve this by ranking documents so the most relevant ones appear first, saving time and improving user experience.

Where it fits

Before learning TF-IDF and BM25, you should understand basic search concepts like keywords and documents. After this, you can explore how Elasticsearch uses these scores in queries and how to tune search relevance for better results.

Mental Model

Core Idea

TF-IDF and BM25 score documents by balancing how often words appear in a document against how rare those words are across all documents, adjusting for document length and word repetition.

Think of it like...

Imagine you are looking for a book in a library. TF-IDF is like noticing how many times a word appears in a book (term frequency) and how unique that word is in the whole library (inverse document frequency). BM25 adds the idea that longer books might naturally have more words, so it adjusts your attention to avoid favoring just longer books.

┌───────────────┐       ┌─────────────────────────────┐
│ Search Query  │──────▶│ Calculate Term Frequency (TF)│
└───────────────┘       └─────────────┬───────────────┘
                                   │
                                   ▼
                      ┌─────────────────────────────┐
                      │ Calculate Inverse Document  │
                      │ Frequency (IDF)             │
                      └─────────────┬───────────────┘
                                   │
                                   ▼
                      ┌─────────────────────────────┐
                      │ Combine TF and IDF to score  │
                      │ documents (TF-IDF)           │
                      └─────────────┬───────────────┘
                                   │
                                   ▼
                      ┌─────────────────────────────┐
                      │ Adjust scores for document   │
                      │ length and term saturation   │
                      │ (BM25)                      │
                      └─────────────┬───────────────┘
                                   │
                                   ▼
                      ┌─────────────────────────────┐
                      │ Rank documents by score      │
                      └─────────────────────────────┘

Build-Up - 7 Steps

1

Foundation

Understanding Term Frequency (TF)

Concept: Term Frequency counts how often a word appears in a single document.

Term Frequency (TF) measures the number of times a word appears in a document. For example, if the word 'apple' appears 3 times in a document of 100 words, the TF is 3. This shows how important the word is in that document.

Result

You get a number showing how common a word is in one document.

Understanding TF helps you see why words repeated more in a document might be more relevant to that document's topic.

2

Foundation

Grasping Inverse Document Frequency (IDF)

Concept: Inverse Document Frequency measures how rare a word is across all documents.

IDF looks at how many documents contain a word. If a word appears in many documents, it is less useful for distinguishing one document from another. For example, common words like 'the' appear everywhere, so their IDF is low. Rare words have high IDF, making them more important for search.

Result

You get a number that tells how unique or common a word is across all documents.

Knowing IDF helps you understand why rare words carry more weight in search relevance.

3

Intermediate

Combining TF and IDF for Scoring

🤔Before reading on: do you think a word that appears often but is common across documents should have a high or low score? Commit to your answer.

Concept: TF-IDF multiplies term frequency by inverse document frequency to score words in documents.

TF-IDF score = TF × IDF. This means a word scores higher if it appears often in a document but is rare across documents. For example, 'apple' appearing 3 times in one document but rarely elsewhere will have a high TF-IDF score, making that document more relevant for 'apple'.

Result

Documents get scores that reflect both word frequency and uniqueness.

Understanding TF-IDF scoring explains how search engines rank documents by relevance, balancing common and rare words.

4

Intermediate

Limitations of TF-IDF in Search

🤔Before reading on: do you think longer documents always get higher TF-IDF scores? Yes or no? Commit to your answer.

Concept: TF-IDF does not adjust for document length or repeated terms beyond frequency, which can bias scores.

Longer documents may have higher term counts simply because they have more words, not because they are more relevant. Also, TF-IDF treats each occurrence equally, so repeating a word many times can inflate scores unfairly. This can cause longer or verbose documents to rank higher even if they are less relevant.

Result

TF-IDF can produce biased rankings favoring longer documents or repeated words.

Knowing TF-IDF's limits prepares you to understand why improved methods like BM25 are needed.

5

Intermediate

Introducing BM25 Scoring Method

Concept: BM25 improves TF-IDF by adjusting for document length and term saturation.

BM25 scores documents by considering term frequency, inverse document frequency, document length, and how repeated terms contribute less after a point. It uses parameters to control how much document length affects the score and how term frequency saturates. This makes BM25 better at ranking documents fairly regardless of length.

Result

Search results become more balanced and relevant, avoiding bias toward long documents.

Understanding BM25 shows how search engines refine scoring to improve user search experience.

6

Advanced

BM25 Formula and Parameters Explained

🤔Before reading on: do you think increasing the parameter controlling term frequency saturation will make repeated words count more or less? Commit to your answer.

Concept: BM25 formula uses parameters k1 and b to tune term frequency impact and document length normalization.

BM25 score for a term is calculated as: score = IDF × ((TF × (k1 + 1)) / (TF + k1 × (1 - b + b × (docLength / avgDocLength)))) - k1 controls how quickly term frequency saturates (usually around 1.2-2.0). - b controls how much document length affects the score (0 means no length normalization, 1 means full normalization). These parameters let you adjust scoring to fit your data and search needs.

Result

You can fine-tune search relevance by adjusting BM25 parameters.

Knowing BM25 parameters empowers you to customize search ranking for different document collections.

7

Expert

How Elasticsearch Implements BM25 by Default

🤔Before reading on: do you think Elasticsearch allows changing BM25 parameters per field or only globally? Commit to your answer.

Concept: Elasticsearch uses BM25 as the default similarity scoring algorithm and allows per-field parameter tuning.

In Elasticsearch, BM25 is the default similarity algorithm for text fields. You can configure k1 and b parameters per field in the index settings to optimize search relevance. Elasticsearch also combines BM25 with other features like field length norms and index-time statistics for efficient scoring. Understanding this helps you tune your search engine for best results.

Result

You can improve search quality by customizing BM25 settings in Elasticsearch.

Knowing Elasticsearch's BM25 implementation details helps you leverage its full power for real-world search applications.

Under the Hood

TF-IDF works by counting word occurrences in documents and across the whole collection, then multiplying these counts to score relevance. BM25 builds on this by adding a formula that normalizes scores based on document length and limits how much repeated words increase the score. Elasticsearch stores term statistics and document lengths in its inverted index, allowing fast calculation of these scores at query time.

Why designed this way?

TF-IDF was designed to highlight words that are important in a document but rare overall, solving the problem of common words diluting search relevance. BM25 was created to fix TF-IDF's bias toward longer documents and repeated terms, making search results fairer and more accurate. Elasticsearch adopted BM25 as default because it balances effectiveness and efficiency for large-scale search.

┌───────────────┐       ┌───────────────┐       ┌───────────────┐
│ Document Text │──────▶│ Inverted Index│──────▶│ Term Statistics│
└───────────────┘       └───────────────┘       └───────────────┘
                                                      │
                                                      ▼
                                         ┌─────────────────────────┐
                                         │ Calculate TF and IDF     │
                                         └─────────────┬───────────┘
                                                       │
                                                       ▼
                                         ┌─────────────────────────┐
                                         │ Apply BM25 formula with  │
                                         │ length normalization     │
                                         └─────────────┬───────────┘
                                                       │
                                                       ▼
                                         ┌─────────────────────────┐
                                         │ Score and rank documents │
                                         └─────────────────────────┘

Myth Busters - 4 Common Misconceptions

Quick: Does TF-IDF always give better search results than simple keyword matching? Commit yes or no.

Common Belief: TF-IDF always improves search results compared to just matching keywords.

Reality: TF-IDF improves relevance but can still rank long or verbose documents too high without length normalization.

Why it matters: Relying only on TF-IDF can lead to poor search quality, frustrating users with irrelevant long documents.

Quick: Is BM25 just a complicated version of TF-IDF with no real benefit? Commit yes or no.

Common Belief: BM25 is just a complex TF-IDF variant without practical advantages.

Reality: BM25 significantly improves ranking by adjusting for document length and term saturation, leading to better search relevance.

Why it matters: Ignoring BM25 means missing out on more accurate and fair search results, especially in collections with varied document lengths.

Quick: Does increasing term frequency always increase BM25 score linearly? Commit yes or no.

Common Belief: More occurrences of a word always increase BM25 score proportionally.

Reality: BM25 uses saturation, so after a point, more occurrences add less to the score to avoid overemphasizing repeated words.

Why it matters: Misunderstanding this can cause wrong tuning of search parameters, leading to skewed rankings.

Quick: Can you change BM25 parameters globally and per field in Elasticsearch? Commit yes or no.

Common Belief: BM25 parameters can only be set globally for the whole index.

Reality: Elasticsearch allows setting BM25 parameters per field, enabling fine-grained control over scoring.

Why it matters: Not knowing this limits your ability to optimize search relevance for different types of content.

Expert Zone

1

BM25's length normalization parameter b can be tuned to handle collections with very short or very long documents differently, which many overlook.

2

The saturation effect controlled by k1 prevents term frequency from dominating scores, but setting k1 too low can underweight important repeated terms.

3

Elasticsearch combines BM25 with other scoring factors like field norms and coordination factors, which subtly influence final rankings beyond the BM25 formula.

When NOT to use

BM25 and TF-IDF are less effective for semantic or contextual search where word meaning matters more than frequency. In such cases, use vector search or neural embeddings. Also, for very short texts or structured data, simpler scoring or exact matching may be better.

Production Patterns

In production, Elasticsearch users often tune BM25 parameters per field to balance recall and precision. They combine BM25 with filters and boosting to prioritize certain documents. Monitoring search logs helps adjust parameters over time for evolving data.

Connections

Information Retrieval

TF-IDF and BM25 are foundational algorithms in the field of information retrieval.

Understanding these scoring methods deepens knowledge of how search engines retrieve and rank information efficiently.

Probability Theory

BM25 scoring is based on probabilistic models estimating the likelihood a document is relevant given a query.

Knowing probability concepts helps grasp why BM25 uses saturation and normalization to model relevance realistically.

Human Attention and Memory

TF-IDF and BM25 mimic how humans focus on rare but important words when searching for information.

Recognizing this connection explains why these algorithms feel intuitive and effective in ranking search results.

Common Pitfalls

#1 Ignoring document length causes biased rankings.

Wrong approach:

Using raw TF-IDF scores without length normalization:
SELECT * FROM documents ORDER BY tf_idf_score DESC;

Correct approach:

Use BM25 scoring in Elasticsearch which normalizes for document length:
GET /_search
{
"query": {
"match": {
"content": {
"query": "search terms",
"operator": "and"
}
}
}
}

Root cause:

Not accounting for document length makes longer documents unfairly score higher.

#2 Setting BM25 parameters without understanding their effect.

Wrong approach:

Setting k1 to 0 or b to 0 without testing:
PUT /my_index
{
"settings": {
"similarity": {
"default": {
"type": "BM25",
"k1": 0,
"b": 0
}
}
}
}

Correct approach:

Use recommended defaults or tune parameters gradually:
PUT /my_index
{
"settings": {
"similarity": {
"default": {
"type": "BM25",
"k1": 1.2,
"b": 0.75
}
}
}
}

Root cause:

Misunderstanding parameters leads to poor scoring and irrelevant results.

#3 Assuming TF-IDF and BM25 are interchangeable without impact.

Wrong approach:

Switching from BM25 to TF-IDF in Elasticsearch without re-tuning:
PUT /my_index
{
"settings": {
"similarity": {
"default": {
"type": "classic"
}
}
}
}

Correct approach:

Evaluate and tune search relevance after changing similarity algorithms.

Root cause:

Different algorithms behave differently; ignoring this causes unexpected search quality drops.

Key Takeaways

TF-IDF scores documents by balancing how often words appear in a document against how rare they are across all documents.

BM25 improves on TF-IDF by adjusting for document length and limiting the impact of repeated words to produce fairer rankings.

Elasticsearch uses BM25 as the default scoring method and allows tuning its parameters per field for better search relevance.

Understanding the math and parameters behind BM25 helps you customize search engines to your data and user needs.

Misusing or misunderstanding these scoring methods can lead to poor search results, so careful tuning and evaluation are essential.

Practice

1. What is the main difference between TF-IDF and BM25 scoring in Elasticsearch?

easy

A. BM25 ignores inverse document frequency, TF-IDF uses it heavily.

B. TF-IDF is the default scoring algorithm in Elasticsearch, BM25 is optional.

C. BM25 considers term frequency saturation and document length normalization, TF-IDF does not.

D. TF-IDF scores are always higher than BM25 scores for the same query.

Solution

1. Step 1: Understand TF-IDF basics

   TF-IDF scores words by multiplying term frequency (TF) with inverse document frequency (IDF), but it does not adjust for document length or term frequency saturation.

2. Step 2: Understand BM25 improvements

   BM25 improves TF-IDF by adding term frequency saturation (diminishing returns for repeated terms) and normalizing by document length, making scores more balanced.

3. Final Answer:

   BM25 considers term frequency saturation and document length normalization, TF-IDF does not. -> Option C

4. Quick Check:

   BM25 adds saturation and length normalization [OK]

Hint: BM25 adjusts TF-IDF with saturation and length normalization [OK]

Common Mistakes:

• Thinking TF-IDF is default in Elasticsearch
• Believing BM25 ignores IDF
• Assuming TF-IDF always scores higher

2. Which Elasticsearch query parameter correctly switches the similarity scoring to TF-IDF instead of BM25?

easy

A. ""{
"similarity": "classic"
}"""

B. ""{
"similarity": "bm25"
}"""

C. ""{
"similarity": "tfidf"
}"""

D. ""{
"similarity": "default"
}"""

Solution

1. Step 1: Recall Elasticsearch similarity options

   Elasticsearch uses "bm25" by default. To switch to TF-IDF, you use the "classic" similarity setting.

2. Step 2: Match parameter to TF-IDF

   The "classic" similarity corresponds to TF-IDF scoring, so setting "similarity": "classic" enables TF-IDF.

3. Final Answer:

   { "similarity": "classic" } -> Option A

4. Quick Check:

   TF-IDF similarity is "classic" [OK]

Hint: Use similarity: classic to enable TF-IDF scoring [OK]

Common Mistakes:

• Using "tfidf" as similarity name (not valid)
• Assuming "default" switches to TF-IDF
• Confusing bm25 with TF-IDF

3. Given the following Elasticsearch query using BM25, what will happen to the score if a term appears multiple times in a document?

medium

A. The score increases but with diminishing returns due to term frequency saturation.

B. The score increases linearly with each additional term occurrence.

C. The score stays the same regardless of term frequency.

D. The score decreases as term frequency increases.

Solution

1. Step 1: Understand BM25 term frequency effect

   BM25 applies a saturation function to term frequency, so each additional occurrence adds less to the score than the previous one.

2. Step 2: Compare to linear increase

   Unlike TF-IDF which can increase linearly, BM25 limits the impact of repeated terms to avoid overvaluing them.

3. Final Answer:

   The score increases but with diminishing returns due to term frequency saturation. -> Option A

4. Quick Check:

   BM25 uses saturation, not linear increase [OK]

Hint: BM25 scores repeated terms with diminishing returns [OK]

Common Mistakes:

• Assuming linear score increase with term frequency
• Thinking score stays constant with repeats
• Believing score decreases with more terms

4. You configured Elasticsearch to use TF-IDF similarity but notice search results are not ranking as expected. Which of the following is a likely cause?

medium

A. TF-IDF requires a special query syntax different from standard queries.

B. TF-IDF does not use inverse document frequency, so scores are random.

C. BM25 must be disabled in the cluster settings to use TF-IDF.

D. You forgot to reindex the data after changing similarity settings.

Solution

1. Step 1: Understand similarity changes require reindexing

   Changing similarity settings affects how documents are scored, but Elasticsearch applies these only after reindexing the data.

2. Step 2: Identify common mistake

   If you don't reindex, old scores remain, so results won't reflect the new TF-IDF similarity.

3. Final Answer:

   You forgot to reindex the data after changing similarity settings. -> Option D

4. Quick Check:

   Reindexing needed after similarity change [OK]

Hint: Always reindex after changing similarity settings [OK]

Common Mistakes:

• Believing TF-IDF ignores IDF
• Thinking BM25 must be disabled cluster-wide
• Assuming query syntax changes for TF-IDF

5. You want to improve search relevance for a collection of very short documents using Elasticsearch. Which scoring approach and adjustment is best to handle short documents effectively?

hard

A. Use BM25 with a higher k1 parameter to ignore term frequency saturation.

B. Use BM25 with a lower b parameter to reduce length normalization effect.

C. Use TF-IDF similarity because it handles short documents better by default.

D. Use TF-IDF and disable inverse document frequency to boost short docs.

Solution

1. Step 1: Understand BM25 parameters for short docs

   BM25's b parameter controls length normalization. Lowering b reduces penalty on short documents, improving their scores.

2. Step 2: Compare to TF-IDF and other options

   TF-IDF lacks length normalization control. Increasing k1 ignores saturation, which can hurt relevance. Disabling IDF is incorrect.

3. Final Answer:

   Use BM25 with a lower b parameter to reduce length normalization effect. -> Option B

4. Quick Check:

   Lower b helps short docs in BM25 [OK]

Hint: Lower BM25 b parameter to favor short documents [OK]

Common Mistakes:

• Assuming TF-IDF is better for short docs by default
• Increasing k1 to ignore saturation hurts relevance
• Disabling IDF reduces search quality