TF-IDF and BM25 scoring in Elasticsearch - Deep Dive

Overview - TF-IDF and BM25 scoring

What is it?

TF-IDF and BM25 are methods used to rank documents by how relevant they are to a search query. TF-IDF stands for Term Frequency-Inverse Document Frequency, which measures how important a word is in a document compared to all documents. BM25 is a more advanced scoring method that improves on TF-IDF by considering document length and term saturation. Both help search engines like Elasticsearch find the best matches for your search words.

Why it matters

Without these scoring methods, search engines would treat all words and documents equally, making search results less useful and less relevant. Imagine searching for a recipe and getting random pages instead of the best matches. TF-IDF and BM25 solve this by ranking documents so the most relevant ones appear first, saving time and improving user experience.

Where it fits

Before learning TF-IDF and BM25, you should understand basic search concepts like keywords and documents. After this, you can explore how Elasticsearch uses these scores in queries and how to tune search relevance for better results.

Mental Model

Core Idea

TF-IDF and BM25 score documents by balancing how often words appear in a document against how rare those words are across all documents, adjusting for document length and word repetition.

Think of it like...

Imagine you are looking for a book in a library. TF-IDF is like noticing how many times a word appears in a book (term frequency) and how unique that word is in the whole library (inverse document frequency). BM25 adds the idea that longer books might naturally have more words, so it adjusts your attention to avoid favoring just longer books.

┌───────────────┐       ┌─────────────────────────────┐
│ Search Query  │──────▶│ Calculate Term Frequency (TF)│
└───────────────┘       └─────────────┬───────────────┘
                                   │
                                   ▼
                      ┌─────────────────────────────┐
                      │ Calculate Inverse Document  │
                      │ Frequency (IDF)             │
                      └─────────────┬───────────────┘
                                   │
                                   ▼
                      ┌─────────────────────────────┐
                      │ Combine TF and IDF to score  │
                      │ documents (TF-IDF)           │
                      └─────────────┬───────────────┘
                                   │
                                   ▼
                      ┌─────────────────────────────┐
                      │ Adjust scores for document   │
                      │ length and term saturation   │
                      │ (BM25)                      │
                      └─────────────┬───────────────┘
                                   │
                                   ▼
                      ┌─────────────────────────────┐
                      │ Rank documents by score      │
                      └─────────────────────────────┘

Build-Up - 7 Steps

1

Foundation

Understanding Term Frequency (TF)

Concept: Term Frequency counts how often a word appears in a single document.

Term Frequency (TF) measures the number of times a word appears in a document. For example, if the word 'apple' appears 3 times in a document of 100 words, the TF is 3. This shows how important the word is in that document.

Result

You get a number showing how common a word is in one document.

Understanding TF helps you see why words repeated more in a document might be more relevant to that document's topic.

2

Foundation

Grasping Inverse Document Frequency (IDF)

Concept: Inverse Document Frequency measures how rare a word is across all documents.

IDF looks at how many documents contain a word. If a word appears in many documents, it is less useful for distinguishing one document from another. For example, common words like 'the' appear everywhere, so their IDF is low. Rare words have high IDF, making them more important for search.

Result

You get a number that tells how unique or common a word is across all documents.

Knowing IDF helps you understand why rare words carry more weight in search relevance.

3

Intermediate

Combining TF and IDF for Scoring

🤔Before reading on: do you think a word that appears often but is common across documents should have a high or low score? Commit to your answer.

Concept: TF-IDF multiplies term frequency by inverse document frequency to score words in documents.

TF-IDF score = TF × IDF. This means a word scores higher if it appears often in a document but is rare across documents. For example, 'apple' appearing 3 times in one document but rarely elsewhere will have a high TF-IDF score, making that document more relevant for 'apple'.

Result

Documents get scores that reflect both word frequency and uniqueness.

Understanding TF-IDF scoring explains how search engines rank documents by relevance, balancing common and rare words.

4

Intermediate

Limitations of TF-IDF in Search

🤔Before reading on: do you think longer documents always get higher TF-IDF scores? Yes or no? Commit to your answer.

Concept: TF-IDF does not adjust for document length or repeated terms beyond frequency, which can bias scores.

Longer documents may have higher term counts simply because they have more words, not because they are more relevant. Also, TF-IDF treats each occurrence equally, so repeating a word many times can inflate scores unfairly. This can cause longer or verbose documents to rank higher even if they are less relevant.

Result

TF-IDF can produce biased rankings favoring longer documents or repeated words.

Knowing TF-IDF's limits prepares you to understand why improved methods like BM25 are needed.

5

Intermediate

Introducing BM25 Scoring Method

Concept: BM25 improves TF-IDF by adjusting for document length and term saturation.

BM25 scores documents by considering term frequency, inverse document frequency, document length, and how repeated terms contribute less after a point. It uses parameters to control how much document length affects the score and how term frequency saturates. This makes BM25 better at ranking documents fairly regardless of length.

Result

Search results become more balanced and relevant, avoiding bias toward long documents.

Understanding BM25 shows how search engines refine scoring to improve user search experience.

6

Advanced

BM25 Formula and Parameters Explained

🤔Before reading on: do you think increasing the parameter controlling term frequency saturation will make repeated words count more or less? Commit to your answer.

Concept: BM25 formula uses parameters k1 and b to tune term frequency impact and document length normalization.

BM25 score for a term is calculated as: score = IDF × ((TF × (k1 + 1)) / (TF + k1 × (1 - b + b × (docLength / avgDocLength)))) - k1 controls how quickly term frequency saturates (usually around 1.2-2.0). - b controls how much document length affects the score (0 means no length normalization, 1 means full normalization). These parameters let you adjust scoring to fit your data and search needs.

Result

You can fine-tune search relevance by adjusting BM25 parameters.

Knowing BM25 parameters empowers you to customize search ranking for different document collections.

7

Expert

How Elasticsearch Implements BM25 by Default

🤔Before reading on: do you think Elasticsearch allows changing BM25 parameters per field or only globally? Commit to your answer.

Concept: Elasticsearch uses BM25 as the default similarity scoring algorithm and allows per-field parameter tuning.

In Elasticsearch, BM25 is the default similarity algorithm for text fields. You can configure k1 and b parameters per field in the index settings to optimize search relevance. Elasticsearch also combines BM25 with other features like field length norms and index-time statistics for efficient scoring. Understanding this helps you tune your search engine for best results.

Result

You can improve search quality by customizing BM25 settings in Elasticsearch.

Knowing Elasticsearch's BM25 implementation details helps you leverage its full power for real-world search applications.

Under the Hood

TF-IDF works by counting word occurrences in documents and across the whole collection, then multiplying these counts to score relevance. BM25 builds on this by adding a formula that normalizes scores based on document length and limits how much repeated words increase the score. Elasticsearch stores term statistics and document lengths in its inverted index, allowing fast calculation of these scores at query time.

Why designed this way?

TF-IDF was designed to highlight words that are important in a document but rare overall, solving the problem of common words diluting search relevance. BM25 was created to fix TF-IDF's bias toward longer documents and repeated terms, making search results fairer and more accurate. Elasticsearch adopted BM25 as default because it balances effectiveness and efficiency for large-scale search.

┌───────────────┐       ┌───────────────┐       ┌───────────────┐
│ Document Text │──────▶│ Inverted Index│──────▶│ Term Statistics│
└───────────────┘       └───────────────┘       └───────────────┘
                                                      │
                                                      ▼
                                         ┌─────────────────────────┐
                                         │ Calculate TF and IDF     │
                                         └─────────────┬───────────┘
                                                       │
                                                       ▼
                                         ┌─────────────────────────┐
                                         │ Apply BM25 formula with  │
                                         │ length normalization     │
                                         └─────────────┬───────────┘
                                                       │
                                                       ▼
                                         ┌─────────────────────────┐
                                         │ Score and rank documents │
                                         └─────────────────────────┘

Myth Busters - 4 Common Misconceptions

Quick: Does TF-IDF always give better search results than simple keyword matching? Commit yes or no.

Common Belief: TF-IDF always improves search results compared to just matching keywords.

Reality: TF-IDF improves relevance but can still rank long or verbose documents too high without length normalization.

Why it matters: Relying only on TF-IDF can lead to poor search quality, frustrating users with irrelevant long documents.

Quick: Is BM25 just a complicated version of TF-IDF with no real benefit? Commit yes or no.

Common Belief: BM25 is just a complex TF-IDF variant without practical advantages.

Reality: BM25 significantly improves ranking by adjusting for document length and term saturation, leading to better search relevance.

Why it matters: Ignoring BM25 means missing out on more accurate and fair search results, especially in collections with varied document lengths.

Quick: Does increasing term frequency always increase BM25 score linearly? Commit yes or no.

Common Belief: More occurrences of a word always increase BM25 score proportionally.

Reality: BM25 uses saturation, so after a point, more occurrences add less to the score to avoid overemphasizing repeated words.

Why it matters: Misunderstanding this can cause wrong tuning of search parameters, leading to skewed rankings.

Quick: Can you change BM25 parameters globally and per field in Elasticsearch? Commit yes or no.

Common Belief: BM25 parameters can only be set globally for the whole index.

Reality: Elasticsearch allows setting BM25 parameters per field, enabling fine-grained control over scoring.

Why it matters: Not knowing this limits your ability to optimize search relevance for different types of content.

Expert Zone

1

BM25's length normalization parameter b can be tuned to handle collections with very short or very long documents differently, which many overlook.

2

The saturation effect controlled by k1 prevents term frequency from dominating scores, but setting k1 too low can underweight important repeated terms.

3

Elasticsearch combines BM25 with other scoring factors like field norms and coordination factors, which subtly influence final rankings beyond the BM25 formula.

When NOT to use

BM25 and TF-IDF are less effective for semantic or contextual search where word meaning matters more than frequency. In such cases, use vector search or neural embeddings. Also, for very short texts or structured data, simpler scoring or exact matching may be better.

Production Patterns

In production, Elasticsearch users often tune BM25 parameters per field to balance recall and precision. They combine BM25 with filters and boosting to prioritize certain documents. Monitoring search logs helps adjust parameters over time for evolving data.

Connections

Information Retrieval

TF-IDF and BM25 are foundational algorithms in the field of information retrieval.

Understanding these scoring methods deepens knowledge of how search engines retrieve and rank information efficiently.

Probability Theory

BM25 scoring is based on probabilistic models estimating the likelihood a document is relevant given a query.

Knowing probability concepts helps grasp why BM25 uses saturation and normalization to model relevance realistically.

Human Attention and Memory

TF-IDF and BM25 mimic how humans focus on rare but important words when searching for information.

Recognizing this connection explains why these algorithms feel intuitive and effective in ranking search results.

Common Pitfalls

#1 Ignoring document length causes biased rankings.

Wrong approach:

Using raw TF-IDF scores without length normalization:
SELECT * FROM documents ORDER BY tf_idf_score DESC;

Correct approach:

Use BM25 scoring in Elasticsearch which normalizes for document length:
GET /_search
{
"query": {
"match": {
"content": {
"query": "search terms",
"operator": "and"
}
}
}
}

Root cause:

Not accounting for document length makes longer documents unfairly score higher.

#2 Setting BM25 parameters without understanding their effect.

Wrong approach:

Setting k1 to 0 or b to 0 without testing:
PUT /my_index
{
"settings": {
"similarity": {
"default": {
"type": "BM25",
"k1": 0,
"b": 0
}
}
}
}

Correct approach:

Use recommended defaults or tune parameters gradually:
PUT /my_index
{
"settings": {
"similarity": {
"default": {
"type": "BM25",
"k1": 1.2,
"b": 0.75
}
}
}
}

Root cause:

Misunderstanding parameters leads to poor scoring and irrelevant results.

#3 Assuming TF-IDF and BM25 are interchangeable without impact.

Wrong approach:

Switching from BM25 to TF-IDF in Elasticsearch without re-tuning:
PUT /my_index
{
"settings": {
"similarity": {
"default": {
"type": "classic"
}
}
}
}

Correct approach:

Evaluate and tune search relevance after changing similarity algorithms.

Root cause:

Different algorithms behave differently; ignoring this causes unexpected search quality drops.

Key Takeaways

TF-IDF scores documents by balancing how often words appear in a document against how rare they are across all documents.

BM25 improves on TF-IDF by adjusting for document length and limiting the impact of repeated words to produce fairer rankings.

Elasticsearch uses BM25 as the default scoring method and allows tuning its parameters per field for better search relevance.

Understanding the math and parameters behind BM25 helps you customize search engines to your data and user needs.

Misusing or misunderstanding these scoring methods can lead to poor search results, so careful tuning and evaluation are essential.