Bird
Raised Fist0

Ranking Metrics - NDCG, MAP, MRR - How Search and Rec Systems Measure Quality

THEORYintermediateInterview freq: very_high
Choose your preparation mode6 modes available
LearnTraceDebugTrade-offChallenge
⚖️
NDCG vs MAP

In a recommendation system with a focus on user satisfaction, would you prioritize NDCG or MAP as your primary evaluation metric? Justify your choice.

📊
Comparison
DimensionNDCGMAP
Sensitivity to ranking positionsHighMedium
Handling of graded relevanceHighLow
ComplexityMediumLow
InterpretabilityMediumHigh
Suitability for binary relevanceLowHigh
✅
Verdict
Default to NDCG when relevance is graded and top positions are crucial. Switch to MAP when dealing with binary relevance and uniform precision across queries is needed.
💡
Key Insight
The choice between NDCG and MAP hinges on the nature of relevance data and the importance of ranking positions.
⚠️
Interview Trap
❌ Wrong: Always use NDCG because it considers ranking positions.
Ignoring the fact that NDCG's complexity and sensitivity to relevance grading may not be suitable for all datasets.
✅ Consider the nature of your relevance data and the importance of ranking positions. If relevance is binary and uniformity across queries is key, MAP might be more appropriate.
NDCG
"Effectively captures the importance of ranking positions by considering the graded relevance of items."
Weaknesses
  • Complexity increases with the number of relevance levels.
  • Sensitive to the choice of discount function.
  • May not reflect user satisfaction if relevance grades are not well-defined.
Ideal When
  • When relevance is graded (e.g., 5-star ratings).
  • When top positions are significantly more important (e.g., first page of search results).
  • When n > 10K and relevance grades are available.
Google uses NDCG in its search algorithms to ensure that the most relevant results appear at the top of the search results page.
MAP
"Provides a single-figure measure of quality across all queries, emphasizing precision at all ranks."
Weaknesses
  • Assumes binary relevance, which may not capture nuanced user preferences.
  • Less sensitive to the order of relevant documents.
  • Can be misleading if the number of relevant documents varies greatly between queries.
Ideal When
  • When relevance is binary (e.g., relevant or not).
  • When precision across all ranks is important.
  • When n < 10K and queries have a similar number of relevant documents.
Amazon uses MAP to evaluate the effectiveness of its product search algorithms, focusing on precision across a wide range of queries.
🐍
NDCG - Example
▼ Show
Python
import numpy as np

def dcg(relevances):
    return np.sum((2**relevances - 1) / np.log2(np.arange(2, relevances.size + 2)))

def ndcg(relevances, ideal_relevances):
    return dcg(relevances) / dcg(ideal_relevances)

relevances = np.array([3, 2, 3, 0, 1, 2])
ideal_relevances = np.sort(relevances)[::-1]
print(ndcg(relevances, ideal_relevances))
🐍
MAP - Example
▼ Show
Python
def average_precision(relevances):
    relevances = np.asarray(relevances) != 0
    out = [np.mean(relevances[:k + 1]) for k in range(relevances.size) if relevances[k]]
    return np.mean(out) if out else 0

relevances = [1, 0, 1, 1, 0, 1]
print(average_precision(relevances))
✅
Use NDCG When
  • •When relevance is graded and n > 10K.
  • •When top positions are crucial for user satisfaction.
  • •When you have a well-defined discount function for ranking.
✅
Use MAP When
  • •When relevance is binary and n < 10K.
  • •When precision across all ranks is equally important.
  • •When queries have a similar number of relevant documents.
↳
Nuances
  • NDCG may not be ideal if relevance grades are not well-defined, as it relies heavily on these grades.
  • MAP can be misleading if the number of relevant documents varies greatly between queries, as it assumes uniformity.
  • In scenarios where user satisfaction is tied to the top few results, NDCG often provides a better reflection of quality than MAP.

Practice

(1/5)
1. In the context of LSTM networks, which component is primarily responsible for deciding what information to keep or discard from the cell state?
easy
A. The forget gate
B. The input gate
C. The output gate
D. The activation function

Solution

  1. Step 1: Identify the role of each gate in LSTM.

    The forget gate determines what information to discard from the cell state.

  2. Step 2: Eliminate wrong options.

    The input gate decides what new information to store, the output gate decides what to output, and the activation function is not a gate.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Forget gate discards info in LSTM. [OK]
Hint: Forget gate discards info in LSTM. [OK]
Common Mistakes:
2. A neural network with dropout is not improving validation accuracy despite training for many epochs. What is a likely cause?
medium
A. The dropout rate is too high, causing underfitting.
B. The batch size is too small, causing noisy gradients.
C. The learning rate is too high, causing instability.
D. The dropout rate is too low, causing overfitting.

Solution

  1. Step 1: Analyze dropout impact

    High dropout rates can lead to underfitting as too many neurons are deactivated.

  2. Step 2: Eliminate wrong options

    The dropout rate is too low, causing overfitting. is incorrect because low dropout rates do not cause underfitting. The learning rate is too high, causing instability. is incorrect because learning rate issues would cause erratic training, not just poor validation accuracy. The batch size is too small, causing noisy gradients. is incorrect because small batch sizes affect convergence speed, not dropout effectiveness.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    High dropout can underfit. [OK]
Hint: High dropout can underfit. [OK]
Common Mistakes:
3. A model trained with Adam optimizer is not converging. What is the most likely cause?
medium
A. Batch size is too large
B. Learning rate is too high
C. Adam is not suitable for non-convex problems
D. Weight decay is too low

Solution

  1. Step 1: Identify common issues with Adam.

    Adam can be sensitive to high learning rates, causing divergence.

  2. Step 2: Eliminate less likely causes.

    Adam handles non-convex problems well, and batch size/weight decay are less likely to cause non-convergence.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    High lr can cause Adam to diverge. [check]
Hint: High lr can cause Adam to diverge. [check]
Common Mistakes:
4. You are designing a transformer model for a language translation task. Which normalization technique would you choose and why?
hard
A. Layer Normalization, because it is more suitable for transformer architectures.
B. Batch Normalization, because it is effective for large datasets.
C. Batch Normalization, because it reduces training time.
D. Layer Normalization, because it improves model interpretability.

Solution

  1. Step 1: Consider the architecture of transformers.

    Transformers process sequences and often use self-attention mechanisms, where Layer Normalization is more effective.

  2. Step 2: Evaluate the benefits of Layer Normalization.

    Layer Normalization normalizes across features, which is beneficial for the attention mechanism in transformers.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Layer Norm fits transformers' self-attention. [OK]
Hint: Layer Norm fits transformers' self-attention. [OK]
Common Mistakes:
5. In a fraud detection system, you need to balance precision and recall. Which metric would you choose and why?
hard
A. F1 Score, because it balances both precision and recall.
B. Precision, because false positives are more costly.
C. Recall, because missing a fraud case is more critical.
D. Accuracy, because it gives an overall performance measure.

Solution

  1. Step 1: Understand the need for balance.

    Fraud detection requires a balance between catching frauds and minimizing false alarms.

  2. Step 2: Choose the metric that balances precision and recall.

    F1 Score is the harmonic mean of precision and recall, providing a balance.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    F1 Score balances precision and recall. [OK]
Hint: F1 Score balances precision and recall. [OK]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard