Bird
Raised Fist0

Ranking Metrics - NDCG, MAP, MRR - How Search and Rec Systems Measure Quality

THEORYintermediateInterview freq: very_high
Choose your preparation mode6 modes available
▶
Execution Trace
Step 1/4
NDCG Calculation

NDCG score is computed as DCG/IDCG, where DCG is 5.0 and IDCG is 6.0, resulting in 0.833.

🎯 Understanding how NDCG normalizes DCG to account for ideal ranking.
dcg = 5.0
idcg = 6.0
ndcg = dcg / idcg
MAP Calculation

MAP is calculated as the mean of average precisions: (0.8 + 0.6) / 2 = 0.7.

🎯 Grasping how MAP averages precision across queries.
avg_precisions = [0.8, 0.6]
map_score = sum(avg_precisions) / len(avg_precisions)
MRR Calculation

MRR is computed as the average reciprocal rank: (1/1 + 1/3) / 2 = 0.667.

🎯 Recognizing how MRR focuses on the rank of the first relevant item.
reciprocal_ranks = [1/1, 1/3]
mrr = sum(reciprocal_ranks) / len(reciprocal_ranks)
Impact of Position on NDCG

A relevant item at position 1 contributes more to DCG than at position 3.

🎯 Understanding the importance of position in NDCG calculations.
dcg_position_1 = 3.0
dcg_position_3 = 1.0
📊
Diagram
Diagram
┌───────────────────────────────┐
│          Ranking Metrics      │
├────────────┬──────────────────┤
│ NDCG       │ Normalized DCG   │
│            │ Measures ranking │
│            │ quality based on │
│            │ position         │
├────────────┼──────────────────┤
│ MAP        │ Mean Avg Prec.   │
│            │ Avg precision of │
│            │ relevant items   │
├────────────┼──────────────────┤
│ MRR        │ Mean Recip Rank  │
│            │ Avg reciprocal   │
│            │ rank of first    │
│            │ relevant item    │
└────────────┴──────────────────┘
  • NDCG: Normalized Discounted Cumulative Gain
  • MAP: Mean Average Precision
  • MRR: Mean Reciprocal Rank
🔑
Key Moments
▲
Q: Why does NDCG use a logarithmic discount factor?
A: The logarithmic discount factor reduces the contribution of items as their rank position increases, emphasizing the importance of higher-ranked items.
Q: How does MAP differ from precision?
A: MAP averages the precision at each relevant item across multiple queries, while precision measures the fraction of relevant items in a single query.
Q: What is the significance of the first relevant item in MRR?
A: MRR emphasizes the rank of the first relevant item, rewarding systems that retrieve relevant items earlier in the ranking.

This diagram and trace explain the ranking metrics NDCG, MAP, and MRR used in search and recommendation systems. NDCG evaluates ranking quality by considering the position of items, MAP averages precision across queries, and MRR focuses on the rank of the first relevant item. Understanding these metrics helps in assessing the effectiveness of ranking algorithms, emphasizing different aspects of ranking quality such as position, precision, and early retrieval of relevant items.

Primary Focus

✓ Understanding ranking metrics in search and recommendation systems

Practice

(1/5)
1.
import torch
x = torch.tensor([-1.0, 0.0, 1.0])
output = torch.nn.functional.relu(x)
print(output)
What is the output of this code?
easy
A. tensor([0.0, 0.0, 1.0])
B. tensor([-1.0, 0.0, 1.0])
C. tensor([0.0, 0.0, 0.0])
D. tensor([-1.0, 0.0, 0.0])

Solution

  1. Step 1: Understand ReLU behavior.

    ReLU outputs zero for negative inputs and the input itself for non-negative inputs.
  2. Step 2: Apply ReLU to each element.

    For -1.0, output is 0.0; for 0.0, output is 0.0; for 1.0, output is 1.0.
  3. Final Answer:

    Option A -> Option A
  4. Quick Check:

    ReLU zeros negatives, keeps positives. [OK]
Hint: ReLU zeros negatives, keeps positives. [OK]
Common Mistakes:
2. Given the following Python code snippet, what will be the output if the dataset contains data leakage?
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
model = LogisticRegression()
model.fit(X_train, y_train)
y_pred = model.predict(X_test)
print(accuracy_score(y_test, y_pred))
easy
A. The accuracy will be lower than expected.
B. The code will throw an error due to data leakage.
C. The accuracy will be exactly as expected.
D. The accuracy will be higher than expected.

Solution

  1. Step 1: Understand the impact of data leakage

    Data leakage often results in overly optimistic performance metrics.
  2. Step 2: Analyze the code behavior

    The code will execute without errors, but if leakage exists, the model will perform better than it should.
  3. Final Answer:

    Option D -> Option D
  4. Quick Check:

    Leakage inflates accuracy [OK]
Hint: Leakage inflates accuracy [OK]
Common Mistakes:
3. In the context of Word2Vec, which of the following best describes the purpose of the 'skip-gram' model?
easy
A. Generates a sequence of words from a given context.
B. Predicts the next word in a sequence given the previous words.
C. Clusters words into predefined categories.
D. Predicts surrounding words given a target word.

Solution

  1. Step 1: Understand skip-gram model purpose

    The skip-gram model in Word2Vec is designed to predict surrounding words given a target word.
  2. Step 2: Eliminate wrong options

    Predicts the next word in a sequence given the previous words. describes a language model task, Clusters words into predefined categories. is about clustering, and Generates a sequence of words from a given context. is about sequence generation, none of which are the purpose of skip-gram.
  3. Final Answer:

    Option D -> Option D
  4. Quick Check:

    Skip-gram predicts context from target [OK]
Hint: Skip-gram predicts context from target [OK]
Common Mistakes:
4. Given the following Python code snippet, what will be the output?
import numpy as np
business_goals = np.array(['increase sales', 'reduce churn', 'improve satisfaction'])
ml_problems = np.array(['classification', 'regression', 'clustering'])
result = np.char.add(business_goals, ml_problems)
print(result)
easy
A. ['increase salesclassification', 'reduce churnregression', 'improve satisfactionclustering']
B. ['increase sales', 'reduce churn', 'improve satisfaction']
C. ['classification', 'regression', 'clustering']
D. Throws an error due to incompatible types

Solution

  1. Step 1: Understand the operation

    The np.char.add function concatenates corresponding elements of two arrays element-wise.
  2. Step 2: Eliminate wrong options

    ['increase sales', 'reduce churn', 'improve satisfaction'] and C are incorrect because they ignore the concatenation operation. Throws an error due to incompatible types is incorrect because np.char.add handles string arrays correctly.
  3. Final Answer:

    Option A -> Option A
  4. Quick Check:

    np.char.add concatenates strings element-wise. [OK]
Hint: np.char.add concatenates strings element-wise. [OK]
Common Mistakes:
5. In a CI/CD pipeline for ML, when should you prefer using a canary deployment over a blue-green deployment?
medium
A. When you want zero downtime during deployment.
B. When you need to rollback quickly in case of failure.
C. When you want to completely replace the old version with the new one immediately.
D. When you need to test new features on a small subset of users.

Solution

  1. Step 1: Understand deployment strategies.

    Canary deployment gradually rolls out changes to a small subset, while blue-green switches between two environments.
  2. Step 2: Match scenarios to strategies.

    Canary is ideal for testing on a small subset (B), blue-green is for zero downtime (A) and quick rollback (D).
  3. Final Answer:

    Option D -> Option D
  4. Quick Check:

    Canary = gradual rollout. [OK]
Hint: Canary = gradual rollout. [OK]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard