Parent-child document retrieval in Prompt Engineering / GenAI - Model Metrics & Evaluation

Metrics & Evaluation - Parent-child document retrieval

Which metric matters for Parent-child document retrieval and WHY

In parent-child document retrieval, the goal is to find the correct child documents linked to a parent document or vice versa. The key metrics are Precision and Recall. Precision tells us how many retrieved documents are actually correct, while Recall tells us how many correct documents we found out of all possible correct ones. Since missing relevant child or parent documents can be costly, Recall is often very important. However, too many wrong matches (low Precision) can confuse users. So, both metrics matter to balance accuracy and completeness.

Confusion matrix for Parent-child document retrieval

                Predicted Relevant   Predicted Not Relevant
Actual Relevant        TP = 80               FN = 20
Actual Not Relevant    FP = 10               TN = 90

Total samples = 80 + 20 + 10 + 90 = 200

Precision = TP / (TP + FP) = 80 / (80 + 10) = 0.89
Recall = TP / (TP + FN) = 80 / (80 + 20) = 0.80

This matrix shows how many parent-child pairs were correctly retrieved (TP), missed (FN), wrongly retrieved (FP), or correctly ignored (TN).

Precision vs Recall tradeoff with examples

Imagine a system retrieving child documents for a parent article:

• High Precision, Low Recall: The system returns only very confident matches, so most retrieved are correct, but it misses many relevant child documents. This is good if you want to avoid wrong links but bad if you want complete information.
• High Recall, Low Precision: The system returns many child documents including most relevant ones, but also many irrelevant ones. This is good if you want to find all possible matches but bad if you want to avoid noise.

Choosing the right balance depends on the use case. For example, a legal document search might prioritize Recall to not miss any related documents, while a recommendation system might prioritize Precision to avoid irrelevant suggestions.

What good vs bad metric values look like for this use case

• Good: Precision and Recall both above 0.85 means most retrieved parent-child pairs are correct and most relevant pairs are found.
• Acceptable: Precision around 0.75 and Recall around 0.75 means some errors and misses but still useful retrieval.
• Bad: Precision below 0.5 or Recall below 0.5 means many wrong matches or many relevant pairs missed, making the retrieval unreliable.

Common pitfalls in metrics for Parent-child document retrieval

• Accuracy paradox: If most documents have no children, a model that always predicts no child will have high accuracy but be useless.
• Data leakage: If child documents appear in training and test sets, metrics will be overly optimistic.
• Overfitting: Very high training metrics but poor test metrics indicate the model memorizes links instead of generalizing.
• Ignoring class imbalance: If relevant parent-child pairs are rare, accuracy is misleading; focus on Precision and Recall instead.

Self-check question

Your parent-child retrieval model has 98% accuracy but only 12% recall on relevant child documents. Is it good for production? Why or why not?

Answer: No, it is not good. The high accuracy likely comes from many irrelevant pairs correctly predicted as irrelevant. But the very low recall means the model misses most relevant child documents, which defeats the purpose of retrieval. Improving recall is critical.

Key Result

Precision and Recall are key metrics; high recall ensures relevant parent-child documents are found, while high precision ensures retrieved links are correct.