Parent-child document retrieval in Prompt Engineering / GenAI - Model Metrics & Evaluation

Metrics & Evaluation - Parent-child document retrieval

Which metric matters for Parent-child document retrieval and WHY

In parent-child document retrieval, the goal is to find the correct child documents linked to a parent document or vice versa. The key metrics are Precision and Recall. Precision tells us how many retrieved documents are actually correct, while Recall tells us how many correct documents we found out of all possible correct ones. Since missing relevant child or parent documents can be costly, Recall is often very important. However, too many wrong matches (low Precision) can confuse users. So, both metrics matter to balance accuracy and completeness.

Confusion matrix for Parent-child document retrieval

                Predicted Relevant   Predicted Not Relevant
Actual Relevant        TP = 80               FN = 20
Actual Not Relevant    FP = 10               TN = 90

Total samples = 80 + 20 + 10 + 90 = 200

Precision = TP / (TP + FP) = 80 / (80 + 10) = 0.89
Recall = TP / (TP + FN) = 80 / (80 + 20) = 0.80

This matrix shows how many parent-child pairs were correctly retrieved (TP), missed (FN), wrongly retrieved (FP), or correctly ignored (TN).

Precision vs Recall tradeoff with examples

Imagine a system retrieving child documents for a parent article:

• High Precision, Low Recall: The system returns only very confident matches, so most retrieved are correct, but it misses many relevant child documents. This is good if you want to avoid wrong links but bad if you want complete information.
• High Recall, Low Precision: The system returns many child documents including most relevant ones, but also many irrelevant ones. This is good if you want to find all possible matches but bad if you want to avoid noise.

Choosing the right balance depends on the use case. For example, a legal document search might prioritize Recall to not miss any related documents, while a recommendation system might prioritize Precision to avoid irrelevant suggestions.

What good vs bad metric values look like for this use case

• Good: Precision and Recall both above 0.85 means most retrieved parent-child pairs are correct and most relevant pairs are found.
• Acceptable: Precision around 0.75 and Recall around 0.75 means some errors and misses but still useful retrieval.
• Bad: Precision below 0.5 or Recall below 0.5 means many wrong matches or many relevant pairs missed, making the retrieval unreliable.

Common pitfalls in metrics for Parent-child document retrieval

• Accuracy paradox: If most documents have no children, a model that always predicts no child will have high accuracy but be useless.
• Data leakage: If child documents appear in training and test sets, metrics will be overly optimistic.
• Overfitting: Very high training metrics but poor test metrics indicate the model memorizes links instead of generalizing.
• Ignoring class imbalance: If relevant parent-child pairs are rare, accuracy is misleading; focus on Precision and Recall instead.

Self-check question

Your parent-child retrieval model has 98% accuracy but only 12% recall on relevant child documents. Is it good for production? Why or why not?

Answer: No, it is not good. The high accuracy likely comes from many irrelevant pairs correctly predicted as irrelevant. But the very low recall means the model misses most relevant child documents, which defeats the purpose of retrieval. Improving recall is critical.

Key Result

Precision and Recall are key metrics; high recall ensures relevant parent-child documents are found, while high precision ensures retrieved links are correct.

Practice

1. What is the main purpose of parent-child document retrieval in GenAI systems?

easy

A. To find related documents where one is the parent and others are children

B. To sort documents alphabetically

C. To delete duplicate documents automatically

D. To translate documents into different languages

Solution

1. Step 1: Understand parent-child relationship

   Parent-child document retrieval means finding documents linked by a hierarchical relationship, where one document is the parent and others are its children.

2. Step 2: Identify retrieval goal

   The goal is to retrieve documents that are connected in this way, not just any documents or unrelated tasks like sorting or translating.

3. Final Answer:

   To find related documents where one is the parent and others are children -> Option A

4. Quick Check:

   Parent-child retrieval = find related hierarchical documents [OK]

Hint: Think hierarchy: parent document with linked child documents [OK]

Common Mistakes:

• Confusing retrieval with sorting or translation
• Ignoring the hierarchical link between documents
• Assuming it deletes or modifies documents

2. Which of the following is the correct syntax to query child documents given a parent ID in a GenAI retrieval system?

easy

A. query = {"parent": "12345"}

B. query = {"child_of": "12345"}

C. query = {"parent_id": "12345"}

D. query = {"child_id": "12345"}

Solution

1. Step 1: Identify correct key for parent ID

   In GenAI retrieval, the key to specify parent document ID for child retrieval is usually "parent_id".

2. Step 2: Check other options for correctness

   Options like "child_of", "parent", or "child_id" are not standard or correct keys for this query.

3. Final Answer:

   query = {"parent_id": "12345"} -> Option C

4. Quick Check:

   Use "parent_id" key to query children [OK]

Hint: Look for "parent_id" key to find children documents [OK]

Common Mistakes:

• Using incorrect keys like "child_of" or "child_id"
• Confusing parent and child identifiers
• Omitting quotes around keys or values

3. Given the following code snippet for retrieving child documents, what will be the output if the parent ID has two children with IDs 'c1' and 'c2'?

parent_id = 'p123'
children = retrieve_children(parent_id)
print(children)

medium

A. ['c1', 'c2']

B. ['p123']

C. []

D. Error: retrieve_children not defined

Solution

1. Step 1: Understand function purpose

   The function retrieve_children(parent_id) is designed to return a list of child document IDs for the given parent ID.

2. Step 2: Analyze given data

   Since the parent ID 'p123' has two children with IDs 'c1' and 'c2', the function should return these IDs in a list.

3. Final Answer:

   ['c1', 'c2'] -> Option A

4. Quick Check:

   retrieve_children returns child IDs list [OK]

Hint: Function returns list of children IDs for given parent [OK]

Common Mistakes:

• Assuming it returns parent ID instead of children
• Expecting empty list when children exist
• Confusing function name or missing definition

4. You have this code snippet to retrieve parent documents but it raises an error:

def get_parent(child_id):
    return retrieve_parent(child_id)

print(get_parent('c123'))

What is the most likely cause of the error?

medium

A. The function get_parent has wrong indentation

B. The child_id 'c123' does not exist

C. The print statement syntax is incorrect

D. The function retrieve_parent is not defined or imported

Solution

1. Step 1: Check function usage

   The function get_parent calls retrieve_parent, which must be defined or imported to work.

2. Step 2: Identify error cause

   If retrieve_parent is missing, Python raises a NameError. Other options like child ID missing or print syntax error would cause different errors.

3. Final Answer:

   The function retrieve_parent is not defined or imported -> Option D

4. Quick Check:

   Undefined function causes NameError [OK]

Hint: Check if all called functions are defined or imported [OK]

Common Mistakes:

• Assuming child ID missing causes this error
• Thinking print syntax is wrong
• Ignoring missing function definitions

5. You want to retrieve all child documents for multiple parent documents efficiently. Which approach best applies parent-child document retrieval in GenAI to achieve this?

hard

A. Query each parent ID separately in a loop and combine results

B. Batch query using a list of parent IDs to fetch all children at once

C. Retrieve all documents and filter children manually by parent ID

D. Use a random sampling of documents ignoring parent-child links

Solution

1. Step 1: Understand efficiency in retrieval

   Batch querying multiple parent IDs at once reduces repeated calls and speeds up retrieval.

2. Step 2: Compare approaches

   Querying separately is slower; filtering all documents wastes resources; random sampling ignores relationships.

3. Final Answer:

   Batch query using a list of parent IDs to fetch all children at once -> Option B

4. Quick Check:

   Batch queries improve efficiency in parent-child retrieval [OK]

Hint: Batch queries reduce calls and speed retrieval [OK]

Common Mistakes:

• Querying parents one by one causing slow performance
• Filtering all documents instead of targeted retrieval
• Ignoring parent-child relationships in sampling