Parent-child document retrieval in Prompt Engineering / GenAI - Model Pipeline Trace

Model Pipeline - Parent-child document retrieval

This pipeline helps find related documents where some are parents and others are children. It learns to match children to their parents using text features and relationships.

Data Flow - 5 Stages

1Raw documents input

1000 documents (mixed parents and children)→Load documents with parent-child links→1000 documents with metadata

Document 1: Parent, Document 2: Child of Document 1

↓

2Text preprocessing

1000 documents with raw text→Clean text, remove stopwords, tokenize→1000 documents with token lists

Original: 'The quick brown fox' → Tokens: ['quick', 'brown', 'fox']

↓

3Feature extraction

1000 documents with tokens→Convert tokens to embeddings (vectors)→1000 documents x 300-dim vectors

Document vector: [0.12, -0.05, ..., 0.33]

↓

4Parent-child pair creation

1000 documents with embeddings→Pair child documents with their parents→800 pairs (child vector + parent vector)

Pair: Child vector + Parent vector

↓

5Model training

800 pairs of vectors→Train neural network to score parent-child match→Trained model

Model learns to output high score for true pairs

Training Trace - Epoch by Epoch

Loss
1.0 | *       
0.8 |  *      
0.6 |   *     
0.4 |    *    
0.2 |     *   
0.0 +---------
      1 2 3 4 5
      Epochs

Epoch	Loss ↓	Accuracy ↑	Observation
1	0.85	0.6	Model starts learning basic patterns
2	0.65	0.72	Loss decreases, accuracy improves
3	0.5	0.8	Model captures parent-child relations better
4	0.4	0.85	Training converging, good match scores
5	0.35	0.88	Final epoch, stable performance

Prediction Trace - 5 Layers

Layer 1: Input child document embedding

Layer 2: Input parent document embedding

Layer 3: Concatenate embeddings

Layer 4: Neural network layers

Layer 5: Output layer with sigmoid

Model Quiz - 3 Questions

What happens to the data shape after feature extraction?

A Documents are converted to raw text

B Documents are split into sentences

C Documents become vectors with fixed length

D Documents are paired with unrelated documents

Key Insight

This visualization shows how a model learns to connect child documents to their parents by turning text into vectors and training on pairs. The decreasing loss and increasing accuracy tell us the model is improving its understanding of document relationships.