Answer span extraction in NLP - Model Pipeline Trace

Model Pipeline - Answer span extraction

This pipeline finds the exact part of a text that answers a question. It reads the question and text, then predicts the start and end positions of the answer inside the text.

Data Flow - 5 Stages

1Input data

1000 samples x 2 texts (question and context)→Raw question and context pairs→1000 samples x 2 texts

Question: 'Where is the Eiffel Tower?'; Context: 'The Eiffel Tower is in Paris, France.'

↓

2Tokenization

1000 samples x 2 texts→Split texts into tokens and convert to numbers→1000 samples x 128 tokens

Tokens: ['[CLS]', 'Where', 'is', 'the', 'Eiffel', 'Tower', '?', '[SEP]', 'The', 'Eiffel', 'Tower', 'is', 'in', 'Paris', ',', 'France', '.', '[SEP]']

↓

3Model input preparation

1000 samples x 128 tokens→Create input IDs, attention masks, and token type IDs→1000 samples x 128 tokens x 3 arrays

Input IDs: [101, 2073, 2003, 1996, 3000, 2433, 1029, 102, 1996, 3000, 2433, 2003, 1999, 3000, 1010, 3000, 1012, 102]

↓

4Model prediction

1000 samples x 128 tokens x 3 arrays→Predict start and end logits for answer span→1000 samples x 128 start logits + 128 end logits

Start logits: [0.1, 0.2, ..., 5.0, ..., 0.1]; End logits: [0.1, 0.1, ..., 4.8, ..., 0.2]

↓

5Answer span extraction

1000 samples x 128 start logits + 128 end logits→Select tokens with highest start and end logits to form answer→1000 samples x answer text

Answer: 'Paris, France'

Training Trace - Epoch by Epoch

Loss
1.2 |*       
0.8 |  *     
0.5 |    *   
0.3 |      * 
0.25|       *
    +--------
     1 2 3 4 5 Epochs

Epoch	Loss ↓	Accuracy ↑	Observation
1	1.2	0.45	Model starts learning, loss high, accuracy low
2	0.8	0.60	Loss decreases, accuracy improves
3	0.5	0.75	Model learns better answer spans
4	0.3	0.85	Good convergence, loss low, accuracy high
5	0.25	0.88	Training stabilizes with good performance

Prediction Trace - 4 Layers

Layer 1: Tokenization

Layer 2: Model input preparation

Layer 3: Model prediction

Layer 4: Answer span extraction

Model Quiz - 3 Questions

What does the model predict to find the answer in the text?

A The full text of the answer directly

B Start and end positions of the answer span

C Only the start position of the answer

D The question rewritten

Key Insight

Answer span extraction models learn to find exact start and end points of answers in text by predicting positions, not by generating text. This makes them precise for question answering tasks.