Document processing pipeline in NLP - Model Metrics & Evaluation

Metrics & Evaluation - Document processing pipeline

Which metric matters for Document Processing Pipeline and WHY

In document processing, we often want to extract correct information or classify documents accurately. Key metrics are Precision, Recall, and F1 score. Precision tells us how many extracted items are actually correct. Recall tells us how many correct items we found out of all possible correct items. F1 score balances both. These metrics matter because missing important info (low recall) or adding wrong info (low precision) both hurt the pipeline's usefulness.

Confusion Matrix Example

      | Predicted Positive | Predicted Negative |
      |--------------------|--------------------|
      | True Positive (TP)  | False Negative (FN) |
      | False Positive (FP) | True Negative (TN)  |

      Example:
      TP = 80 (correctly extracted entities)
      FP = 20 (wrongly extracted entities)
      FN = 10 (missed entities)
      TN = 890 (correctly ignored non-entities)

      Total samples = 80 + 20 + 10 + 890 = 1000
    

Precision vs Recall Tradeoff with Examples

Imagine a document pipeline extracting names from contracts.

• High Precision, Low Recall: The pipeline extracts only very sure names, so most are correct (few false alarms), but it misses many names. Good if you want very reliable info but can tolerate missing some.
• High Recall, Low Precision: The pipeline extracts many names, catching almost all real ones, but also many wrong ones. Good if missing any name is bad, but you can clean errors later.

Choosing depends on what matters more: missing info or wrong info.

Good vs Bad Metric Values for Document Processing

• Good: Precision and Recall both above 0.85, F1 score above 0.85 means the pipeline extracts info accurately and mostly completely.
• Bad: Precision below 0.5 means many wrong extractions; Recall below 0.5 means many missed items; F1 below 0.6 means poor balance and unreliable extraction.

Common Pitfalls in Metrics for Document Processing

• Accuracy Paradox: If most documents have no entities, accuracy can be high by always predicting no entities, but the model is useless.
• Data Leakage: Using test documents in training inflates metrics falsely.
• Overfitting: Very high training metrics but low test metrics means the pipeline learned noise, not real patterns.
• Ignoring Class Imbalance: Many documents may have few entities; metrics must consider this to avoid misleading results.

Self Check

Your document processing pipeline has 98% accuracy but only 12% recall on extracting key entities. Is it good for production? Why or why not?

Answer: No, it is not good. The high accuracy is misleading because most parts of documents have no entities, so predicting no entities often is correct. But 12% recall means it misses 88% of important entities, which defeats the purpose of extraction. You need to improve recall while keeping precision reasonable.

Key Result

Precision, Recall, and F1 score are key to measure how well a document processing pipeline extracts correct and complete information.

Practice

1. What is the main purpose of a document processing pipeline in NLP?

easy

A. To break down text tasks into smaller, manageable steps

B. To store documents in a database

C. To translate documents into multiple languages

D. To generate random text from documents

Solution

1. Step 1: Understand the pipeline concept

   A document processing pipeline divides a big task into smaller steps to handle text better.

2. Step 2: Identify the main goal

   The goal is to make complex text easier to process by breaking it down.

3. Final Answer:

   To break down text tasks into smaller, manageable steps -> Option A

4. Quick Check:

   Pipeline purpose = break down tasks [OK]

Hint: Think of a pipeline as a step-by-step recipe for text [OK]

Common Mistakes:

• Confusing pipeline with storage or translation
• Thinking pipeline generates text
• Ignoring the step-by-step nature

2. Which of the following is the correct order of steps in a simple document processing pipeline?

easy

A. Stopword Removal -> Lemmatization -> Tokenization

B. Lemmatization -> Tokenization -> Stopword Removal

C. Tokenization -> Stopword Removal -> Lemmatization

D. Tokenization -> Lemmatization -> Stopword Removal

Solution

1. Step 1: Recall common pipeline steps

   Tokenization splits text into words, stopword removal deletes common words, lemmatization reduces words to base form.

2. Step 2: Determine logical order

   First split text (tokenize), then remove stopwords, then lemmatize remaining words.

3. Final Answer:

   Tokenization -> Stopword Removal -> Lemmatization -> Option C

4. Quick Check:

   Order = tokenize, remove stopwords, lemmatize [OK]

Hint: Split text first, then clean, then normalize words [OK]

Common Mistakes:

• Removing stopwords before tokenizing
• Lemmatizing before tokenizing
• Mixing step order randomly

3. Given this Python snippet in a document pipeline:

text = "Cats are running fast"
tokens = text.lower().split()
filtered = [w for w in tokens if w not in ['are', 'is', 'the']]
print(filtered)

What is the output?

medium

A. ['cats', 'running', 'fast']

B. ['Cats', 'are', 'running', 'fast']

C. ['cats', 'are', 'running', 'fast']

D. ['running', 'fast']

Solution

1. Step 1: Lowercase and split text

   "Cats are running fast" becomes ['cats', 'are', 'running', 'fast'] after lower() and split().

2. Step 2: Remove stopwords

   Words 'are', 'is', 'the' are removed, so 'are' is removed from the list.

3. Final Answer:

   ['cats', 'running', 'fast'] -> Option A

4. Quick Check:

   Stopwords removed = ['cats', 'running', 'fast'] [OK]

Hint: Lowercase then remove stopwords from tokens [OK]

Common Mistakes:

• Not lowercasing before filtering
• Including stopwords in output
• Confusing original and filtered lists

4. This code is part of a document pipeline:

def clean_text(text):
    tokens = text.split()
    tokens = [t.lower() for t in tokens]
    tokens = [t for t in tokens if t not in stopwords]
    tokens = lemmatize(tokens)
    return tokens

stopwords = ['and', 'the', 'is']

print(clean_text("The cats and dogs are playing"))

What is the error here?

medium

A. text.split() should be text.lower().split()

B. lemmatize function is not defined

C. stopwords list is empty

D. tokens list is not returned

Solution

1. Step 1: Check function definitions

   The code calls lemmatize(tokens) but no lemmatize function is defined or imported.

2. Step 2: Verify other parts

   stopwords list is defined, tokens are returned, and text is split correctly.

3. Final Answer:

   lemmatize function is not defined -> Option B

4. Quick Check:

   Missing lemmatize function causes error [OK]

Hint: Check if all functions used are defined or imported [OK]

Common Mistakes:

• Assuming lemmatize is built-in
• Ignoring missing function errors
• Thinking stopwords list is empty

5. You want to build a document processing pipeline that extracts keywords from large documents. Which sequence of steps is best?

hard

A. POS Tagging -> Keyword Extraction -> Tokenization -> Stopword Removal

B. Keyword Extraction -> Tokenization -> Stopword Removal -> POS Tagging

C. Stopword Removal -> Tokenization -> Keyword Extraction -> POS Tagging

D. Tokenization -> Stopword Removal -> POS Tagging -> Keyword Extraction

Solution

1. Step 1: Understand keyword extraction needs

   Extracting keywords requires clean tokens and knowing word types (POS tags) to pick important words.

2. Step 2: Arrange logical steps

   First tokenize text, remove stopwords to clean, then tag parts of speech, finally extract keywords based on tags.

3. Final Answer:

   Tokenization -> Stopword Removal -> POS Tagging -> Keyword Extraction -> Option D

4. Quick Check:

   Pipeline order = tokenize, clean, tag, extract [OK]

Hint: Clean tokens before tagging and extracting keywords [OK]

Common Mistakes:

• Extracting keywords before tokenizing
• Tagging before cleaning tokens
• Wrong step order breaks pipeline