Dependency parsing in NLP - Model Metrics & Evaluation

Metrics & Evaluation - Dependency parsing

Which metric matters for Dependency Parsing and WHY

Dependency parsing finds the relationships between words in a sentence. The key metrics are Unlabeled Attachment Score (UAS) and Labeled Attachment Score (LAS).

UAS measures how many words are correctly linked to their parent word, ignoring the type of relation. LAS measures how many words are correctly linked with the right relation type.

These metrics matter because they show how well the model understands sentence structure and meaning.

Confusion Matrix or Equivalent Visualization

Dependency parsing does not use a classic confusion matrix like classification. Instead, we count correct and incorrect arcs (links) between words.

Total words: 1000
Correct head links (UAS): 900
Incorrect head links: 100
Correct labeled links (LAS): 850
Incorrect labeled links: 150

UAS = 900 / 1000 = 0.90 (90%)
LAS = 850 / 1000 = 0.85 (85%)
    

This shows 90% of words have the correct parent word, and 85% have the correct parent and relation type.

Precision vs Recall Tradeoff in Dependency Parsing

Dependency parsing focuses on accuracy of links, so precision and recall are less common metrics here. But if we think of arcs as predictions:

• Precision: How many predicted links are correct?
• Recall: How many true links did the model find?

High precision means few wrong links, high recall means few missed links. A parser with high precision but low recall misses many relations, while one with high recall but low precision adds many wrong links.

In practice, UAS and LAS balance these aspects by measuring correct links over total words.

What Good vs Bad Metric Values Look Like

Good dependency parsers have UAS and LAS above 85% on standard datasets.

• Good: UAS = 90%, LAS = 85% or higher. This means most words are linked correctly with the right relation.
• Bad: UAS below 70%, LAS below 60%. This means many words are linked incorrectly or with wrong relations, making sentence understanding poor.

Higher LAS is harder to achieve because it requires correct relation types, not just correct links.

Common Metrics Pitfalls in Dependency Parsing

• Ignoring relation labels: Reporting only UAS hides errors in relation types, which are important for meaning.
• Data leakage: Training and testing on overlapping sentences inflates scores falsely.
• Overfitting: Very high scores on training but low on test data show the model memorizes rather than generalizes.
• Ignoring sentence length: Longer sentences are harder to parse; average scores may hide poor performance on complex sentences.

Self Check: Your model has 98% UAS but 50% LAS. Is it good?

No, this means the model finds the correct parent word for most words (98%), but only half the time it assigns the correct relation type (50%).

This is a problem because understanding sentence meaning depends on correct relations, not just links.

You should improve the model to raise LAS closer to UAS for better real-world use.

Key Result

Unlabeled Attachment Score (UAS) and Labeled Attachment Score (LAS) are key metrics showing how well a dependency parser finds correct word links and relation types.

Practice

1. What is the main purpose of dependency parsing in Natural Language Processing?

easy

A. To show how words in a sentence are connected

B. To translate sentences into another language

C. To count the number of words in a sentence

D. To generate new sentences automatically

Solution

1. Step 1: Understand dependency parsing

   Dependency parsing analyzes sentence structure by showing relationships between words.

2. Step 2: Compare options

   Only To show how words in a sentence are connected correctly describes this purpose; others describe different NLP tasks.

3. Final Answer:

   To show how words in a sentence are connected -> Option A

4. Quick Check:

   Dependency parsing = word connections [OK]

Hint: Dependency parsing = word connection map [OK]

Common Mistakes:

• Confusing parsing with translation
• Thinking it counts words only
• Mixing with sentence generation

2. Which of the following is the correct way to access the dependency label of a token using spaCy in Python?
doc = nlp('I love cats')

easy

A. doc[1].dep_

B. doc.dep_[1]

C. doc[1].dependency

D. doc.dep[1]

Solution

1. Step 1: Recall spaCy token attributes

   In spaCy, each token has a dep_ attribute accessed by doc[index].dep_.

2. Step 2: Check options for correct syntax

   Only doc[1].dep_ uses correct attribute and indexing syntax.

3. Final Answer:

   doc[1].dep_ -> Option A

4. Quick Check:

   Token dependency label = doc[index].dep_ [OK]

Hint: Use token.dep_ to get dependency label [OK]

Common Mistakes:

• Using wrong attribute name like dep or dependency
• Trying to index dep_ attribute
• Confusing token and doc object

3. Given the code below, what will be the output?

import spacy
nlp = spacy.load('en_core_web_sm')
doc = nlp('She eats an apple')
for token in doc:
    print(f'{token.text} -> {token.dep_}')

medium

A. She -> det
eats -> dobj
an -> nsubj
apple -> ROOT

B. She -> dobj
eats -> nsubj
an -> ROOT
apple -> det

C. She -> ROOT
eats -> nsubj
an -> dobj
apple -> det

D. She -> nsubj
eats -> ROOT
an -> det
apple -> dobj

Solution

1. Step 1: Understand dependency roles in sentence

   In 'She eats an apple', 'eats' is the main verb (ROOT), 'She' is subject (nsubj), 'an' is determiner (det), 'apple' is direct object (dobj).

2. Step 2: Match roles to output

   She -> nsubj eats -> ROOT an -> det apple -> dobj correctly matches each word to its dependency label.

3. Final Answer:

   She -> nsubj eats -> ROOT an -> det apple -> dobj -> Option D

4. Quick Check:

   Subject = nsubj, Verb = ROOT, Object = dobj [OK]

Hint: Main verb is ROOT; subject is nsubj; object is dobj [OK]

Common Mistakes:

• Mixing subject and object labels
• Confusing determiner with object
• Assuming first word is ROOT

4. Identify the error in this spaCy dependency parsing code:

import spacy
nlp = spacy.load('en_core_web_sm')
doc = nlp('Dogs bark loudly')
for token in doc:
    print(token.dep)

medium

A. Incorrect model name in spacy.load

B. doc should be a list, not a spaCy Doc object

C. Missing underscore in token.dep_ attribute

D. print statement syntax is wrong

Solution

1. Step 1: Check token attribute usage

   spaCy tokens use dep_ (with underscore) to get dependency label as string; dep without underscore returns an integer ID.

2. Step 2: Verify code correctness

   Code uses token.dep which prints integer IDs, not readable labels; likely intended to print labels, so underscore is missing.

3. Final Answer:

   Missing underscore in token.dep_ attribute -> Option C

4. Quick Check:

   Use token.dep_ for labels, not token.dep [OK]

Hint: Use token.dep_ (with underscore) for readable labels [OK]

Common Mistakes:

• Using token.dep instead of token.dep_
• Assuming doc is wrong type
• Thinking print syntax is incorrect

5. You want to extract all verbs and their direct objects from a sentence using dependency parsing. Which approach is best?

hard

A. Use only token text without parsing dependencies

B. Find tokens with POS tag 'VERB' and check their children with dependency label 'dobj'

C. Extract tokens with POS tag 'NOUN' ignoring dependencies

D. Select tokens with dependency label 'nsubj' only

Solution

1. Step 1: Understand task requirements

   We want verbs and their direct objects, so we need to find verbs and check which tokens depend on them as direct objects (dobj).

2. Step 2: Evaluate options

   Find tokens with POS tag 'VERB' and check their children with dependency label 'dobj' correctly finds verbs and their dobj children. Others ignore dependencies or focus on subjects or nouns only.

3. Final Answer:

   Find tokens with POS tag 'VERB' and check their children with dependency label 'dobj' -> Option B

4. Quick Check:

   Verbs + dobj children = correct extraction [OK]

Hint: Look for verbs and their dobj children in dependency tree [OK]

Common Mistakes:

• Ignoring dependency labels
• Selecting only subjects
• Using POS tags without dependencies