NER with NLTK in NLP - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

What is the output of this NER chunking code?

Given the following Python code using NLTK for Named Entity Recognition (NER), what is the output?

NLP

import nltk
from nltk import word_tokenize, pos_tag, ne_chunk
sentence = "Apple is looking at buying U.K. startup for $1 billion"
tokens = word_tokenize(sentence)
pos_tags = pos_tag(tokens)
chunks = ne_chunk(pos_tags)
print(list(chunks))

A [('Apple', 'NNP'), ('is', 'VBZ'), ('looking', 'VBG'), ('at', 'IN'), Tree('ORGANIZATION', [('buying', 'VBG')]), ('U.K.', 'NNP'), ('startup', 'NN'), ('for', 'IN'), ('$', '$'), ('1', 'CD'), ('billion', 'CD')]

B [Tree('GPE', [('Apple', 'NNP')]), ('is', 'VBZ'), ('looking', 'VBG'), ('at', 'IN'), ('buying', 'VBG'), Tree('GPE', [('U.K.', 'NNP')]), ('startup', 'NN'), ('for', 'IN'), ('$', '$'), ('1', 'CD'), ('billion', 'CD')]

C [('Apple', 'NNP'), ('is', 'VBZ'), ('looking', 'VBG'), ('at', 'IN'), ('buying', 'VBG'), ('U.K.', 'NNP'), ('startup', 'NN'), ('for', 'IN'), ('$', '$'), ('1', 'CD'), ('billion', 'CD')]

D [Tree('ORGANIZATION', [('Apple', 'NNP')]), ('is', 'VBZ'), ('looking', 'VBG'), ('at', 'IN'), ('buying', 'VBG'), Tree('GPE', [('U.K.', 'NNP')]), ('startup', 'NN'), ('for', 'IN'), ('$', '$'), ('1', 'CD'), ('billion', 'CD')]

💡 Hint

Look at how NLTK labels named entities like organizations and geopolitical entities.

NLTK's ne_chunk labels 'Apple' as an ORGANIZATION and 'U.K.' as a GPE (Geopolitical Entity). The output is a list mixing Tree objects for entities and tuples for other tokens.

🧠 Conceptual

intermediate

Which NLTK function performs POS tagging before NER?

In the NLTK pipeline for Named Entity Recognition, which function is responsible for assigning part-of-speech tags to tokens before chunking?

A ne_chunk()

B word_tokenize()

C pos_tag()

D sent_tokenize()

💡 Hint

POS tags describe the role of each word like noun or verb.

pos_tag() assigns part-of-speech tags to tokens, which ne_chunk() then uses to identify named entities.

❓ Hyperparameter

advanced

What is the effect of using binary=True in ne_chunk()?

In NLTK's ne_chunk function, setting binary=True changes the output. What does this parameter do?

A It merges all named entities into a single 'NE' label without specific types.

B It disables named entity recognition and returns only POS tags.

C It enables recognition of only person names, ignoring other entity types.

D It outputs entities as plain strings instead of Tree objects.

💡 Hint

Think about simplifying entity categories.

binary=True groups all named entities under a generic 'NE' label, ignoring specific types like ORGANIZATION or GPE.

🔧 Debug

advanced

Why does this NER code raise a LookupError?

Consider this code snippet: import nltk sentence = "Google is a tech giant" tokens = nltk.word_tokenize(sentence) pos_tags = nltk.pos_tag(tokens) chunks = nltk.ne_chunk(pos_tags) When running, it raises a LookupError about missing 'maxent_ne_chunker'. What is the cause?

A The sentence contains unknown words causing the error.

B The 'maxent_ne_chunker' model is not downloaded in NLTK data.

C pos_tag() was not called before ne_chunk().

D word_tokenize() requires a language parameter that is missing.

💡 Hint

Check if all required NLTK models are installed.

NLTK requires the 'maxent_ne_chunker' model to perform named entity chunking. If not downloaded, it raises LookupError.

❓ Model Choice

expert

Which NLTK model is used internally by ne_chunk for NER?

NLTK's ne_chunk function uses a pre-trained model internally. Which model does it use for Named Entity Recognition?

A A Maximum Entropy classifier trained on the ACE corpus

B A Hidden Markov Model trained on the CoNLL 2003 dataset

C A Maximum Entropy classifier trained on the CONLL 2000 corpus

D A Maximum Entropy classifier trained on the ACE corpus and Treebank data

💡 Hint

Consider the training data NLTK mentions for ne_chunk.

NLTK's ne_chunk uses a Maximum Entropy classifier trained on the ACE corpus for NER.

Practice

1. What is the main purpose of Named Entity Recognition (NER) in Natural Language Processing?

easy

A. To count the number of words in a sentence

B. To translate text from one language to another

C. To find names of people, places, and organizations in text

D. To correct spelling mistakes in text

Solution

1. Step 1: Understand NER's role

   NER is designed to identify and classify named entities like people, places, and organizations in text.

2. Step 2: Compare with other NLP tasks

   Translation, word counting, and spell checking are different tasks unrelated to NER.

3. Final Answer:

   To find names of people, places, and organizations in text -> Option C

4. Quick Check:

   NER = Find names [OK]

Hint: NER extracts names and places from text quickly [OK]

Common Mistakes:

• Confusing NER with translation
• Thinking NER counts words
• Mixing NER with spell checking

2. Which NLTK function is used to perform Named Entity Recognition after POS tagging?

easy

A. ne_chunk()

B. word_tokenize()

C. pos_tag()

D. sent_tokenize()

Solution

1. Step 1: Identify NLTK functions for NER

   NLTK uses ne_chunk() to recognize named entities from POS-tagged tokens.

2. Step 2: Differentiate from other functions

   word_tokenize() splits text into words, pos_tag() tags parts of speech, and sent_tokenize() splits text into sentences.

3. Final Answer:

   ne_chunk() -> Option A

4. Quick Check:

   NER uses ne_chunk() [OK]

Hint: Use ne_chunk() after pos_tag() for NER in NLTK [OK]

Common Mistakes:

• Using word_tokenize() for NER
• Confusing pos_tag() with NER
• Trying sent_tokenize() for entity recognition

3. What will be the output type of ne_chunk(pos_tag(word_tokenize(text))) in NLTK?

medium

A. A plain string with entity labels

B. A list of strings

C. A dictionary mapping words to entity types

D. A tree structure with named entities as subtrees

Solution

1. Step 1: Understand ne_chunk output

   The ne_chunk() function returns a tree structure where named entities are subtrees labeled with entity types.

2. Step 2: Compare output types

   It is not a list, dictionary, or plain string but a hierarchical tree that can be traversed.

3. Final Answer:

   A tree structure with named entities as subtrees -> Option D

4. Quick Check:

   ne_chunk output = tree structure [OK]

Hint: ne_chunk returns a tree, not a list or dict [OK]

Common Mistakes:

• Expecting a list of strings
• Thinking output is a dictionary
• Assuming output is a plain string

4. Given the code snippet:

import nltk
text = "Apple is looking at buying U.K. startup"
tokens = nltk.word_tokenize(text)
pos_tags = nltk.pos_tag(tokens)
entities = nltk.ne_chunk(pos_tags, binary=True)
print(entities)

What is the likely error in this code?

medium

A. Missing import for ne_chunk

B. Incorrect argument 'binary=True' in ne_chunk

C. pos_tag requires a list of sentences, not tokens

D. word_tokenize should be called after ne_chunk

Solution

1. Step 1: Check ne_chunk parameters

   The ne_chunk() function's binary=True limits it to binary NER (labels entities simply as NE, typically focusing on PERSON), which is incorrect for standard NER requiring specific types like PERSON, ORGANIZATION, GPE.

2. Step 2: Verify other parts

   Imports are correct with import nltk, pos_tag() accepts tokenized words, and preprocessing order is proper.

3. Final Answer:

   Incorrect argument 'binary=True' in ne_chunk -> Option B

4. Quick Check:

   binary=True limits to binary NER [OK]

Hint: Use binary=False for detailed entity types in ne_chunk [OK]

Common Mistakes:

• Using binary=True for detailed NER
• Calling word_tokenize after ne_chunk
• Misunderstanding pos_tag input

5. You want to extract only PERSON entities from a text using NLTK's ne_chunk. Which approach correctly filters PERSON entities from the chunked tree?

hard

A. Traverse the tree and select subtrees with label 'PERSON'

B. Use pos_tag to find tokens tagged as 'PERSON'

C. Filter tokens containing capital letters only

D. Use word_tokenize and select words starting with 'P'

Solution

1. Step 1: Understand ne_chunk output structure

   Named entities are subtrees labeled with entity types like 'PERSON', so we must traverse the tree to find these subtrees.

2. Step 2: Evaluate filtering methods

   pos_tag does not label entities, only parts of speech. Capital letters or starting with 'P' are unreliable heuristics.

3. Final Answer:

   Traverse the tree and select subtrees with label 'PERSON' -> Option A

4. Quick Check:

   Filter PERSON by subtree label [OK]

Hint: Filter PERSON entities by subtree label in ne_chunk tree [OK]

Common Mistakes:

• Using pos_tag to find entities
• Filtering by capitalization only
• Selecting words by first letter