Entity linking concept in NLP - ML Experiment: Train & Evaluate

Experiment - Entity linking concept

Problem:You want to build a model that links names or phrases in text to real-world entities, like matching 'Apple' to the company or the fruit depending on context.

Current Metrics:Training accuracy: 95%, Validation accuracy: 70%

Issue:The model is overfitting: it performs very well on training data but poorly on new, unseen text.

Your Task

Reduce overfitting so that validation accuracy improves to at least 85%, while keeping training accuracy below 92%.

You can only change model architecture and training hyperparameters.

You cannot add more training data.

Hint 1

Hint 2

Hint 3

Hint 4

Solution

NLP

import tensorflow as tf
from tensorflow.keras.layers import Input, Embedding, LSTM, Dense, Dropout
from tensorflow.keras.models import Model
from tensorflow.keras.callbacks import EarlyStopping

# Sample data placeholders (replace with real data loading)
X_train = tf.random.uniform((1000, 10), maxval=1000, dtype=tf.int32)
y_train = tf.random.uniform((1000,), maxval=5, dtype=tf.int32)
X_val = tf.random.uniform((200, 10), maxval=1000, dtype=tf.int32)
y_val = tf.random.uniform((200,), maxval=5, dtype=tf.int32)

vocab_size = 1000
embedding_dim = 64
num_classes = 5

inputs = Input(shape=(10,))
embedding = Embedding(vocab_size, embedding_dim)(inputs)
lstm = LSTM(64)(embedding)
drop = Dropout(0.5)(lstm)  # Added dropout
outputs = Dense(num_classes, activation='softmax')(drop)

model = Model(inputs, outputs)
model.compile(optimizer=tf.keras.optimizers.Adam(learning_rate=0.001),
              loss='sparse_categorical_crossentropy',
              metrics=['accuracy'])

early_stop = EarlyStopping(monitor='val_loss', patience=3, restore_best_weights=True)

history = model.fit(X_train, y_train, epochs=30, batch_size=32,
                    validation_data=(X_val, y_val), callbacks=[early_stop])

Added a Dropout layer with rate 0.5 after the LSTM layer to reduce overfitting.

Implemented EarlyStopping callback to stop training when validation loss stops improving.

Lowered learning rate to 0.001 for smoother convergence.

Results Interpretation

Before: Training accuracy was 95%, validation accuracy was 70%, showing overfitting.

After: Training accuracy dropped to 90%, validation accuracy improved to 87%, indicating better generalization.

Adding dropout and early stopping helps the model avoid memorizing training data and improves its ability to perform well on new, unseen text.

Bonus Experiment

Try using a simpler model architecture with fewer LSTM units or replace LSTM with a GRU layer to see if it further reduces overfitting.

💡 Hint

Simpler models have fewer parameters and are less likely to memorize training data, which can improve validation accuracy.

Practice

1. What is the main goal of entity linking in natural language processing?

easy

A. To connect words or phrases in text to real-world entities in a database

B. To translate text from one language to another

C. To summarize long documents into short sentences

D. To generate new text based on input prompts

Solution

1. Step 1: Understand entity linking purpose

   Entity linking matches text mentions to specific entities like people, places, or things in a knowledge base.

2. Step 2: Compare with other NLP tasks

   Unlike translation, summarization, or text generation, entity linking focuses on identifying and connecting entities.

3. Final Answer:

   To connect words or phrases in text to real-world entities in a database -> Option A

4. Quick Check:

   Entity linking = connecting text to entities [OK]

Hint: Entity linking = matching text to known entities [OK]

Common Mistakes:

• Confusing entity linking with translation
• Thinking entity linking summarizes text
• Mixing entity linking with text generation

2. Which of the following is the correct way to describe the output of an entity linking system?

easy

A. A mapping from text mentions to unique entity IDs

B. A list of translated sentences

C. A summary of the input text

D. A generated paragraph based on input keywords

Solution

1. Step 1: Identify entity linking output type

   Entity linking outputs pairs linking text mentions to unique IDs representing entities in a knowledge base.

2. Step 2: Eliminate unrelated outputs

   Translated sentences, summaries, or generated paragraphs are outputs of other NLP tasks, not entity linking.

3. Final Answer:

   A mapping from text mentions to unique entity IDs -> Option A

4. Quick Check:

   Entity linking output = mention to entity ID map [OK]

Hint: Entity linking output = mention linked to entity ID [OK]

Common Mistakes:

• Confusing output with translation or summarization
• Thinking output is raw text instead of mappings
• Ignoring the unique ID aspect of entities

3. Given the text: 'Apple released a new product.' and an entity linking system that links 'Apple' to the company entity, what would be the expected output?

medium

A. [('Apple', 'fruit_entity_id')]

B. [('Apple', 'unknown')]

C. [('Apple', 'company_entity_id')]

D. [('Apple', 'city_entity_id')]

Solution

1. Step 1: Analyze the context of 'Apple'

   In the sentence about releasing a product, 'Apple' refers to the company, not the fruit or city.

2. Step 2: Match mention to correct entity

   The entity linking system should link 'Apple' to the company entity ID.

3. Final Answer:

   [('Apple', 'company_entity_id')] -> Option C

4. Quick Check:

   Context guides entity linking to company [OK]

Hint: Use sentence context to pick correct entity [OK]

Common Mistakes:

• Linking 'Apple' to fruit without context
• Choosing unknown entity when context is clear
• Confusing city with company entity

4. Consider this entity linking output: [('Paris', 'city_entity_id'), ('Paris', 'person_entity_id')]. What is the likely problem here?

medium

A. The system generated new entities not in the text

B. The system translated 'Paris' incorrectly

C. The system summarized the text instead of linking

D. The system failed to disambiguate between entities with the same name

Solution

1. Step 1: Understand entity ambiguity

   'Paris' can refer to a city or a person; entity linking must choose the correct one based on context.

2. Step 2: Identify error type

   Output shows both entities linked, indicating failure to pick the right one (disambiguation error).

3. Final Answer:

   The system failed to disambiguate between entities with the same name -> Option D

4. Quick Check:

   Ambiguity causes multiple entity links [OK]

Hint: Check if system picks one correct entity per mention [OK]

Common Mistakes:

• Thinking it's a translation error
• Confusing linking with summarization
• Assuming system invented new entities

5. You have a sentence: 'Jordan scored 30 points.' The entity linking system links 'Jordan' to both a country and a basketball player entity. How can you improve the system to pick the correct entity?

hard

A. Always link to the most popular entity

B. Use the sentence context to disambiguate entities

C. Ignore ambiguous mentions to avoid errors

D. Randomly select one entity when ambiguous

Solution

1. Step 1: Identify the ambiguity problem

   'Jordan' can mean a country or a basketball player; system must decide based on context.

2. Step 2: Apply context-based disambiguation

   Using words like 'scored' and 'points' helps the system link to the basketball player, not the country.

3. Final Answer:

   Use the sentence context to disambiguate entities -> Option B

4. Quick Check:

   Context helps pick correct entity [OK]

Hint: Use nearby words to clarify entity meaning [OK]

Common Mistakes:

• Always picking the most popular entity blindly
• Skipping ambiguous mentions instead of resolving
• Randomly choosing entities without context