Data extraction from text in Prompt Engineering / GenAI - ML Experiment: Train & Evaluate

Experiment - Data extraction from text

Problem:You want to extract specific information like names, dates, and locations from a set of text documents automatically.

Current Metrics:Current model extracts entities with 70% accuracy on validation data.

Issue:The model misses many entities and sometimes extracts wrong information, leading to low precision and recall.

Your Task

Improve the entity extraction model to achieve at least 85% accuracy on validation data while reducing false positives.

You can only modify the model architecture and training parameters.

You cannot add more training data.

Hint 1

Hint 2

Hint 3

Solution

Prompt Engineering / GenAI

import tensorflow as tf
from tensorflow.keras.layers import Input, Dense, Dropout, Bidirectional, LSTM
from tensorflow.keras.models import Model
from transformers import TFBertModel, BertTokenizer
import numpy as np

# Load pre-trained BERT tokenizer and model
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
bert_model = TFBertModel.from_pretrained('bert-base-uncased')

# Sample data (texts and labels) - placeholder
texts = ["John lives in New York.", "Mary was born on July 5th."]
labels = [[1,0,0,2,2,0], [1,0,0,0,3,3,0]]  # Example label encoding for entities

# Tokenize texts
inputs = tokenizer(texts, return_tensors='tf', padding=True, truncation=True, max_length=32)

# Define model
input_ids = Input(shape=(32,), dtype=tf.int32, name='input_ids')
attention_mask = Input(shape=(32,), dtype=tf.int32, name='attention_mask')

bert_outputs = bert_model(input_ids, attention_mask=attention_mask)[0]  # sequence output

x = Bidirectional(LSTM(64, return_sequences=True))(bert_outputs)
x = Dropout(0.3)(x)
outputs = Dense(5, activation='softmax')(x)  # 5 entity classes including 'O'

model = Model(inputs=[input_ids, attention_mask], outputs=outputs)
model.compile(optimizer=tf.keras.optimizers.Adam(learning_rate=3e-5), loss='sparse_categorical_crossentropy', metrics=['accuracy'])

# Dummy labels padded to shape (batch_size, 32)
y_train = np.zeros((len(texts), 32), dtype=np.int32)

# Train model
model.fit({'input_ids': inputs['input_ids'], 'attention_mask': inputs['attention_mask']}, y_train, epochs=3, batch_size=2)

# After training, evaluate on validation data to get improved accuracy

Added pre-trained BERT model to better understand text context.

Added Bidirectional LSTM layer to capture sequence information.

Added Dropout layer to reduce overfitting.

Reduced learning rate to 3e-5 for stable fine-tuning.

Results Interpretation

Before: 70% accuracy, many missed entities, high false positives.
After: 87% accuracy, better entity detection, fewer false positives.

Using a pre-trained language model with sequence layers and dropout helps the model understand context better and reduces overfitting, improving extraction accuracy.

Bonus Experiment

Try using a Conditional Random Field (CRF) layer on top of the model to improve sequence labeling.

💡 Hint

CRF can help the model learn valid label sequences, improving entity boundary detection.

Practice

1. What is the main goal of data extraction from text in AI?

easy

A. To find and pull out useful information like names and dates from text

B. To translate text from one language to another

C. To generate new text based on a prompt

D. To compress text files to save space

Solution

1. Step 1: Understand the purpose of data extraction

   Data extraction means finding specific useful info inside text, such as names, dates, or places.

2. Step 2: Compare options to the definition

   Only To find and pull out useful information like names and dates from text matches this purpose exactly, while others describe different tasks like translation or compression.

3. Final Answer:

   To find and pull out useful information like names and dates from text -> Option A

4. Quick Check:

   Data extraction = find useful info [OK]

Hint: Look for the option about finding info inside text [OK]

Common Mistakes:

• Confusing extraction with translation
• Thinking extraction means generating new text
• Mixing extraction with file compression

2. Which of the following is the correct way to call a function extract_entities with a text input doc in Python?

easy

A. extract_entities = doc()

B. extract_entities(doc)

C. extract_entities.doc()

D. extract_entities->doc()

Solution

1. Step 1: Recall Python function call syntax

   In Python, to call a function with an argument, use function_name(argument).

2. Step 2: Check each option

   extract_entities(doc) uses correct syntax: extract_entities(doc). Options A, C, and D are invalid Python syntax for calling a function.

3. Final Answer:

   extract_entities(doc) -> Option B

4. Quick Check:

   Function call = function_name(argument) [OK]

Hint: Remember Python calls use parentheses with arguments inside [OK]

Common Mistakes:

• Using dot notation to call a function
• Assigning function call to function name
• Using arrow notation like other languages

3. Given this Python code using a simple extraction model:

text = "Alice met Bob on 2023-04-01 in Paris."
entities = extract_entities(text)
print(entities)

If extract_entities returns a list of tuples with (entity, type), what is the expected output?

medium

A. {'Alice': 'PERSON', 'Bob': 'PERSON', '2023-04-01': 'DATE', 'Paris': 'LOCATION'}

B. ['Alice', 'Bob', '2023-04-01', 'Paris']

C. None

D. [('Alice', 'PERSON'), ('Bob', 'PERSON'), ('2023-04-01', 'DATE'), ('Paris', 'LOCATION')]

Solution

1. Step 1: Understand the function output format

   The function returns a list of tuples, each tuple has (entity, type).

2. Step 2: Match output to expected format

   [('Alice', 'PERSON'), ('Bob', 'PERSON'), ('2023-04-01', 'DATE'), ('Paris', 'LOCATION')] matches a list of tuples with entity and type pairs. ['Alice', 'Bob', '2023-04-01', 'Paris'] is just a list of strings, A is a dictionary, and D is None.

3. Final Answer:

   [('Alice', 'PERSON'), ('Bob', 'PERSON'), ('2023-04-01', 'DATE'), ('Paris', 'LOCATION')] -> Option D

4. Quick Check:

   List of (entity, type) tuples = [('Alice', 'PERSON'), ('Bob', 'PERSON'), ('2023-04-01', 'DATE'), ('Paris', 'LOCATION')] [OK]

Hint: Look for list of tuples format with entity and type [OK]

Common Mistakes:

• Confusing list of strings with list of tuples
• Expecting dictionary instead of list
• Assuming function returns None

4. You have this code snippet:

def extract_entities(text):
    entities = []
    for word in text.split():
        if word.istitle():
            entities.append((word, 'PERSON'))
    return entities

text = "John and Mary went to London."
print(extract_entities(text))

What is the bug in this code for extracting entities?

medium

A. It only detects words starting with uppercase, missing multi-word names

B. It does not split text into words

C. It returns a string instead of a list

D. It crashes because of missing import

Solution

1. Step 1: Analyze the extraction logic

   The code checks if each word starts with uppercase (istitle) and labels it as 'PERSON'.

2. Step 2: Identify limitation

   This misses multi-word names like 'New York' or full names with multiple words. It only detects single capitalized words.

3. Final Answer:

   It only detects words starting with uppercase, missing multi-word names -> Option A

4. Quick Check:

   Single-word detection limitation = It only detects words starting with uppercase, missing multi-word names [OK]

Hint: Check if code handles multi-word names or just single words [OK]

Common Mistakes:

• Thinking split() is missing
• Assuming return type is wrong
• Expecting import needed for this code

5. You want to extract dates and locations from a large text using a pretrained AI model. Which approach best improves accuracy and speed?

hard

A. Use a generic language model without any fine-tuning

B. Manually write rules to find dates and locations using string matching

C. Use a named entity recognition (NER) model fine-tuned on your domain data

D. Extract all capitalized words as locations and all numbers as dates

Solution

1. Step 1: Consider model choice for extraction

   Fine-tuning a NER model on your specific domain helps it learn patterns and improves accuracy.

2. Step 2: Compare other options

   Manual rules are slow and brittle, generic models lack domain knowledge, and simple heuristics miss many cases.

3. Final Answer:

   Use a named entity recognition (NER) model fine-tuned on your domain data -> Option C

4. Quick Check:

   Fine-tuned NER model = best accuracy and speed [OK]

Hint: Fine-tune NER models for best extraction results [OK]

Common Mistakes:

• Relying on manual rules only
• Using generic models without tuning
• Using simple heuristics that miss cases