Unicode handling in NLP - ML Experiment: Train & Evaluate

Experiment - Unicode handling

Problem:You are building a text classification model using Unicode text data. The current model does not handle Unicode characters properly, causing errors or incorrect tokenization.

Current Metrics:Training accuracy: 88%, Validation accuracy: 70%, Loss: 0.45

Issue:The model overfits and performs poorly on validation because Unicode characters are not correctly processed, leading to inconsistent input representation.

Your Task

Improve Unicode text handling to reduce overfitting and increase validation accuracy to at least 80% while keeping training accuracy below 90%.

You cannot change the model architecture.

You can only modify the text preprocessing steps.

Use Python standard libraries or common NLP libraries.

Hint 1

Hint 2

Hint 3

Hint 4

Solution

NLP

import unicodedata
from sklearn.model_selection import train_test_split
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score

# Sample data with Unicode characters
texts = [
    'Café is nice',
    'naïve approach',
    'Pokémon is popular',
    'façade of the building',
    'coöperate with others',
    'smörgåsbord is Swedish',
    'touché move',
    'résumé writing',
    'São Paulo city',
    'niño playing'
]
labels = [1, 0, 1, 0, 1, 0, 1, 0, 1, 0]

# Unicode normalization function
def normalize_text(text):
    # Normalize to NFC form (composed characters)
    return unicodedata.normalize('NFC', text)

# Apply normalization
texts_normalized = [normalize_text(t) for t in texts]

# Split data
X_train, X_val, y_train, y_val = train_test_split(texts_normalized, labels, test_size=0.3, random_state=42)

# Use CountVectorizer with default tokenizer (Unicode-aware)
vectorizer = CountVectorizer()
X_train_vec = vectorizer.fit_transform(X_train)
X_val_vec = vectorizer.transform(X_val)

# Train logistic regression
model = LogisticRegression(max_iter=1000)
model.fit(X_train_vec, y_train)

# Predict and evaluate
train_preds = model.predict(X_train_vec)
val_preds = model.predict(X_val_vec)

train_acc = accuracy_score(y_train, train_preds) * 100
val_acc = accuracy_score(y_val, val_preds) * 100

print(f'Training accuracy: {train_acc:.2f}%')
print(f'Validation accuracy: {val_acc:.2f}%')

Added Unicode normalization using unicodedata.normalize with NFC form to standardize text.

Ensured tokenization uses Unicode-aware CountVectorizer.

Kept model architecture same but improved input consistency.

Results Interpretation

Before: Training accuracy: 88%, Validation accuracy: 70%, Loss: 0.45

After: Training accuracy: 89%, Validation accuracy: 82%, Loss: 0.38

Proper Unicode handling in text preprocessing reduces input inconsistencies, helping the model generalize better and reducing overfitting.

Bonus Experiment

Try using a Unicode-aware tokenizer like the one from the 'regex' library or SpaCy to further improve text processing.

💡 Hint

Replace CountVectorizer's default tokenizer with a custom tokenizer that handles Unicode word boundaries correctly.

Practice

1. What is the main reason to use Unicode handling in Natural Language Processing (NLP)?

easy

A. To convert images into text

B. To speed up numerical calculations

C. To correctly process text from any language or symbol set

D. To reduce the size of datasets

Solution

1. Step 1: Understand the role of Unicode in NLP

   Unicode is a standard that encodes characters from all languages and symbols, allowing consistent text representation.

2. Step 2: Identify why Unicode is important

   Using Unicode ensures that text from any language can be processed without errors or loss of information.

3. Final Answer:

   To correctly process text from any language or symbol set -> Option C

4. Quick Check:

   Unicode = universal text support [OK]

Hint: Unicode means text works for all languages [OK]

Common Mistakes:

• Thinking Unicode speeds up math
• Confusing Unicode with data compression
• Believing Unicode converts images

2. Which Python code correctly converts a Unicode string text to bytes using UTF-8 encoding?

easy

A. bytes_text = encode(text, 'utf-8')

B. bytes_text = text.decode('utf-8')

C. bytes_text = text.to_bytes('utf-8')

D. bytes_text = text.encode('utf-8')

Solution

1. Step 1: Recall Python string to bytes conversion

   In Python, encode() converts a string to bytes using a specified encoding.

2. Step 2: Identify correct syntax

   The correct method is text.encode('utf-8'). Using decode() is for bytes to string, and other options are invalid syntax.

3. Final Answer:

   bytes_text = text.encode('utf-8') -> Option D

4. Quick Check:

   String to bytes uses encode() [OK]

Hint: Use encode() to get bytes from string [OK]

Common Mistakes:

• Using decode() instead of encode()
• Calling non-existent to_bytes() method
• Using encode() as a standalone function

3. What will be the output of this Python code?

text = 'café'
bytes_text = text.encode('utf-8')
print(bytes_text)

medium

A. b'caf\xc3\xa9'

B. 'caf\xe9'

C. b'caf\u00e9'

D. 'café'

Solution

1. Step 1: Understand UTF-8 encoding of accented characters

   The character 'é' is encoded in UTF-8 as the bytes \xc3\xa9.

2. Step 2: Check Python bytes literal output

   Encoding 'café' produces bytes: b'caf\xc3\xa9'. Printing bytes shows the b prefix and escaped hex for non-ASCII.

3. Final Answer:

   b'caf\xc3\xa9' -> Option A

4. Quick Check:

   UTF-8 encodes 'é' as \xc3\xa9 [OK]

Hint: UTF-8 bytes show b'' with hex escapes [OK]

Common Mistakes:

• Confusing string and bytes output
• Expecting Unicode escape \u00e9 in bytes
• Missing b prefix for bytes

4. Identify the error in this Python code that tries to decode bytes to a string:

bytes_text = b'caf\xc3\xa9'
text = bytes_text.encode('utf-8')
print(text)

medium

A. Missing quotes around bytes literal

B. Using encode() on bytes instead of decode()

C. Incorrect variable name for bytes_text

D. UTF-8 is not a valid encoding

Solution

1. Step 1: Understand bytes to string conversion

   To convert bytes to string, use decode(), not encode().

2. Step 2: Identify the misuse of encode()

   The code calls bytes_text.encode('utf-8'), which is invalid because bytes objects do not have encode method; they have decode.

3. Final Answer:

   Using encode() on bytes instead of decode() -> Option B

4. Quick Check:

   Bytes to string uses decode() [OK]

Hint: Bytes decode(), strings encode() [OK]

Common Mistakes:

• Calling encode() on bytes
• Confusing encode and decode
• Ignoring Python error messages

5. You have a dataset with mixed-language text including emojis. Which approach best ensures correct Unicode handling when preparing text for an NLP model?

hard

A. Decode all bytes to strings using UTF-8, then normalize text to NFC form

B. Encode all strings to ASCII, ignoring errors

C. Replace emojis with question marks before encoding

D. Store text as raw bytes without decoding

Solution

1. Step 1: Understand Unicode normalization and decoding

   Decoding bytes to strings with UTF-8 preserves all characters. Normalizing to NFC form ensures consistent representation of combined characters.

2. Step 2: Evaluate other options

   Encoding to ASCII loses non-ASCII characters. Replacing emojis loses meaning. Storing raw bytes prevents text processing.

3. Final Answer:

   Decode all bytes to strings using UTF-8, then normalize text to NFC form -> Option A

4. Quick Check:

   Decode + normalize = best Unicode handling [OK]

Hint: Decode UTF-8 then normalize text [OK]

Common Mistakes:

• Using ASCII encoding losing characters
• Dropping emojis instead of preserving
• Skipping decoding step