Multilingual models in NLP - Model Metrics & Evaluation

Metrics & Evaluation - Multilingual models

Which metric matters for multilingual models and WHY

For multilingual models, accuracy and F1 score are key metrics. They show how well the model understands and predicts across different languages. Since languages vary, balanced performance is important. We want the model to do well on all languages, not just one. So, metrics like macro-averaged F1 (which treats each language equally) help us see if the model is fair and effective everywhere.

Confusion matrix example for a multilingual classification task

      Language: English, Spanish, French
      ---------------------------------
      |          | Pred Eng | Pred Spa | Pred Fre |
      |----------|----------|----------|----------|
      | True Eng |    45    |    3     |    2     |
      | True Spa |    4     |   40     |    6     |
      | True Fre |    1     |    5     |   44     |
      ---------------------------------
    

This matrix shows how often the model predicted each language correctly or confused it with another. For example, it predicted English correctly 45 times but confused Spanish as French 6 times.

Precision vs Recall tradeoff in multilingual models

Imagine a model that detects spam messages in multiple languages. If it has high precision, it means when it says a message is spam, it is usually right. This avoids annoying users by marking good messages as spam.

If it has high recall, it finds most spam messages, even if some good messages get marked wrongly. This is important to catch all spam.

For multilingual models, the tradeoff matters per language. Some languages might have less data, so recall might be lower there. We want to balance precision and recall so the model works well for all languages.

What good vs bad metric values look like for multilingual models

Good: Macro F1 scores above 0.8 across all languages show balanced and strong performance. Precision and recall are close, meaning the model is both accurate and finds most correct answers.

Bad: High accuracy overall but very low F1 or recall in some languages means the model ignores or fails those languages. For example, 95% accuracy but 0.3 F1 on a low-resource language is bad.

Common pitfalls in evaluating multilingual models

• Accuracy paradox: High overall accuracy can hide poor results on smaller languages.
• Data leakage: If training and test data overlap in any language, metrics look better than reality.
• Overfitting: Model may memorize frequent languages but fail on rare ones.
• Ignoring language imbalance: Not using macro-averaged metrics can bias evaluation toward dominant languages.

Self-check question

Your multilingual model has 98% accuracy overall but only 12% recall on a low-resource language. Is it good for production? Why or why not?

Answer: No, it is not good. The low recall means the model misses most correct cases in that language. Even if overall accuracy is high, the model fails users of that language. You should improve recall or balance performance before production.

Key Result

Balanced metrics like macro F1 score are key to ensure multilingual models perform well across all languages.

Practice

1. What is the main advantage of using a multilingual model in natural language processing?

easy

A. It can understand and process multiple languages with a single model.

B. It requires training a separate model for each language.

C. It only works for English language tasks.

D. It uses more resources than training individual models.

Solution

1. Step 1: Understand the purpose of multilingual models

   Multilingual models are designed to handle many languages using one model instead of separate ones.

2. Step 2: Compare advantages

   This approach saves time and resources by avoiding multiple models for different languages.

3. Final Answer:

   It can understand and process multiple languages with a single model. -> Option A

4. Quick Check:

   Multilingual model advantage = single model for many languages [OK]

Hint: Multilingual means one model for many languages [OK]

Common Mistakes:

• Thinking multilingual models only work for English
• Assuming separate models are needed per language
• Believing multilingual models use more resources

2. Which of the following is the correct way to load a multilingual model using Hugging Face Transformers in Python?

easy

A. model = AutoModel.from_pretrained('xlm-roberta-base')

B. model = AutoModel.from_pretrained('gpt2')

C. model = AutoModel.from_pretrained('bert-base-uncased')

D. model = AutoModel.from_pretrained('bert-large-cased')

Solution

1. Step 1: Identify multilingual model names

   'xlm-roberta-base' is a well-known multilingual model supporting many languages.

2. Step 2: Check other options

   'bert-base-uncased' and 'bert-large-cased' are English-only models; 'gpt2' is a generative English model.

3. Final Answer:

   model = AutoModel.from_pretrained('xlm-roberta-base') -> Option A

4. Quick Check:

   Multilingual model name = 'xlm-roberta-base' [OK]

Hint: Look for 'xlm' or 'multilingual' in model name [OK]

Common Mistakes:

• Choosing English-only models for multilingual tasks
• Confusing generative models with multilingual encoders
• Using model names without checking language support

3. Consider this Python code using Hugging Face Transformers:

from transformers import AutoTokenizer, AutoModelForSequenceClassification

tokenizer = AutoTokenizer.from_pretrained('xlm-roberta-base')
model = AutoModelForSequenceClassification.from_pretrained('xlm-roberta-base')
inputs = tokenizer('Bonjour, comment ça va?', return_tensors='pt')
outputs = model(**inputs)
print(outputs.logits.shape)

What will be the printed output shape?

medium

A. torch.Size([1, 1])

B. torch.Size([1, 2])

C. torch.Size([1, 512])

D. torch.Size([1, 768])

Solution

1. Step 1: Understand model type and output

   The model is for sequence classification, which outputs logits for each class. The default 'xlm-roberta-base' classification head has 2 classes.

2. Step 2: Determine output shape

   Batch size is 1 (one sentence), so output logits shape is [1, 2].

3. Final Answer:

   torch.Size([1, 2]) -> Option B

4. Quick Check:

   Sequence classification logits shape = [batch, classes] = [1, 2] [OK]

Hint: Classification logits shape = batch size x number of classes [OK]

Common Mistakes:

• Confusing hidden size with output logits shape
• Assuming output shape matches input token length
• Ignoring batch size dimension

4. You tried to use a multilingual model but got this error:
ValueError: Tokenizer does not have a pad token.
What is the best way to fix this error?

medium

A. Use a different model that does not require padding.

B. Add padding=True when calling the tokenizer.

C. Manually set the pad token with tokenizer.pad_token = tokenizer.eos_token.

D. Ignore the error and continue training.

Solution

1. Step 1: Understand the error cause

   The tokenizer lacks a pad token, which is needed to pad sequences to the same length.

2. Step 2: Fix by assigning pad token

   Assigning the pad token to an existing token like eos_token solves the issue.

3. Final Answer:

   Manually set the pad token with tokenizer.pad_token = tokenizer.eos_token. -> Option C

4. Quick Check:

   Set pad token manually to fix padding error [OK]

Hint: Set pad token manually if missing in tokenizer [OK]

Common Mistakes:

• Ignoring padding requirement
• Trying to skip padding without fixing tokenizer
• Switching models unnecessarily

5. You want to build a multilingual sentiment analysis system supporting English, Spanish, and French. Which approach best balances accuracy and resource use?

hard

A. Train separate models for each language from scratch.

B. Use a rule-based system with language-specific sentiment dictionaries.

C. Use an English-only model and translate all inputs to English before analysis.

D. Use a single pretrained multilingual model fine-tuned on combined data from all three languages.

Solution

1. Step 1: Consider resource and accuracy trade-offs

   Training separate models is resource-heavy; rule-based systems lack accuracy; translation adds errors.

2. Step 2: Choose multilingual fine-tuning

   Fine-tuning one multilingual pretrained model on combined data leverages shared knowledge and saves resources.

3. Final Answer:

   Use a single pretrained multilingual model fine-tuned on combined data from all three languages. -> Option D

4. Quick Check:

   Multilingual fine-tuning balances accuracy and efficiency [OK]

Hint: Fine-tune one multilingual model on all languages together [OK]

Common Mistakes:

• Training separate models wastes resources
• Relying on translation reduces accuracy
• Using rule-based methods limits performance