Multilingual sentiment in NLP - Model Metrics & Evaluation

Metrics & Evaluation - Multilingual sentiment

Which metric matters for Multilingual Sentiment and WHY

For multilingual sentiment analysis, accuracy, precision, recall, and F1 score are important. Accuracy shows overall correct predictions. Precision tells us how many predicted positive sentiments are truly positive. Recall shows how many actual positive sentiments were found. F1 balances precision and recall, which is key because missing or wrongly labeling sentiment can confuse users. Since languages differ, these metrics help check if the model works well across all languages.

Confusion Matrix Example

    Actual \ Predicted | Positive | Negative | Neutral
    -----------------------------------------------
    Positive           |   80     |   10     |  10
    Negative           |   15     |   70     |  15
    Neutral            |   5      |   10     |  85
    -----------------------------------------------
    Total samples = 300
    

From this matrix, for the Positive class:
Precision = 80 / (80 + 15 + 5) = 80 / 100 = 0.8
Recall = 80 / (80 + 10 + 10) = 80 / 100 = 0.8
F1 = 2 * (0.8 * 0.8) / (0.8 + 0.8) = 0.8

Precision vs Recall Tradeoff with Examples

In multilingual sentiment, if the model has high precision but low recall, it means it rarely mislabels sentiment but misses many true sentiments. For example, it might only detect very clear positive reviews but miss subtle ones in some languages.

If recall is high but precision is low, the model finds most positive sentiments but also wrongly labels many neutral or negative as positive, confusing users.

Balancing precision and recall (using F1 score) is important to give reliable sentiment results across languages.

Good vs Bad Metric Values for Multilingual Sentiment

Good: Accuracy above 80%, Precision and Recall above 75%, and F1 score near 0.8 or higher across all languages. This means the model correctly understands sentiment in different languages well.

Bad: Accuracy below 60%, Precision or Recall below 50%, or large differences in metrics between languages. This shows the model struggles with some languages or confuses sentiments.

Common Metrics Pitfalls

• Accuracy paradox: High accuracy can be misleading if one sentiment class dominates (e.g., mostly neutral reviews).
• Data leakage: If training data leaks language-specific hints, the model may seem better but fail on new languages.
• Overfitting: Very high training metrics but low test metrics means the model memorizes language patterns instead of generalizing.
• Ignoring class imbalance: Some sentiments or languages may have fewer samples, skewing metrics.

Self Check

Your multilingual sentiment model has 98% accuracy but only 12% recall on positive sentiment in Spanish. Is it good for production?

Answer: No. Despite high accuracy, the very low recall means the model misses most positive sentiments in Spanish. Users will get poor sentiment detection in that language, so the model needs improvement before production.

Key Result

F1 score balancing precision and recall is key to reliable multilingual sentiment detection.

Practice

1. What is the main advantage of using a multilingual sentiment analysis model?

easy

A. It can analyze sentiment in multiple languages with one model.

B. It only works for English text.

C. It requires training a new model for each language.

D. It ignores the language and treats all text the same.

Solution

1. Step 1: Understand multilingual sentiment models

   These models are designed to handle text in many languages without needing separate models for each.

2. Step 2: Compare options

   It can analyze sentiment in multiple languages with one model. correctly states the advantage. Options B, C, and D are incorrect because they limit the model to one language or misunderstand its function.

3. Final Answer:

   It can analyze sentiment in multiple languages with one model. -> Option A

4. Quick Check:

   Multilingual model = multiple languages [OK]

Hint: Multilingual means many languages, not just one [OK]

Common Mistakes:

• Thinking it only works for English
• Believing you need separate models per language
• Assuming language is ignored

2. Which of the following is the correct way to load a pretrained multilingual sentiment model using Hugging Face Transformers in Python?

easy

A. model = AutoModelForSequenceClassification.from_pretrained('nlptown/bert-base-multilingual-uncased-sentiment')

B. model = AutoTokenizer.from_pretrained('nlptown/bert-base-multilingual-uncased-sentiment')

C. model = AutoConfig.from_pretrained('nlptown/bert-base-multilingual-uncased-sentiment')

D. model = AutoModel.from_pretrained('nlptown/bert-base-multilingual-uncased-sentiment')

Solution

1. Step 1: Identify the correct class for sentiment classification

   For sentiment tasks, use AutoModelForSequenceClassification to load the model with classification head.

2. Step 2: Review options

   model = AutoModelForSequenceClassification.from_pretrained('nlptown/bert-base-multilingual-uncased-sentiment') uses AutoModelForSequenceClassification correctly. model = AutoModel.from_pretrained('nlptown/bert-base-multilingual-uncased-sentiment') loads a base model without classification head. model = AutoTokenizer.from_pretrained('nlptown/bert-base-multilingual-uncased-sentiment') loads tokenizer, not model. model = AutoConfig.from_pretrained('nlptown/bert-base-multilingual-uncased-sentiment') loads config only.

3. Final Answer:

   model = AutoModelForSequenceClassification.from_pretrained('nlptown/bert-base-multilingual-uncased-sentiment') -> Option A

4. Quick Check:

   SequenceClassification = sentiment model [OK]

Hint: Use AutoModelForSequenceClassification for sentiment tasks [OK]

Common Mistakes:

• Using AutoModel without classification head
• Confusing tokenizer with model
• Loading only config without weights

3. Given the following Python code snippet using the 'nlptown/bert-base-multilingual-uncased-sentiment' model, what will be the output sentiment label for the input text "Je suis très content" (French for "I am very happy")?

from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch

tokenizer = AutoTokenizer.from_pretrained('nlptown/bert-base-multilingual-uncased-sentiment')
model = AutoModelForSequenceClassification.from_pretrained('nlptown/bert-base-multilingual-uncased-sentiment')

inputs = tokenizer("Je suis très content", return_tensors="pt")
outputs = model(**inputs)
probs = torch.nn.functional.softmax(outputs.logits, dim=1)
label = torch.argmax(probs).item() + 1  # labels 1 to 5
print(label)

medium

A. 1 (Very Negative)

B. 5 (Very Positive)

C. 3 (Neutral)

D. 2 (Negative)

Solution

1. Step 1: Understand the input sentiment

   The French sentence "Je suis très content" means "I am very happy", which is a positive sentiment.

2. Step 2: Interpret model output labels

   The model outputs labels from 1 (very negative) to 5 (very positive). Since the sentence is very positive, the highest probability label should be 5.

3. Final Answer:

   5 (Very Positive) -> Option B

4. Quick Check:

   Positive sentence = label 5 [OK]

Hint: Happy words usually map to highest positive label [OK]

Common Mistakes:

• Confusing label numbers with sentiment polarity
• Ignoring language and assuming English only
• Not adding 1 to zero-based index

4. You run this code to analyze sentiment but get an error:

from transformers import AutoTokenizer, AutoModelForSequenceClassification

model = AutoModelForSequenceClassification.from_pretrained('nlptown/bert-base-multilingual-uncased-sentiment')
tokenizer = AutoTokenizer.from_pretrained('nlptown/bert-base-multilingual-uncased-sentiment')

inputs = tokenizer('Das ist schlecht', return_tensors='pt')
outputs = model(inputs)

What is the cause of the error?

medium

A. Missing import for torch library.

B. Tokenizer is loaded after the model, causing mismatch.

C. The input text is in German, which the model cannot process.

D. Model expects keyword arguments, but inputs passed as positional argument.

Solution

1. Step 1: Check how model is called

   The model expects inputs as keyword arguments like model(**inputs), but here inputs are passed as a single positional argument.

2. Step 2: Analyze other options

   Tokenizer order does not cause error. The model supports German. Missing torch import would cause a different error.

3. Final Answer:

   Model expects keyword arguments, but inputs passed as positional argument. -> Option D

4. Quick Check:

   Use model(**inputs) not model(inputs) [OK]

Hint: Pass inputs with ** to model call [OK]

Common Mistakes:

• Passing inputs without unpacking as keyword args
• Blaming language support incorrectly
• Ignoring error message details

5. You want to build a multilingual sentiment analysis app that supports English, Spanish, and Chinese. Which approach best balances accuracy and simplicity?

hard

A. Train separate sentiment models for each language from scratch.

B. Translate all texts to English and use an English-only sentiment model.

C. Use a pretrained multilingual sentiment model like 'nlptown/bert-base-multilingual-uncased-sentiment'.

D. Use a simple keyword-based sentiment dictionary for each language.

Solution

1. Step 1: Evaluate training effort and coverage

   Training separate models is costly and complex. Keyword-based methods lack accuracy. Translating text adds errors and latency.

2. Step 2: Consider pretrained multilingual models

   Pretrained multilingual models support many languages with good accuracy and easy setup, balancing simplicity and performance.

3. Final Answer:

   Use a pretrained multilingual sentiment model like 'nlptown/bert-base-multilingual-uncased-sentiment'. -> Option C

4. Quick Check:

   Pretrained multilingual = best balance [OK]

Hint: Pretrained multilingual models save time and support many languages [OK]

Common Mistakes:

• Assuming training separate models is easier
• Ignoring translation errors
• Overestimating keyword-based method accuracy