Self-hosted LLMs (Llama, Mistral) in Prompt Engineering / GenAI - Model Metrics & Evaluation

Metrics & Evaluation - Self-hosted LLMs (Llama, Mistral)

Which metric matters for Self-hosted LLMs and WHY

For self-hosted large language models like Llama and Mistral, key metrics include perplexity and accuracy on downstream tasks. Perplexity measures how well the model predicts text, showing its understanding of language patterns. Accuracy on tasks like question answering or summarization shows real-world usefulness. These metrics matter because they tell us if the model generates sensible, relevant text and performs well on specific jobs.

Confusion matrix or equivalent visualization

For language models, a confusion matrix is less common. Instead, we use perplexity and task-specific accuracy. For example, on a classification task, a confusion matrix might look like this:

      | Predicted Positive | Predicted Negative |
      |--------------------|--------------------|
      | True Positive (TP)  | False Negative (FN) |
      | False Positive (FP) | True Negative (TN)  |
    

From this, we calculate precision, recall, and F1 score to understand model errors.

Precision vs Recall tradeoff with concrete examples

When using self-hosted LLMs for tasks like spam detection or content moderation, precision and recall tradeoffs matter:

• High Precision: The model rarely marks good content as spam. Useful when false alarms are costly.
• High Recall: The model catches most spam, even if some good content is flagged. Important when missing spam is risky.

Choosing which to prioritize depends on the use case. For example, in medical text analysis, high recall is critical to catch all important info.

What "good" vs "bad" metric values look like for self-hosted LLMs

Good metrics:

• Low perplexity (e.g., below 20) indicating strong language understanding.
• High accuracy (above 85%) on specific tasks like classification or summarization.
• Balanced precision and recall (both above 80%) for classification tasks.

Bad metrics:

• High perplexity (above 50), meaning the model struggles to predict text.
• Low accuracy (below 60%) on tasks, showing poor performance.
• Very low recall (below 50%) causing missed important cases.

Common pitfalls in metrics for self-hosted LLMs

• Accuracy paradox: High accuracy can be misleading if data is imbalanced (e.g., many easy examples).
• Data leakage: Using test data during training inflates metrics falsely.
• Overfitting: Model performs well on training but poorly on new data, hiding true performance.
• Ignoring task-specific metrics: Using only perplexity without checking real task results can miss issues.

Self-check question

Your self-hosted LLM has 98% accuracy on a classification task but only 12% recall on the important class. Is it good for production? Why or why not?

Answer: No, it is not good. The low recall means the model misses most important cases, which can be critical depending on the task. High accuracy alone is misleading if the model ignores the key class.

Key Result

For self-hosted LLMs, low perplexity and balanced precision-recall on tasks show strong, reliable performance.

Practice

1. What is the main advantage of using self-hosted LLMs like Llama or Mistral?

easy

A. You keep full control and privacy over your data

B. They always run faster than cloud models

C. They require no installation or setup

D. They provide unlimited free internet access

Solution

1. Step 1: Understand self-hosted LLMs purpose

   Self-hosted LLMs run on your own machines, so your data stays private and under your control.

2. Step 2: Compare options

   Cloud models may send data externally; self-hosted models avoid this, ensuring privacy.

3. Final Answer:

   You keep full control and privacy over your data -> Option A

4. Quick Check:

   Privacy and control = B [OK]

Hint: Self-hosted means data stays with you, so privacy is key [OK]

Common Mistakes:

• Thinking self-hosted models are always faster
• Assuming no setup is needed
• Confusing self-hosted with cloud services

2. Which Python code snippet correctly loads a Llama model using the Hugging Face Transformers library?

easy

A. from transformers import LlamaForCausalLM; model = LlamaForCausalLM.from_pretrained('llama-model')

B. import llama; model = llama.load('llama-model')

C. from transformers import MistralModel; model = MistralModel.load('llama-model')

D. model = load_model('llama-model')

Solution

1. Step 1: Identify correct library and class

   The Hugging Face Transformers library uses LlamaForCausalLM to load Llama models.

2. Step 2: Check method to load model

   from_pretrained is the standard method to load pretrained models in Transformers.

3. Final Answer:

   from transformers import LlamaForCausalLM; model = LlamaForCausalLM.from_pretrained('llama-model') -> Option A

4. Quick Check:

   Transformers + from_pretrained = C [OK]

Hint: Use Transformers library and from_pretrained to load models [OK]

Common Mistakes:

• Using wrong import names
• Calling non-existent load methods
• Confusing Mistral and Llama classes

3. Given this code snippet using a Mistral model, what will be the output type of output?

from transformers import MistralForCausalLM, MistralTokenizer
model = MistralForCausalLM.from_pretrained('mistral-base')
tokenizer = MistralTokenizer.from_pretrained('mistral-base')
inputs = tokenizer('Hello world', return_tensors='pt')
outputs = model.generate(**inputs)
output = tokenizer.decode(outputs[0])

medium

A. An error because generate is not defined

B. A tensor of token IDs

C. A list of token probabilities

D. A decoded string of generated text

Solution

1. Step 1: Understand model.generate output

   model.generate returns token IDs as tensors representing generated text tokens.

2. Step 2: Decode tokens to string

   tokenizer.decode converts token IDs to a readable string.

3. Final Answer:

   A decoded string of generated text -> Option D

4. Quick Check:

   generate + decode = string output [OK]

Hint: generate returns tokens; decode converts tokens to string [OK]

Common Mistakes:

• Thinking output is raw tensor
• Confusing probabilities with tokens
• Assuming generate method is missing

4. You try to load a Llama model with this code but get an error:

from transformers import LlamaForCausalLM
model = LlamaForCausalLM.load('llama-model')

What is the likely cause of the error?

medium

A. LlamaForCausalLM cannot be imported from transformers

B. The model name 'llama-model' is invalid

C. The method load() does not exist; should use from_pretrained()

D. You need to install the Mistral library first

Solution

1. Step 1: Check method names in Transformers

   Transformers models use from_pretrained() to load models, not load().

2. Step 2: Identify error cause

   Using load() causes AttributeError because it is not defined for LlamaForCausalLM.

3. Final Answer:

   The method load() does not exist; should use from_pretrained() -> Option C

4. Quick Check:

   Use from_pretrained, not load [OK]

Hint: Use from_pretrained() to load models, not load() [OK]

Common Mistakes:

• Assuming load() is valid method
• Blaming model name without checking method
• Confusing Llama and Mistral imports

5. You want to run a self-hosted Llama model on your local machine but it has limited RAM. Which approach helps reduce memory usage while keeping reasonable performance?

hard

A. Use a cloud service instead of local hosting

B. Use quantization to reduce model size and load with 8-bit precision

C. Run the model on CPU without any batching

D. Load the full 32-bit model without any optimization

Solution

1. Step 1: Understand memory constraints

   Limited RAM means loading full 32-bit models is heavy and slow.

2. Step 2: Apply quantization

   Quantization reduces model size by using lower precision (e.g., 8-bit), saving memory and keeping decent speed.

3. Step 3: Evaluate other options

   Loading full model wastes memory; CPU without batching is slow; cloud is not self-hosted.

4. Final Answer:

   Use quantization to reduce model size and load with 8-bit precision -> Option B

5. Quick Check:

   Quantization saves memory and keeps performance [OK]

Hint: Quantize models to 8-bit for less RAM use [OK]

Common Mistakes:

• Loading full 32-bit model ignoring RAM limits
• Running without batching causing slow speed
• Switching to cloud defeats self-hosting purpose