LLM scaling laws in Prompt Engineering / GenAI - Model Metrics & Evaluation

Metrics & Evaluation - LLM scaling laws

Which metric matters for LLM scaling laws and WHY

When studying how large language models (LLMs) improve as they get bigger, the key metric is loss, especially cross-entropy loss. This loss measures how well the model predicts the next word. Lower loss means better predictions.

We focus on loss because scaling laws show a smooth, predictable drop in loss as model size, data, and compute increase. This helps us understand how much bigger or longer to train a model to get better results.

Confusion matrix or equivalent visualization

LLM scaling laws don't use confusion matrices like classification tasks. Instead, we look at loss curves that show loss values on the y-axis and model size, data amount, or compute on the x-axis.

Model Size (billions) | Loss
---------------------|-------
0.1                  | 3.5
1                    | 2.8
10                   | 2.1
100                  | 1.6

This shows loss steadily decreasing as model size grows.
    

Precision vs Recall tradeoff (or equivalent) with concrete examples

LLM scaling laws focus on loss reduction, which balances many small prediction errors. Unlike classification, there is no direct precision or recall.

However, there is a tradeoff between model size and training data. Bigger models need more data to avoid overfitting. Too little data means the model memorizes instead of learning, causing poor generalization.

Example: A 10B parameter model trained on 1B tokens may overfit (high loss on new data). But trained on 100B tokens, it learns better and loss drops.

What "good" vs "bad" metric values look like for LLM scaling laws

Good: Loss decreases smoothly as model size and data increase. This means the model is learning well and scaling predictably.

Bad: Loss plateaus or increases when scaling up. This suggests the model is too big for the data or training time, causing overfitting or underfitting.

For example, a 100B parameter model with loss 1.6 is good if a 10B model has loss 2.1. But if the 100B model's loss is 2.5, that is bad and means scaling failed.

Metrics pitfalls

• Ignoring data quality: Scaling laws assume good, clean data. Poor data can hide true scaling benefits.
• Overfitting: Large models trained on too little data show low training loss but high loss on new data.
• Compute limits: Not training long enough or with enough compute can make loss look worse than it should.
• Misinterpreting loss: Loss is a proxy for quality but doesn't capture all aspects like creativity or factual accuracy.

Self-check question

Your 50B parameter LLM has a training loss of 1.5 but a validation loss of 3.0. Is this good for scaling? Why or why not?

Answer: This is not good. The large gap between training and validation loss means the model is overfitting. It memorizes training data but fails to generalize. For good scaling, training and validation loss should both decrease smoothly and stay close.

Key Result

LLM scaling laws focus on cross-entropy loss decreasing smoothly as model size and data increase, indicating better prediction quality.

Practice

1. What do LLM scaling laws primarily describe in language model training?

easy

A. The syntax rules for writing code in AI frameworks

B. How model size, data amount, and compute resources affect performance

C. The best way to label data for supervised learning

D. How to deploy models on mobile devices

Solution

1. Step 1: Understand the purpose of scaling laws

   LLM scaling laws explain the relationship between model size, data, and compute with model performance.

2. Step 2: Match the description to options

   Only How model size, data amount, and compute resources affect performance correctly describes this relationship, while others talk about unrelated topics.

3. Final Answer:

   How model size, data amount, and compute resources affect performance -> Option B

4. Quick Check:

   Scaling laws = model size, data, compute impact [OK]

Hint: Focus on model size, data, and compute impact keywords [OK]

Common Mistakes:

• Confusing scaling laws with coding syntax
• Thinking scaling laws are about data labeling
• Assuming scaling laws relate to deployment

2. Which of the following is the correct formula representing a simplified LLM scaling law for loss L as a function of model parameters N and dataset size D?

easy

A. L = a / (N + D)

B. L = a + b * N + c * D

C. L = a * log(N) + b * log(D)

D. L = a * N^(-b) + c * D^(-d)

Solution

1. Step 1: Recall the typical scaling law form

   Scaling laws often show loss decreases as power laws of model size and data, like L = a * N^(-b) + c * D^(-d).

2. Step 2: Compare options to this form

   L = a * N^(-b) + c * D^(-d) matches the power law form; others use linear or logarithmic forms which are incorrect.

3. Final Answer:

   L = a * N^(-b) + c * D^(-d) -> Option D

4. Quick Check:

   Loss decreases as power laws of N and D [OK]

Hint: Look for power law (exponent) form in the formula [OK]

Common Mistakes:

• Choosing linear formulas instead of power laws
• Confusing logarithmic with power law forms
• Ignoring the negative exponents for loss decrease

3. Consider this Python code simulating a simplified LLM loss calculation:

def loss(N, D, a=1.0, b=0.5, c=1.0, d=0.3):
    return a * N**(-b) + c * D**(-d)

print(round(loss(1000, 10000), 4))

What is the output?

medium

A. 0.0947

B. 0.1265

C. 0.0316

D. 1.0000

Solution

1. Step 1: Calculate each term separately

   N=1000, b=0.5: 1000**(-0.5) = 1/sqrt(1000) ≈ 0.0316
   D=10000, d=0.3: 10000**(-0.3) ≈ 0.0631

2. Step 2: Sum the terms and round to 4 decimals

   1.0 * 0.0316 + 1.0 * 0.0631 = 0.0947

3. Final Answer:

   0.0947 -> Option A

4. Quick Check:

   N**(-0.5) + D**(-0.3) ≈ 0.0316 + 0.0631 = 0.0947 [OK]

Hint: Calculate each power term separately, then sum [OK]

Common Mistakes:

• Calculating only one term instead of sum
• Mixing up exponents or signs
• Rounding too early causing errors

4. The following code aims to compute loss using LLM scaling laws but has a bug:

def loss(N, D, a=1.0, b=0.5, c=1.0, d=0.3):
    return a * N**b + c * D**d

print(round(loss(1000, 10000), 4))

What is the main error?

medium

A. Function should return a tuple, not a single value

B. Missing multiplication operator between variables

C. Exponents should be negative to show loss decreases with size

D. Parameters a and c should be integers only

Solution

1. Step 1: Identify the intended formula

   LLM scaling laws show loss decreases as model size and data increase, so exponents must be negative.

2. Step 2: Check the code exponents

   The code uses positive exponents (N**b and D**d), which incorrectly increase loss with size.

3. Final Answer:

   Exponents should be negative to show loss decreases with size -> Option C

4. Quick Check:

   Negative exponents mean loss decreases as size grows [OK]

Hint: Remember loss decreases, so exponents must be negative [OK]

Common Mistakes:

• Thinking multiplication is missing
• Believing return type must be tuple
• Assuming parameter types must be integers

5. You want to reduce the loss of a large language model efficiently. According to LLM scaling laws, which strategy is best if you have limited compute but can increase data or model size?

hard

A. Increase dataset size moderately while keeping model size fixed

B. Increase model size drastically without adding data

C. Keep both model size and data fixed and train longer

D. Reduce dataset size to speed up training

Solution

1. Step 1: Understand compute constraints and scaling laws

   Scaling laws show loss improves with both model size and data, but compute limits large model increases.

2. Step 2: Choose strategy fitting limited compute

   Increasing data moderately is cheaper than drastically increasing model size, so Increase dataset size moderately while keeping model size fixed is best.

3. Final Answer:

   Increase dataset size moderately while keeping model size fixed -> Option A

4. Quick Check:

   Limited compute favors data increase over big model growth [OK]

Hint: With limited compute, grow data before model size [OK]

Common Mistakes:

• Thinking bigger model always better regardless of compute
• Ignoring compute limits and training time
• Reducing data harms performance