LLM scaling laws in Prompt Engineering / GenAI - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

Understanding the relationship between model size and performance

Which statement best describes the general trend observed in LLM scaling laws regarding model size and performance?

A Performance improves as a power-law function of the number of parameters in the model.

B Performance improves logarithmically with the number of parameters in the model.

C Performance improves linearly with the number of parameters in the model.

D Performance remains constant regardless of the number of parameters.

💡 Hint

Think about how small increases in size can lead to significant improvements, but not in a simple linear way.

LLM scaling laws show that performance improves following a power-law relationship with model size, meaning gains are significant but not linear or logarithmic.

❓ Metrics

intermediate

Evaluating loss behavior with increased compute

According to LLM scaling laws, how does the training loss typically change as the amount of compute used for training increases?

A Training loss remains unchanged regardless of compute.

B Training loss decreases following a power-law with increased compute.

C Training loss decreases exponentially with increased compute.

D Training loss increases as compute increases.

💡 Hint

Consider how more compute allows better fitting but with diminishing returns.

Training loss decreases following a power-law as compute increases, showing diminishing returns but consistent improvement.

❓ Model Choice

advanced

Choosing model size for fixed compute budget

Given a fixed compute budget, which strategy aligns best with LLM scaling laws to minimize training loss?

A Train multiple small models independently and average their outputs.

B Train a smaller model for more training steps.

C Train a very large model with fewer training steps.

D Balance model size and training steps to optimize compute usage.

💡 Hint

Think about how compute is split between model size and training duration.

LLM scaling laws suggest an optimal balance between model size and training steps to best use compute and minimize loss.

🔧 Debug

advanced

Identifying incorrect interpretation of scaling laws

Which of the following interpretations of LLM scaling laws is incorrect?

A Loss decreases smoothly as model size and compute increase.

B Compute-efficient training requires balancing model size and data.

C Doubling model parameters always halves the training loss.

D Increasing dataset size improves performance up to a point.

💡 Hint

Consider if the relationship between parameters and loss is linear or not.

Doubling parameters does not halve loss; improvements follow a power-law with diminishing returns, so this interpretation is incorrect.

❓ Predict Output

expert

Predicting training loss from scaling law formula

Given the scaling law formula for training loss:
loss = a * (N)^-b + c where N is the number of parameters, a=10, b=1/3, and c=0.1. What is the training loss when N=1000000?

Prompt Engineering / GenAI

a = 10
b = 1/3
c = 0.1
N = 1000000
loss = a * (N)**(-b) + c
print(round(loss, 4))

A 0.2

B 0.4

C 0.3

D 0.5

💡 Hint

Calculate N to the power of -b first, then multiply by a and add c.

1000000 = 10^6, so (10^6)^{-1/3} = 10^{-6/3} = 10^{-2} = 0.01 exactly. Then 10 * 0.01 = 0.1, plus c = 0.1 gives loss = 0.2 exactly. round(0.2, 4) outputs 0.2000, corresponding to 0.2.

Practice

1. What do LLM scaling laws primarily describe in language model training?

easy

A. The syntax rules for writing code in AI frameworks

B. How model size, data amount, and compute resources affect performance

C. The best way to label data for supervised learning

D. How to deploy models on mobile devices

Solution

1. Step 1: Understand the purpose of scaling laws

   LLM scaling laws explain the relationship between model size, data, and compute with model performance.

2. Step 2: Match the description to options

   Only How model size, data amount, and compute resources affect performance correctly describes this relationship, while others talk about unrelated topics.

3. Final Answer:

   How model size, data amount, and compute resources affect performance -> Option B

4. Quick Check:

   Scaling laws = model size, data, compute impact [OK]

Hint: Focus on model size, data, and compute impact keywords [OK]

Common Mistakes:

• Confusing scaling laws with coding syntax
• Thinking scaling laws are about data labeling
• Assuming scaling laws relate to deployment

2. Which of the following is the correct formula representing a simplified LLM scaling law for loss L as a function of model parameters N and dataset size D?

easy

A. L = a / (N + D)

B. L = a + b * N + c * D

C. L = a * log(N) + b * log(D)

D. L = a * N^(-b) + c * D^(-d)

Solution

1. Step 1: Recall the typical scaling law form

   Scaling laws often show loss decreases as power laws of model size and data, like L = a * N^(-b) + c * D^(-d).

2. Step 2: Compare options to this form

   L = a * N^(-b) + c * D^(-d) matches the power law form; others use linear or logarithmic forms which are incorrect.

3. Final Answer:

   L = a * N^(-b) + c * D^(-d) -> Option D

4. Quick Check:

   Loss decreases as power laws of N and D [OK]

Hint: Look for power law (exponent) form in the formula [OK]

Common Mistakes:

• Choosing linear formulas instead of power laws
• Confusing logarithmic with power law forms
• Ignoring the negative exponents for loss decrease

3. Consider this Python code simulating a simplified LLM loss calculation:

def loss(N, D, a=1.0, b=0.5, c=1.0, d=0.3):
    return a * N**(-b) + c * D**(-d)

print(round(loss(1000, 10000), 4))

What is the output?

medium

A. 0.0947

B. 0.1265

C. 0.0316

D. 1.0000

Solution

1. Step 1: Calculate each term separately

   N=1000, b=0.5: 1000**(-0.5) = 1/sqrt(1000) ≈ 0.0316
   D=10000, d=0.3: 10000**(-0.3) ≈ 0.0631

2. Step 2: Sum the terms and round to 4 decimals

   1.0 * 0.0316 + 1.0 * 0.0631 = 0.0947

3. Final Answer:

   0.0947 -> Option A

4. Quick Check:

   N**(-0.5) + D**(-0.3) ≈ 0.0316 + 0.0631 = 0.0947 [OK]

Hint: Calculate each power term separately, then sum [OK]

Common Mistakes:

• Calculating only one term instead of sum
• Mixing up exponents or signs
• Rounding too early causing errors

4. The following code aims to compute loss using LLM scaling laws but has a bug:

def loss(N, D, a=1.0, b=0.5, c=1.0, d=0.3):
    return a * N**b + c * D**d

print(round(loss(1000, 10000), 4))

What is the main error?

medium

A. Function should return a tuple, not a single value

B. Missing multiplication operator between variables

C. Exponents should be negative to show loss decreases with size

D. Parameters a and c should be integers only

Solution

1. Step 1: Identify the intended formula

   LLM scaling laws show loss decreases as model size and data increase, so exponents must be negative.

2. Step 2: Check the code exponents

   The code uses positive exponents (N**b and D**d), which incorrectly increase loss with size.

3. Final Answer:

   Exponents should be negative to show loss decreases with size -> Option C

4. Quick Check:

   Negative exponents mean loss decreases as size grows [OK]

Hint: Remember loss decreases, so exponents must be negative [OK]

Common Mistakes:

• Thinking multiplication is missing
• Believing return type must be tuple
• Assuming parameter types must be integers

5. You want to reduce the loss of a large language model efficiently. According to LLM scaling laws, which strategy is best if you have limited compute but can increase data or model size?

hard

A. Increase dataset size moderately while keeping model size fixed

B. Increase model size drastically without adding data

C. Keep both model size and data fixed and train longer

D. Reduce dataset size to speed up training

Solution

1. Step 1: Understand compute constraints and scaling laws

   Scaling laws show loss improves with both model size and data, but compute limits large model increases.

2. Step 2: Choose strategy fitting limited compute

   Increasing data moderately is cheaper than drastically increasing model size, so Increase dataset size moderately while keeping model size fixed is best.

3. Final Answer:

   Increase dataset size moderately while keeping model size fixed -> Option A

4. Quick Check:

   Limited compute favors data increase over big model growth [OK]

Hint: With limited compute, grow data before model size [OK]

Common Mistakes:

• Thinking bigger model always better regardless of compute
• Ignoring compute limits and training time
• Reducing data harms performance