LLM scaling laws in Prompt Engineering / GenAI - Interactive Code Practice

Practice - 5 Tasks

1fill in blank

easy

Complete the code to calculate the total number of parameters in a transformer model.

Prompt Engineering / GenAI

total_params = num_layers * num_heads * [1]

Drag options to blanks, or click blank then click option'

Ahidden_size

Bbatch_size

Csequence_length

Dlearning_rate

Attempts:

3 left

💡 Hint

Common Mistakes

Using batch size instead of hidden size

Confusing sequence length with parameter count

The total number of parameters depends on the hidden size, number of layers, and heads.

2fill in blank

medium

Complete the code to compute the loss decay rate over training steps.

Prompt Engineering / GenAI

loss = initial_loss * (step)[1]decay_rate

Drag options to blanks, or click blank then click option'

A*

B**

C+

D-

Attempts:

3 left

💡 Hint

Common Mistakes

Using multiplication instead of exponentiation

Using addition or subtraction incorrectly

Loss typically decays as a power law, so we use the power operator.

3fill in blank

hard

Fix the error in the code to calculate the optimal model size given compute budget.

Prompt Engineering / GenAI

optimal_size = compute_budget [1] (training_steps * batch_size)

Drag options to blanks, or click blank then click option'

A/

B*

C+

D-

Attempts:

3 left

💡 Hint

Common Mistakes

Using multiplication instead of division

Adding or subtracting compute budget incorrectly

Optimal size is compute budget divided by total training tokens (steps * batch size).

4fill in blank

hard

Fill both blanks to create a dictionary of loss values over epochs where loss decreases.

Prompt Engineering / GenAI

losses = {epoch: initial_loss * (decay_rate [1] epoch) for epoch in range(1, num_epochs + 1) if epoch [2] max_epoch}

Drag options to blanks, or click blank then click option'

A**

B<=

C>

D+

Attempts:

3 left

💡 Hint

Common Mistakes

Using addition instead of exponentiation

Using '>' instead of '<=' for filtering epochs

Loss decreases exponentially with epochs (using '**'), and we include epochs up to max_epoch (using '<=').

5fill in blank

hard

Fill all three blanks to create a dictionary mapping model sizes to their expected loss if loss decreases with size.

Prompt Engineering / GenAI

expected_losses = {size: base_loss [1] (size [2] scale_factor) [3] 2 for size in model_sizes if size [2] scale_factor}

Drag options to blanks, or click blank then click option'

A-

B/

C**

D*

Attempts:

3 left

💡 Hint

Common Mistakes

Using subtraction instead of division

Using wrong operator for power

Confusing division with multiplication

Loss decreases with size: base_loss / ((size / scale_factor) ** 2).