Learning rate selection in Computer Vision - Model Metrics & Evaluation

Metrics & Evaluation - Learning rate selection

Which metric matters for Learning Rate Selection and WHY

When choosing a learning rate, the key metric to watch is the training loss and validation loss over time. These show how well the model is learning. A good learning rate helps the loss go down steadily without jumping around or getting stuck.

We also look at accuracy on validation data to see if the model is improving in making correct predictions. If accuracy improves smoothly, the learning rate is likely good.

Why? Because the learning rate controls how big each step is when the model learns. Too big, and the model jumps past good answers. Too small, and learning is very slow or stuck.

Confusion Matrix or Equivalent Visualization

Learning rate itself does not have a confusion matrix, but its effect shows in training curves:

Epoch | Training Loss | Validation Loss | Validation Accuracy
------------------------------------------------------------
  1   |     1.2      |      1.3       |       45%
  2   |     0.9      |      1.0       |       55%
  3   |     0.7      |      0.8       |       65%
  ... |     ...      |      ...       |       ...

Good learning rate: smooth, steady loss decrease and accuracy increase.

Too high learning rate: loss jumps up and down or diverges.

Too low learning rate: loss decreases very slowly or plateaus.

Tradeoff: Learning Rate Size vs Model Performance

High learning rate: Fast learning but risk of missing the best solution. Loss may bounce or increase.

Low learning rate: Stable learning but very slow progress. May get stuck in a bad solution.

Example: Imagine trying to find the bottom of a valley blindfolded.

• Big steps (high learning rate) might make you overshoot the bottom repeatedly.
• Small steps (low learning rate) make you move slowly but carefully.

Choosing the right step size helps you reach the bottom efficiently.

What Good vs Bad Learning Rate Looks Like

Good learning rate:

• Training and validation loss decrease steadily.
• Validation accuracy improves smoothly.
• No sudden jumps or spikes in loss.

Bad learning rate (too high):

• Loss values jump up and down or increase.
• Validation accuracy fluctuates or drops.
• Training may fail to converge.

Bad learning rate (too low):

• Loss decreases very slowly or plateaus.
• Accuracy improves very slowly or not at all.
• Training takes too long.

Common Pitfalls in Learning Rate Selection

• Ignoring validation loss: Only watching training loss can hide overfitting or poor generalization.
• Using a fixed learning rate: Sometimes a learning rate schedule or decay helps improve training.
• Too large initial learning rate: Can cause training to diverge immediately.
• Too small learning rate: Training may get stuck in local minima or take too long.
• Not tuning learning rate with batch size: Larger batch sizes often need different learning rates.

Self Check

Your model training shows loss jumping up and down wildly and validation accuracy is not improving after many epochs. You are using a learning rate of 0.1. Is this good?

Answer: No, this learning rate is likely too high. The jumps in loss and no accuracy improvement mean the model is not learning well. Try lowering the learning rate to get smoother, steady training progress.

Key Result

Learning rate affects training loss and validation accuracy curves; a good rate leads to steady loss decrease and accuracy improvement.

Practice

1.

What does the learning rate control in training a computer vision model?

easy

A. The number of layers in the model

B. The size of the input images

C. How fast the model updates its knowledge

D. The type of activation function used

Solution

1. Step 1: Understand the role of learning rate

   The learning rate determines how much the model changes its weights after seeing each example.

2. Step 2: Connect learning rate to model updates

   A higher learning rate means faster updates, while a lower rate means slower updates.

3. Final Answer:

   How fast the model updates its knowledge -> Option C

4. Quick Check:

   Learning rate controls update speed = C [OK]

Hint: Learning rate = speed of model learning updates [OK]

Common Mistakes:

• Confusing learning rate with model size
• Thinking learning rate changes input data
• Mixing learning rate with activation functions

2.

Which of the following is the correct way to set a learning rate of 0.01 using PyTorch's SGD optimizer?

import torch.optim as optim
optimizer = optim.SGD(model.parameters(), lr=___)

easy

A. 0.01

B. 0.1

C. "0.01"

D. learning_rate

Solution

1. Step 1: Check the expected type for learning rate

   The learning rate parameter expects a float number, not a string or variable name.

2. Step 2: Identify the correct float value for 0.01

   Using 0.01 as a float sets the learning rate correctly.

3. Final Answer:

   0.01 -> Option A

4. Quick Check:

   Learning rate as float = 0.01 [OK]

Hint: Use float numbers, not strings or variables, for lr [OK]

Common Mistakes:

• Using string "0.01" instead of float 0.01
• Passing undefined variable learning_rate
• Setting lr to 0.1 by mistake

3.

Consider this training loop snippet for a vision model:

learning_rate = 0.5
for epoch in range(3):
    loss = train_one_epoch(model, data, learning_rate)
    print(f"Epoch {epoch+1} loss: {loss:.2f}")

If the learning rate is too high, what is the most likely output behavior?

medium

A. Loss becomes zero immediately

B. Loss stays constant

C. Loss steadily decreases each epoch

D. Loss fluctuates or increases wildly

Solution

1. Step 1: Understand effect of high learning rate

   A very high learning rate like 0.5 can cause the model to overshoot the best weights, making training unstable.

2. Step 2: Predict loss behavior with unstable training

   Loss will not steadily decrease but will jump up and down or increase.

3. Final Answer:

   Loss fluctuates or increases wildly -> Option D

4. Quick Check:

   High lr causes unstable loss = A [OK]

Hint: High learning rate causes unstable or rising loss [OK]

Common Mistakes:

• Assuming loss always decreases regardless of lr
• Thinking loss becomes zero immediately
• Confusing constant loss with stable training

4.

Given this code snippet, identify the error related to learning rate usage:

optimizer = torch.optim.Adam(model.parameters(), lr=0.001)
for epoch in range(5):
    loss = train(model, data)
    optimizer.step()
    optimizer.zero_grad()

medium

A. optimizer.step() called before loss.backward()

B. Learning rate is too high for Adam optimizer

C. optimizer.zero_grad() should be called after optimizer.step()

D. Learning rate should be set inside the loop

Solution

1. Step 1: Check optimizer usage order

   Before calling optimizer.step(), gradients must be computed by loss.backward().

2. Step 2: Identify missing backward call

   The code misses loss.backward(), so optimizer.step() updates without gradients.

3. Final Answer:

   optimizer.step() called before loss.backward() -> Option A

4. Quick Check:

   Missing loss.backward() before step = B [OK]

Hint: Call loss.backward() before optimizer.step() [OK]

Common Mistakes:

• Thinking learning rate 0.001 is too high for Adam
• Believing zero_grad() order is wrong here
• Assuming learning rate must change each epoch

5.

You want to train a deep vision model on a new dataset. You start with a learning rate of 0.1 but notice training loss does not decrease. What is the best next step?

hard

A. Remove the learning rate parameter from the optimizer

B. Decrease the learning rate to 0.01 and try again

C. Keep the learning rate at 0.1 and train longer

D. Increase the learning rate to 1.0 for faster learning

Solution

1. Step 1: Analyze why loss does not decrease

   A high learning rate like 0.1 can cause the model to skip the best weights, preventing loss decrease.

2. Step 2: Choose a safer learning rate adjustment

   Lowering the learning rate to 0.01 allows smaller, stable updates to improve training.

3. Final Answer:

   Decrease the learning rate to 0.01 and try again -> Option B

4. Quick Check:

   Lower lr if loss stuck = D [OK]

Hint: Lower learning rate if loss doesn't drop [OK]

Common Mistakes:

• Increasing learning rate when training fails
• Ignoring learning rate and training longer
• Removing learning rate parameter entirely