Learning rate selection in Computer Vision - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

Why is choosing the right learning rate important?

Imagine you are teaching a robot to recognize cats in photos. You adjust how fast it learns by setting a learning rate. What happens if the learning rate is too high?

A The robot will always get perfect results immediately.

B The robot learns too slowly and takes a long time to improve.

C The robot might miss the correct answer because it jumps too much and never settles.

D The robot ignores the photos and guesses randomly.

💡 Hint

Think about how big steps affect finding the right path.

A high learning rate causes the model to take big steps in learning, which can overshoot the best solution and fail to settle, causing unstable training.

❓ Predict Output

intermediate

Output of training loss with different learning rates

Look at the training loss values after 3 epochs with different learning rates. Which learning rate likely caused the loss to increase instead of decrease?

Computer Vision

losses = {
    0.001: [0.9, 0.7, 0.5],
    0.1: [0.9, 1.2, 1.5],
    0.01: [0.9, 0.6, 0.4],
    0.0001: [0.9, 0.85, 0.8]
}

# Which learning rate caused loss to increase?

A 0.1

B 0.001

C 0.01

D 0.0001

💡 Hint

Look for loss values that get bigger over time.

With learning rate 0.1, loss increases from 0.9 to 1.5, indicating unstable training due to too high learning rate.

❓ Hyperparameter

advanced

Choosing learning rate for a convolutional neural network

You train a convolutional neural network on images. You want to try these learning rates: 0.1, 0.01, 0.001, 0.0001. Which learning rate is most likely to cause the model to converge smoothly without overshooting?

A 0.01

B 1.0

C 0.1

D 0.0001

💡 Hint

Think about typical learning rates used in image models.

0.01 is a common starting learning rate for CNNs that balances speed and stability.

❓ Metrics

advanced

Effect of learning rate on validation accuracy

After training a model with different learning rates, you get these validation accuracies:

• 0.1: 60%
• 0.01: 85%
• 0.001: 80%
• 0.0001: 50%

Which learning rate likely caused underfitting?

A 0.1

B 0.01

C 0.001

D 0.0001

💡 Hint

Underfitting means the model learns too slowly and performs poorly.

Learning rate 0.0001 is too small, causing slow learning and low accuracy, a sign of underfitting.

🔧 Debug

expert

Diagnosing training failure due to learning rate

You train a deep neural network but notice the training loss stays very high and does not improve. You suspect the learning rate is the cause. Which of these symptoms best supports that the learning rate is too high?

A Training loss decreases smoothly over epochs.

B Training loss fluctuates wildly and sometimes increases sharply.

C Training loss decreases very slowly and plateaus early.

D Training accuracy is perfect but validation accuracy is low.

💡 Hint

Think about what happens when steps are too big during learning.

Wild fluctuations and sharp increases in loss indicate the learning rate is too high, causing unstable training.

Practice

1.

What does the learning rate control in training a computer vision model?

easy

A. The number of layers in the model

B. The size of the input images

C. How fast the model updates its knowledge

D. The type of activation function used

Solution

1. Step 1: Understand the role of learning rate

   The learning rate determines how much the model changes its weights after seeing each example.

2. Step 2: Connect learning rate to model updates

   A higher learning rate means faster updates, while a lower rate means slower updates.

3. Final Answer:

   How fast the model updates its knowledge -> Option C

4. Quick Check:

   Learning rate controls update speed = C [OK]

Hint: Learning rate = speed of model learning updates [OK]

Common Mistakes:

• Confusing learning rate with model size
• Thinking learning rate changes input data
• Mixing learning rate with activation functions

2.

Which of the following is the correct way to set a learning rate of 0.01 using PyTorch's SGD optimizer?

import torch.optim as optim
optimizer = optim.SGD(model.parameters(), lr=___)

easy

A. 0.01

B. 0.1

C. "0.01"

D. learning_rate

Solution

1. Step 1: Check the expected type for learning rate

   The learning rate parameter expects a float number, not a string or variable name.

2. Step 2: Identify the correct float value for 0.01

   Using 0.01 as a float sets the learning rate correctly.

3. Final Answer:

   0.01 -> Option A

4. Quick Check:

   Learning rate as float = 0.01 [OK]

Hint: Use float numbers, not strings or variables, for lr [OK]

Common Mistakes:

• Using string "0.01" instead of float 0.01
• Passing undefined variable learning_rate
• Setting lr to 0.1 by mistake

3.

Consider this training loop snippet for a vision model:

learning_rate = 0.5
for epoch in range(3):
    loss = train_one_epoch(model, data, learning_rate)
    print(f"Epoch {epoch+1} loss: {loss:.2f}")

If the learning rate is too high, what is the most likely output behavior?

medium

A. Loss becomes zero immediately

B. Loss stays constant

C. Loss steadily decreases each epoch

D. Loss fluctuates or increases wildly

Solution

1. Step 1: Understand effect of high learning rate

   A very high learning rate like 0.5 can cause the model to overshoot the best weights, making training unstable.

2. Step 2: Predict loss behavior with unstable training

   Loss will not steadily decrease but will jump up and down or increase.

3. Final Answer:

   Loss fluctuates or increases wildly -> Option D

4. Quick Check:

   High lr causes unstable loss = A [OK]

Hint: High learning rate causes unstable or rising loss [OK]

Common Mistakes:

• Assuming loss always decreases regardless of lr
• Thinking loss becomes zero immediately
• Confusing constant loss with stable training

4.

Given this code snippet, identify the error related to learning rate usage:

optimizer = torch.optim.Adam(model.parameters(), lr=0.001)
for epoch in range(5):
    loss = train(model, data)
    optimizer.step()
    optimizer.zero_grad()

medium

A. optimizer.step() called before loss.backward()

B. Learning rate is too high for Adam optimizer

C. optimizer.zero_grad() should be called after optimizer.step()

D. Learning rate should be set inside the loop

Solution

1. Step 1: Check optimizer usage order

   Before calling optimizer.step(), gradients must be computed by loss.backward().

2. Step 2: Identify missing backward call

   The code misses loss.backward(), so optimizer.step() updates without gradients.

3. Final Answer:

   optimizer.step() called before loss.backward() -> Option A

4. Quick Check:

   Missing loss.backward() before step = B [OK]

Hint: Call loss.backward() before optimizer.step() [OK]

Common Mistakes:

• Thinking learning rate 0.001 is too high for Adam
• Believing zero_grad() order is wrong here
• Assuming learning rate must change each epoch

5.

You want to train a deep vision model on a new dataset. You start with a learning rate of 0.1 but notice training loss does not decrease. What is the best next step?

hard

A. Remove the learning rate parameter from the optimizer

B. Decrease the learning rate to 0.01 and try again

C. Keep the learning rate at 0.1 and train longer

D. Increase the learning rate to 1.0 for faster learning

Solution

1. Step 1: Analyze why loss does not decrease

   A high learning rate like 0.1 can cause the model to skip the best weights, preventing loss decrease.

2. Step 2: Choose a safer learning rate adjustment

   Lowering the learning rate to 0.01 allows smaller, stable updates to improve training.

3. Final Answer:

   Decrease the learning rate to 0.01 and try again -> Option B

4. Quick Check:

   Lower lr if loss stuck = D [OK]

Hint: Lower learning rate if loss doesn't drop [OK]

Common Mistakes:

• Increasing learning rate when training fails
• Ignoring learning rate and training longer
• Removing learning rate parameter entirely