Backpropagation concept in ML Python - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

What is the main purpose of backpropagation in training a neural network?

Imagine you are teaching a robot to recognize cats in photos. The robot guesses, but sometimes it is wrong. How does backpropagation help the robot improve its guesses?

A It randomly changes the robot's settings without considering the errors.

B It changes the robot's hardware to make it faster at recognizing cats.

C It adjusts the robot's internal settings by calculating how wrong the guess was and changing the settings to reduce future errors.

D It collects more photos of cats to show the robot for better learning.

💡 Hint

Think about how the robot learns from mistakes by adjusting what it knows.

Backpropagation calculates the error between the robot's guess and the correct answer, then adjusts the internal settings (weights) to reduce this error in future guesses.

❓ Predict Output

intermediate

What is the output of the gradient calculation step in backpropagation?

Given a simple neural network with one weight w and loss function L = (w * x - y)^2, where x=2 and y=4, what is the gradient of L with respect to w when w=1?

ML Python

x = 2
y = 4
w = 1
L = (w * x - y)**2
# Calculate dL/dw
dL_dw = 2 * (w * x - y) * x
print(dL_dw)

A -4

B 8

C 4

D -8

💡 Hint

Use the chain rule: dL/dw = 2 * (w*x - y) * x

Calculate error: (1*2 - 4) = -2. Then gradient: 2 * (-2) * 2 = -8.

❓ Model Choice

advanced

Which neural network architecture benefits most from backpropagation through time (BPTT)?

Backpropagation through time is a special version of backpropagation used for certain models. Which model type uses BPTT to learn from sequences?

A Recurrent Neural Networks (RNNs) for language modeling

B Feedforward Neural Networks for tabular data

C Convolutional Neural Networks (CNNs) for image recognition

D Autoencoders for dimensionality reduction

💡 Hint

Think about models that process data step-by-step over time.

BPTT is used to train RNNs because they handle sequences and need to propagate errors back through time steps.

❓ Hyperparameter

advanced

Which hyperparameter directly controls the size of weight updates during backpropagation?

When training a neural network, which setting decides how big each step is when adjusting weights after calculating gradients?

A Learning rate

B Batch size

C Number of epochs

D Activation function

💡 Hint

It is a small number that multiplies the gradient to update weights.

The learning rate scales the gradient during weight updates, controlling how fast or slow the model learns.

❓ Metrics

expert

After training a neural network with backpropagation, which metric best indicates if the model is overfitting?

You trained a model and see that training loss keeps decreasing but validation loss starts increasing. Which metric behavior shows overfitting?

A Both training and validation accuracy increase steadily

B Training accuracy increases while validation accuracy decreases

C Training loss increases while validation loss decreases

D Both training and validation loss decrease steadily

💡 Hint

Overfitting means the model learns training data too well but fails on new data.

When training accuracy improves but validation accuracy worsens, the model is overfitting.

Practice

1. What is the main purpose of backpropagation in training a neural network?

easy

A. To split the data into training and testing sets

B. To randomly change weights without any calculation

C. To increase the size of the neural network

D. To update the weights by moving backward through the network

Solution

1. Step 1: Understand the role of backpropagation

   Backpropagation is used to adjust the weights in a neural network by calculating errors from the output back to the input.

2. Step 2: Identify the correct purpose

   It moves backward through the network to update weights to reduce errors during training.

3. Final Answer:

   To update the weights by moving backward through the network -> Option D

4. Quick Check:

   Backpropagation = weight update backward [OK]

Hint: Backpropagation updates weights backward through the network [OK]

Common Mistakes:

• Thinking backpropagation randomly changes weights
• Confusing backpropagation with data splitting
• Believing it increases network size

2. Which of the following is the correct formula used in backpropagation to update a weight w given learning rate η and error gradient ∂E/∂w?

easy

A. w = w + η * ∂E/∂w

B. w = η * ∂E/∂w

C. w = w - η * ∂E/∂w

D. w = w / (η * ∂E/∂w)

Solution

1. Step 1: Recall the weight update rule in backpropagation

   Weights are updated by subtracting the product of learning rate and error gradient to reduce error.

2. Step 2: Match the formula

   The correct formula is w = w - η * ∂E/∂w, which moves weights opposite to the gradient direction.

3. Final Answer:

   w = w - η * ∂E/∂w -> Option C

4. Quick Check:

   Weight update = subtract learning rate times gradient [OK]

Hint: Weights subtract learning rate times gradient to reduce error [OK]

Common Mistakes:

• Adding instead of subtracting the gradient
• Replacing weight with only the product
• Dividing weights by gradient mistakenly

3. Consider this simple neural network code snippet for one weight update during backpropagation:

weight = 0.5
learning_rate = 0.1
error_gradient = 0.4
weight = weight - learning_rate * error_gradient
print(round(weight, 2))

What will be the printed output?

medium

A. 0.46

B. 0.40

C. 0.50

D. 0.54

Solution

1. Step 1: Calculate the weight update

   weight update = 0.5 - 0.1 * 0.4 = 0.5 - 0.04 = 0.46

2. Step 2: Print the rounded weight

   Rounded to two decimals, weight is 0.46.

3. Final Answer:

   0.46 -> Option A

4. Quick Check:

   0.5 - 0.1*0.4 = 0.46 [OK]

Hint: Subtract learning rate times gradient from weight [OK]

Common Mistakes:

• Adding instead of subtracting the product
• Rounding incorrectly
• Using wrong initial weight value

4. The following code tries to perform one step of backpropagation weight update but has an error:

weight = 0.3
learning_rate = 0.05
error_gradient = 0.2
weight = weight + learning_rate * error_gradient
print(weight)

What is the error and how to fix it?

medium

A. Error: weight should be initialized to zero, fix by setting weight=0

B. Error: Should subtract gradient product, fix by changing + to -

C. Error: learning_rate should be multiplied twice, fix by squaring it

D. No error, code is correct

Solution

1. Step 1: Identify the update rule mistake

   The code adds the product of learning rate and gradient, but backpropagation requires subtraction to reduce error.

2. Step 2: Correct the operator

   Change the + to - so weight = weight - learning_rate * error_gradient.

3. Final Answer:

   Error: Should subtract gradient product, fix by changing + to - -> Option B

4. Quick Check:

   Weight update subtracts gradient product [OK]

Hint: Backpropagation updates subtract gradient product [OK]

Common Mistakes:

• Adding instead of subtracting gradient product
• Changing weight initialization unnecessarily
• Multiplying learning rate incorrectly

5. In a neural network with two layers, if the output error is large, how does backpropagation help reduce this error through the layers?

hard

A. By calculating gradients layer by layer from output to input and updating weights accordingly

B. By increasing the number of neurons in each layer automatically

C. By randomly resetting all weights to zero and retraining

D. By only updating weights in the first layer and ignoring others

Solution

1. Step 1: Understand backpropagation across layers

   Backpropagation calculates error gradients starting from the output layer backward through each layer.

2. Step 2: Update weights in all layers

   It updates weights in every layer based on their contribution to the error, reducing overall error.

3. Final Answer:

   By calculating gradients layer by layer from output to input and updating weights accordingly -> Option A

4. Quick Check:

   Backpropagation = layer-wise gradient update [OK]

Hint: Backpropagation updates all layers backward to reduce error [OK]

Common Mistakes:

• Thinking it increases neurons automatically
• Resetting weights randomly
• Updating only the first layer weights