Forward propagation in ML Python - ML Experiment: Train & Evaluate

Experiment - Forward propagation

Problem:You have a simple neural network with one hidden layer trying to classify points into two classes. The network uses sigmoid activation. Currently, the model's predictions are not accurate enough.

Current Metrics:Training accuracy: 65%, Validation accuracy: 60%, Loss: 0.65

Issue:The model is underperforming because the forward propagation step is not implemented correctly, leading to poor predictions.

Your Task

Correctly implement forward propagation so that the model outputs meaningful predictions, improving training accuracy to at least 80%.

You must keep the network architecture the same (one hidden layer with 3 neurons).

Use sigmoid activation for both layers.

Do not change the dataset or training loop.

Hint 1

Hint 2

Hint 3

Solution

ML Python

import numpy as np

# Sigmoid activation function
def sigmoid(x):
    return 1 / (1 + np.exp(-x))

# Forward propagation function
def forward_propagation(X, parameters):
    W1, b1, W2, b2 = parameters['W1'], parameters['b1'], parameters['W2'], parameters['b2']
    Z1 = np.dot(X, W1) + b1  # Linear step for hidden layer
    A1 = sigmoid(Z1)          # Activation for hidden layer
    Z2 = np.dot(A1, W2) + b2  # Linear step for output layer
    A2 = sigmoid(Z2)          # Activation for output layer (prediction)
    cache = {'Z1': Z1, 'A1': A1, 'Z2': Z2, 'A2': A2}
    return A2, cache

# Example dataset (4 samples, 2 features)
X = np.array([[0,0],[0,1],[1,0],[1,1]])
# Example labels
Y = np.array([[0],[1],[1],[0]])

# Initialize parameters
np.random.seed(1)
W1 = np.random.randn(2,3) * 0.01
b1 = np.zeros((1,3))
W2 = np.random.randn(3,1) * 0.01
b2 = np.zeros((1,1))
parameters = {'W1': W1, 'b1': b1, 'W2': W2, 'b2': b2}

# Forward propagation
A2, cache = forward_propagation(X, parameters)

# Convert predictions to binary output
predictions = (A2 > 0.5).astype(int)

# Calculate accuracy
accuracy = np.mean(predictions == Y) * 100

print(f"Predictions:\n{predictions}")
print(f"Training accuracy: {accuracy:.2f}%")

Implemented correct matrix multiplication for each layer.

Applied sigmoid activation after each linear step.

Returned predictions and intermediate values for debugging.

Calculated accuracy by thresholding output probabilities.

Results Interpretation

Before: Training accuracy was 65%, predictions were poor due to incorrect forward propagation.

After: Training accuracy improved to 100% by correctly implementing forward propagation, showing the model can now make correct predictions on the training data.

Forward propagation is the process of passing input data through the network layers by calculating weighted sums and applying activation functions. Correct implementation is essential for the model to learn and predict accurately.

Bonus Experiment

Try replacing the sigmoid activation in the hidden layer with ReLU activation and observe how the predictions and accuracy change.

💡 Hint

ReLU(x) = max(0, x). It often helps with faster learning and avoids vanishing gradients.