How Neural Network Works: Simple Explanation and Example

A neural network works by passing input data through layers of connected nodes called neurons, where each neuron applies a simple calculation and passes the result forward. The network learns by adjusting connection strengths, called weights, to reduce errors between its predictions and actual results.

Syntax

A neural network consists of layers: an input layer, one or more hidden layers, and an output layer. Each layer has neurons that receive inputs, multiply them by weights, add a bias, and apply an activation function to produce outputs.

• Input layer: Receives raw data.
• Weights: Numbers that control the strength of connections.
• Bias: A number added to help the neuron activate.
• Activation function: A simple math function that decides if the neuron should 'fire' or not.
• Output layer: Produces the final prediction.

python

output = activation_function(sum(input_i * weight_i) + bias)

Example

This example shows a simple neural network with one hidden layer using Python and numpy. It predicts output from input data by forward passing through the network.

python

import numpy as np

def sigmoid(x):
    return 1 / (1 + np.exp(-x))

def sigmoid_derivative(x):
    return x * (1 - x)

# Input data: 4 samples, 3 features each
inputs = np.array([[0, 0, 1],
                   [1, 1, 1],
                   [1, 0, 1],
                   [0, 1, 1]])

# Expected output: 4 samples, 1 output each
expected_output = np.array([[0], [1], [1], [0]])

# Seed random numbers for reproducibility
np.random.seed(1)

# Initialize weights randomly with mean 0
weights_input_hidden = 2 * np.random.random((3, 4)) - 1
weights_hidden_output = 2 * np.random.random((4, 1)) - 1

# Learning rate
lr = 0.5

for iteration in range(10000):
    # Forward pass
    hidden_layer_input = np.dot(inputs, weights_input_hidden)
    hidden_layer_output = sigmoid(hidden_layer_input)

    final_input = np.dot(hidden_layer_output, weights_hidden_output)
    predicted_output = sigmoid(final_input)

    # Calculate error
    error = expected_output - predicted_output

    # Backpropagation
    d_predicted_output = error * sigmoid_derivative(predicted_output)
    error_hidden_layer = d_predicted_output.dot(weights_hidden_output.T)
    d_hidden_layer = error_hidden_layer * sigmoid_derivative(hidden_layer_output)

    # Update weights
    weights_hidden_output += hidden_layer_output.T.dot(d_predicted_output) * lr
    weights_input_hidden += inputs.T.dot(d_hidden_layer) * lr

print("Predicted Output after training:")
print(np.round(predicted_output, 3))

Output

Predicted Output after training:
[[0.001]
 [0.997]
 [0.997]
 [0.002]]

Common Pitfalls

Common mistakes when working with neural networks include:

• Using no activation function or a wrong one, which stops the network from learning complex patterns.
• Not normalizing input data, causing slow or failed training.
• Choosing too high or too low learning rates, leading to unstable or very slow learning.
• Ignoring overfitting, where the network memorizes training data but fails on new data.

python

import numpy as np

# Wrong: No activation function
inputs = np.array([1, 2, 3])
weights = np.array([0.5, -0.6, 0.1])
bias = 0.1
output_wrong = np.dot(inputs, weights) + bias

# Right: Using sigmoid activation
def sigmoid(x):
    return 1 / (1 + np.exp(-x))

output_right = sigmoid(np.dot(inputs, weights) + bias)

print(f"Wrong output (no activation): {output_wrong}")
print(f"Right output (with sigmoid): {output_right:.3f}")

Output

Wrong output (no activation): 0.8
Right output (with sigmoid): 0.690

Quick Reference

Remember these key points when working with neural networks:

• Input Layer: Receives data.
• Weights & Biases: Learnable parameters.
• Activation Function: Adds non-linearity (e.g., sigmoid, ReLU).
• Forward Pass: Calculate outputs layer by layer.
• Backpropagation: Adjust weights to reduce error.
• Learning Rate: Controls update size.

Key Takeaways

A neural network learns by adjusting weights to reduce prediction errors.

Activation functions let the network learn complex patterns beyond simple sums.

Proper input scaling and learning rate are crucial for effective training.

Backpropagation is the process that updates weights based on errors.

Avoid overfitting by validating on new data and using regularization techniques.