Experiment - Why training optimizes model weights

Problem:We want to understand how training changes model weights to make better predictions.

Current Metrics:Initial model accuracy is around 10% (random guessing) on the MNIST digit dataset.

Issue:The model starts with random weights and poor accuracy. We need to see how training improves weights and accuracy.

Your Task

Train a simple neural network on MNIST and observe how weights change and accuracy improves after training.

Use TensorFlow and Keras.

Use a simple model with one hidden layer.

Train for 5 epochs only.

Hint 1

Hint 2

Hint 3

Solution

TensorFlow

import tensorflow as tf
from tensorflow.keras import layers, models

# Load MNIST data
(x_train, y_train), (x_test, y_test) = tf.keras.datasets.mnist.load_data()

# Normalize data
x_train = x_train.astype('float32') / 255.0
x_test = x_test.astype('float32') / 255.0

# Flatten images
x_train = x_train.reshape(-1, 28*28)
x_test = x_test.reshape(-1, 28*28)

# Build simple model
model = models.Sequential([
    layers.Dense(64, activation='relu', input_shape=(28*28,)),
    layers.Dense(10, activation='softmax')
])

# Compile model
model.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'])

# Print weights before training
print('Weights before training:')
for layer in model.layers:
    weights = layer.get_weights()
    if weights:
        print(weights[0].shape, 'weights sample:', weights[0][0][:5])

# Train model
history = model.fit(x_train, y_train, epochs=5, batch_size=32, validation_split=0.1, verbose=2)

# Print weights after training
print('Weights after training:')
for layer in model.layers:
    weights = layer.get_weights()
    if weights:
        print(weights[0].shape, 'weights sample:', weights[0][0][:5])

# Evaluate on test data
test_loss, test_acc = model.evaluate(x_test, y_test, verbose=0)
print(f'Test accuracy after training: {test_acc:.4f}')

Built a simple neural network with one hidden layer.

Normalized and flattened input images.

Compiled model with Adam optimizer and accuracy metric.

Printed weights before and after training to observe changes.

Trained model for 5 epochs to improve accuracy.

Results Interpretation

Before training: Weights are random, accuracy is about 10% (random guessing).

After training: Weights have changed significantly, accuracy improved to ~95% on training and ~94% on test data.

Training adjusts model weights step-by-step to reduce errors, making predictions more accurate. This shows how optimization during training improves model performance.

Bonus Experiment

Try training the same model but with a smaller hidden layer (e.g., 16 neurons) and observe how accuracy and weight changes differ.

💡 Hint

Smaller models may learn slower or less accurately, so compare training curves and weights.