Weight initialization strategies in TensorFlow - Cheat Sheet & Quick Revision

Recall & Review

beginner

What is the purpose of weight initialization in neural networks?

Weight initialization sets the starting values of the model's weights before training. Good initialization helps the model learn faster and avoid problems like vanishing or exploding gradients.

intermediate

Explain the difference between Xavier (Glorot) and He initialization.

Xavier initialization sets weights to keep variance of activations similar across layers, good for sigmoid/tanh activations. He initialization adjusts weights for ReLU activations to better handle their behavior and avoid dying neurons.

beginner

What problem does He initialization help to solve?

He initialization helps prevent the problem of dying ReLU neurons by scaling weights properly so that the variance of outputs stays stable through layers.

beginner

How does random initialization differ from zero initialization, and why is zero initialization usually bad?

Random initialization assigns small random values to weights, breaking symmetry so neurons learn different features. Zero initialization sets all weights to zero, causing all neurons to learn the same thing and preventing effective training.

beginner

Name two common TensorFlow functions used for weight initialization.

Two common TensorFlow initializers are tf.keras.initializers.GlorotUniform() for Xavier initialization and tf.keras.initializers.HeNormal() for He initialization.

Which weight initialization method is best suited for ReLU activation functions?

A He initialization

B Xavier initialization

C Zero initialization

D Random initialization with zeros

He initialization is designed to work well with ReLU activations by scaling weights to keep variance stable.

Why is zero initialization of weights usually a bad idea?

A It makes the model too complex

B It causes all neurons to learn the same features

C It uses too much memory

D It speeds up training too much

Zero initialization causes symmetry where all neurons update identically, preventing the network from learning diverse features.

What does Xavier initialization aim to keep stable across layers?

A Number of neurons

B Learning rate

C Variance of activations

D Bias values

Xavier initialization keeps the variance of activations stable to help gradients flow well during training.

Which TensorFlow initializer corresponds to Xavier initialization?

A tf.keras.initializers.Zeros()

B tf.keras.initializers.HeNormal()

C tf.keras.initializers.RandomNormal()

D tf.keras.initializers.GlorotUniform()

GlorotUniform is TensorFlow's implementation of Xavier initialization.

What problem can occur if weights are initialized with very large values?

A Exploding gradients

B Vanishing gradients

C Faster convergence

D Better accuracy

Large initial weights can cause exploding gradients, making training unstable.

Describe why weight initialization matters and name two common strategies.

Think about how starting weights affect learning speed and stability.

Explain how He initialization helps with ReLU activations and what problem it prevents.

Consider the behavior of ReLU and how weights influence neuron output.