Weight initialization strategies in TensorFlow - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

Why use He initialization in deep neural networks?

He initialization is often recommended for deep networks with ReLU activations. Why is this the case?

A It randomly sets some weights to zero to create sparsity.

B It sets all weights to zero to simplify training.

C It initializes weights with very large values to speed up convergence.

D It helps keep the variance of activations constant across layers, preventing vanishing or exploding gradients.

💡 Hint

Think about how activation variance affects gradient flow in deep networks.

He initialization scales weights to maintain activation variance through layers, especially with ReLU, which helps avoid vanishing or exploding gradients.

❓ Predict Output

intermediate

Output shape after Xavier initialization in TensorFlow

What is the shape of the weights tensor initialized by the following code?

TensorFlow

import tensorflow as tf
initializer = tf.keras.initializers.GlorotUniform()
weights = initializer(shape=(64, 128))
print(weights.shape)

A (64,)

B (128, 64)

C (64, 128)

D (128,)

💡 Hint

Look at the shape argument passed to the initializer.

The initializer creates a tensor with the exact shape specified, here (64, 128).

❓ Hyperparameter

advanced

Choosing initialization for sigmoid activation

Which weight initialization strategy is best suited for a network using sigmoid activation functions to reduce vanishing gradients?

A He initialization

B Xavier (Glorot) initialization

C Random normal with mean 0 and stddev 1

D All zeros initialization

💡 Hint

Consider the activation function's output range and how initialization affects gradient flow.

Xavier initialization is designed to keep variance stable for activations like sigmoid and tanh, helping reduce vanishing gradients.

❓ Metrics

advanced

Effect of poor weight initialization on training metrics

What is the most likely effect on training loss and accuracy if weights are initialized with very large random values?

A Training loss will be unstable and accuracy will be low due to exploding gradients.

B Training loss will decrease smoothly and accuracy will improve quickly.

C Training loss will be zero from the start and accuracy will be perfect.

D Training loss and accuracy will not change because initialization does not affect training.

💡 Hint

Think about how large initial weights affect gradient calculations.

Large initial weights cause activations and gradients to explode, making training unstable and accuracy poor.

🔧 Debug

expert

Identify the error in this custom weight initializer

What error will this TensorFlow custom initializer code raise?

TensorFlow

import tensorflow as tf
class CustomInit(tf.keras.initializers.Initializer):
    def __call__(self, shape, dtype=None):
        return tf.random.uniform(shape, minval=-1, maxval=1, dtype=dtype)

initializer = CustomInit()
weights = initializer(shape=(32, 32))

A No error, code runs correctly

B AttributeError because __call__ method is missing

C ValueError because shape is not a tuple

D TypeError because dtype is not passed to tf.random.uniform

💡 Hint

tf.random.uniform has a default dtype=tf.float32, so no error occurs.

No error occurs because tf.random.uniform uses dtype=tf.float32 by default when not specified.