Weight initialization strategies in TensorFlow - Interactive Code Practice

Practice - 5 Tasks

1fill in blank

easy

Complete the code to initialize weights with zeros in TensorFlow.

TensorFlow

initializer = tf.keras.initializers.[1]()

Drag options to blanks, or click blank then click option'

AHeNormal

BZeros

CGlorotUniform

DRandomNormal

Attempts:

3 left

💡 Hint

Common Mistakes

Using RandomNormal instead of Zeros will initialize weights randomly, not zeros.

GlorotUniform and HeNormal are advanced initializers, not zeros.

The Zeros initializer sets all weights to zero, which is a simple weight initialization strategy.

2fill in blank

medium

Complete the code to initialize weights using He normal initialization in TensorFlow.

TensorFlow

initializer = tf.keras.initializers.[1]()

Drag options to blanks, or click blank then click option'

ARandomUniform

BGlorotUniform

CZeros

DHeNormal

Attempts:

3 left

💡 Hint

Common Mistakes

Choosing GlorotUniform instead of HeNormal may reduce performance for ReLU layers.

RandomUniform does not scale weights based on layer size.

HeNormal initializer is designed for layers with ReLU activation and initializes weights with a normal distribution scaled by the number of input units.

3fill in blank

hard

Fix the error in the code to use Xavier (Glorot) uniform initialization for a Dense layer.

TensorFlow

layer = tf.keras.layers.Dense(64, kernel_initializer=[1])

Drag options to blanks, or click blank then click option'

A'glorot_uniform'

B'random_normal'

C'he_normal'

D'zeros'

Attempts:

3 left

💡 Hint

Common Mistakes

Using 'he_normal' instead of 'glorot_uniform' changes the initialization strategy.

Using 'zeros' will initialize all weights to zero, which is not Xavier initialization.

The correct string identifier for Xavier initialization in TensorFlow is 'glorot_uniform'.

4fill in blank

hard

Fill both blanks to create a Dense layer with He uniform initializer and ReLU activation.

TensorFlow

layer = tf.keras.layers.Dense(128, kernel_initializer=[1], activation=[2])

Drag options to blanks, or click blank then click option'

A'he_uniform'

B'relu'

C'sigmoid'

D'glorot_normal'

Attempts:

3 left

💡 Hint

Common Mistakes

Using 'sigmoid' activation with He uniform initializer is less common and may reduce performance.

Using 'glorot_normal' initializer with ReLU activation is valid but not the best match here.

He uniform initializer is suitable for ReLU activation functions, which is why both are used together here.

5fill in blank

hard

Fill all three blanks to define a Sequential model with two Dense layers using Xavier initialization and softmax activation.

TensorFlow

model = tf.keras.Sequential([
    tf.keras.layers.Dense(64, kernel_initializer=[1], activation=[2]),
    tf.keras.layers.Dense(10, kernel_initializer=[3], activation='softmax')
])

Drag options to blanks, or click blank then click option'

A'glorot_uniform'

B'relu'

C'glorot_normal'

D'he_uniform'

Attempts:

3 left

💡 Hint

Common Mistakes

Using 'he_uniform' initializer in output layer is less common for softmax outputs.

Using 'relu' activation in output layer instead of 'softmax' is incorrect for classification.

The first Dense layer uses 'glorot_uniform' initializer with 'relu' activation. The second Dense layer uses 'glorot_normal' initializer with 'softmax' activation for classification.

Practice

1. Which weight initialization strategy is best suited for layers using ReLU activation in TensorFlow?

easy

A. Zeros initializer

B. GlorotUniform initializer

C. RandomNormal initializer with mean 0

D. HeNormal initializer

Solution

1. Step 1: Understand activation and initialization relation

   ReLU activation benefits from initial weights that maintain variance through layers to avoid dying neurons.

2. Step 2: Match initializer to activation

   HeNormal initializer is designed for ReLU, scaling weights to keep variance stable.

3. Final Answer:

   HeNormal initializer -> Option D

4. Quick Check:

   ReLU uses HeNormal [OK]

Hint: Use HeNormal for ReLU activations to keep variance stable [OK]

Common Mistakes:

• Choosing GlorotUniform for ReLU layers
• Using zeros initializer causing no learning
• RandomNormal without scaling causes slow training

2. Which of the following is the correct syntax to initialize weights with GlorotUniform in TensorFlow?

easy

A. tf.keras.initializers.glorot_uniform()

B. tf.initializers.GlorotUniform()

C. tf.keras.initializers.GlorotUniform()

D. tf.keras.initializers.Glorot_uniform()

Solution

1. Step 1: Recall TensorFlow initializer module

   Initializers are accessed via tf.keras.initializers with exact class names.

2. Step 2: Check correct class name and casing

   The correct class name is GlorotUniform with capital G and U, called as a class with parentheses.

3. Final Answer:

   tf.keras.initializers.GlorotUniform() -> Option C

4. Quick Check:

   Correct class name and module path [OK]

Hint: Use exact class name with correct casing and parentheses [OK]

Common Mistakes:

• Using lowercase function names instead of class
• Missing parentheses after class name
• Using wrong module path tf.initializers

3. What will be the output shape and initializer type of the following TensorFlow layer?

layer = tf.keras.layers.Dense(10, kernel_initializer=tf.keras.initializers.HeNormal())
print(layer.kernel_initializer)

medium

A. GlorotUniform initializer object, output shape (None, 10)

B. HeNormal initializer object, output shape (None, 10)

C. HeNormal initializer object, output shape (10, 10)

D. RandomNormal initializer object, output shape (None, 10)

Solution

1. Step 1: Identify initializer and layer output

   The layer uses HeNormal initializer explicitly for kernel weights and has 10 output units.

2. Step 2: Understand output shape of Dense layer

   Dense layers output shape is (batch_size, units), so shape is (None, 10) before input is given.

3. Final Answer:

   HeNormal initializer object, output shape (None, 10) -> Option B

4. Quick Check:

   HeNormal used and output units = 10 [OK]

Hint: Dense output shape is (None, units); initializer matches given class [OK]

Common Mistakes:

• Confusing output shape with kernel shape
• Assuming default initializer instead of HeNormal
• Mixing initializer types

4. You wrote this code to initialize weights but get an error:

layer = tf.keras.layers.Dense(5, kernel_initializer='he_normal')
model = tf.keras.Sequential([layer])
model.build(input_shape=(None, 3))
print(layer.kernel_initializer())

What is the cause of the error?

medium

A. Calling kernel_initializer as a function without arguments

B. Using string 'he_normal' instead of initializer object

C. Not specifying bias_initializer

D. Missing input_shape in layer constructor

Solution

1. Step 1: Understand kernel_initializer attribute type

   kernel_initializer is an initializer object, not a callable function without arguments.

2. Step 2: Identify error cause

   Calling kernel_initializer() without arguments causes error because it expects shape and dtype parameters.

3. Final Answer:

   Calling kernel_initializer as a function without arguments -> Option A

4. Quick Check:

   Initializer needs shape args, can't call empty () [OK]

Hint: Don't call kernel_initializer() without shape and dtype [OK]

Common Mistakes:

• Confusing string initializer with object
• Ignoring required arguments for initializer call
• Assuming bias_initializer causes this error

5. You want to initialize weights for a model with mixed activations: ReLU in some layers and sigmoid in others. Which strategy best improves training speed and accuracy?

hard

A. Use HeNormal for ReLU layers and GlorotUniform for sigmoid layers

B. Use GlorotUniform for all layers regardless of activation

C. Use RandomNormal with mean 0 and stddev 1 for all layers

D. Use Zeros initializer for all layers to start from zero

Solution

1. Step 1: Match initializers to activation functions

   HeNormal suits ReLU to maintain variance; GlorotUniform suits sigmoid/tanh to avoid saturation.

2. Step 2: Understand impact on training

   Proper matching prevents slow or stuck training and improves accuracy by keeping gradients healthy.

3. Final Answer:

   Use HeNormal for ReLU layers and GlorotUniform for sigmoid layers -> Option A

4. Quick Check:

   Match initializer to activation for best training [OK]

Hint: Match HeNormal to ReLU, GlorotUniform to sigmoid/tanh [OK]

Common Mistakes:

• Using same initializer for all activations
• Using zeros initializer causing no learning
• Using RandomNormal without scaling