Batch normalization in TensorFlow - Cheat Sheet & Quick Revision

Recall & Review

beginner

What is batch normalization in machine learning?

Batch normalization is a technique to make training faster and more stable by normalizing the inputs of each layer. It adjusts and scales the data so the model learns better.

intermediate

Why do we use batch normalization during training?

We use batch normalization to reduce the problem of internal covariate shift, which means the distribution of inputs to layers changes during training. This helps the model train faster and generalize better.

intermediate

Which two parameters does batch normalization learn during training?

Batch normalization learns two parameters: gamma (scale) and beta (shift). These allow the model to restore the original distribution if needed after normalization.

advanced

How does batch normalization behave differently during training and inference?

During training, batch normalization uses the mean and variance of the current batch. During inference, it uses a moving average of mean and variance collected during training for stable predictions.

beginner

Show a simple TensorFlow code snippet to add batch normalization after a dense layer.

import tensorflow as tf

model = tf.keras.Sequential([
    tf.keras.layers.Dense(64, activation='relu'),
    tf.keras.layers.BatchNormalization(),
    tf.keras.layers.Dense(10, activation='softmax')
])

What problem does batch normalization mainly address?

A Vanishing gradients

B Overfitting

C Internal covariate shift

D Data augmentation

Batch normalization reduces internal covariate shift by normalizing layer inputs.

Which parameters are learned by batch normalization?

A Gamma and beta

B Mean and variance

C Weights and biases

D Learning rate and momentum

Batch normalization learns gamma (scale) and beta (shift) parameters.

During inference, batch normalization uses:

A Batch mean and variance

B Moving average of mean and variance

C Random values

D No normalization

Inference uses moving averages collected during training for stable normalization.

Where is batch normalization usually applied in a neural network?

A After linear layers and before activation

B After activation functions

C Before the input layer

D Only at the output layer

Batch normalization is typically applied after linear layers and before activation.

Which of these is a benefit of batch normalization?

A Slower training

B Increases model size

C Removes the need for activation functions

D Helps use higher learning rates

Batch normalization allows using higher learning rates and speeds up training.

Explain in your own words what batch normalization does and why it helps training.

Think about how changing input data affects learning and how normalization fixes it.

Describe how batch normalization behaves differently during training and inference phases.

Consider what data is available during training vs when the model is used.

Practice

1. What is the main purpose of BatchNormalization in a TensorFlow model?

easy

A. To add noise to the inputs for regularization

B. To normalize the inputs of a layer to speed up and stabilize training

C. To increase the learning rate automatically during training

D. To reduce the size of the model by removing neurons

Solution

1. Step 1: Understand batch normalization role

   Batch normalization normalizes inputs to a layer to keep data distribution stable during training.

2. Step 2: Identify its effect on training

   This normalization speeds up training and helps avoid issues like vanishing or exploding gradients.

3. Final Answer:

   To normalize the inputs of a layer to speed up and stabilize training -> Option B

4. Quick Check:

   BatchNormalization normalizes inputs [OK]

Hint: BatchNormalization normalizes inputs to stabilize training [OK]

Common Mistakes:

• Thinking it reduces model size
• Confusing it with learning rate adjustment
• Assuming it adds noise for regularization

2. Which of the following is the correct way to add a batch normalization layer after a dense layer in TensorFlow?

easy

A. model.add(tf.keras.layers.BatchNormalization(64)) before model.add(tf.keras.layers.Dense())

B. model.add(tf.keras.layers.Dense(64, batch_normalization=True))

C. model.add(tf.keras.layers.BatchNormalization()) after model.add(tf.keras.layers.Dense(64))

D. model.add(tf.keras.layers.Dense(64)) with batch_norm=True parameter

Solution

1. Step 1: Recall batch normalization usage

   BatchNormalization is a separate layer added after Dense, not a parameter inside Dense.

2. Step 2: Check syntax correctness

   model.add(tf.keras.layers.BatchNormalization()) after model.add(tf.keras.layers.Dense(64)) correctly adds BatchNormalization after Dense with no extra parameters.

3. Final Answer:

   model.add(tf.keras.layers.BatchNormalization()) after model.add(tf.keras.layers.Dense(64)) -> Option C

4. Quick Check:

   BatchNormalization is a separate layer added after Dense [OK]

Hint: BatchNormalization is a separate layer, not a Dense parameter [OK]

Common Mistakes:

• Trying to pass batch_normalization as Dense parameter
• Adding BatchNormalization before Dense layer
• Passing units to BatchNormalization layer

3. What will be the output shape of the following TensorFlow model snippet?

model = tf.keras.Sequential([
  tf.keras.layers.Dense(10, input_shape=(20,)),
  tf.keras.layers.BatchNormalization(),
  tf.keras.layers.Activation('relu')
])
output_shape = model.output_shape

medium

A. (None, 10)

B. (None, 20)

C. (10, None)

D. (20, 10)

Solution

1. Step 1: Analyze Dense layer output

   Dense(10) with input shape (20,) outputs shape (None, 10), where None is batch size.

2. Step 2: Check BatchNormalization and Activation effect

   BatchNormalization and Activation do not change shape, so output remains (None, 10).

3. Final Answer:

   (None, 10) -> Option A

4. Quick Check:

   Dense output shape = (None, 10) [OK]

Hint: BatchNormalization keeps shape same as previous layer output [OK]

Common Mistakes:

• Confusing input shape with output shape
• Assuming BatchNormalization changes output shape
• Mixing batch size dimension with feature dimension

4. Identify the error in this TensorFlow code snippet using batch normalization:

model = tf.keras.Sequential()
model.add(tf.keras.layers.BatchNormalization(input_shape=(32,)))
model.add(tf.keras.layers.Dense(64))
model.add(tf.keras.layers.Activation('relu'))

medium

A. BatchNormalization should not be the first layer with input_shape

B. Activation should be before BatchNormalization

C. Dense layer must come before BatchNormalization

D. BatchNormalization layer should come after Dense layer, not before

Solution

1. Step 1: Check layer order logic

   BatchNormalization normalizes outputs of previous layer, so it should come after Dense, not before.

2. Step 2: Validate input shape usage

   Input shape should be specified in first layer that processes input, usually Dense or Conv layer.

3. Final Answer:

   BatchNormalization layer should come after Dense layer, not before -> Option D

4. Quick Check:

   BatchNormalization follows Dense layer [OK]

Hint: BatchNormalization normalizes outputs, so place it after Dense [OK]

Common Mistakes:

• Placing BatchNormalization as first layer with input_shape
• Putting Activation before BatchNormalization
• Assuming BatchNormalization changes input shape

5. You want to add batch normalization to a convolutional neural network in TensorFlow. Which of the following is the best practice?

hard

A. Add BatchNormalization immediately after each convolutional layer and before activation

B. Add BatchNormalization only after the last convolutional layer

C. Add BatchNormalization before convolutional layers

D. Add BatchNormalization only after activation layers

Solution

1. Step 1: Understand batch normalization placement

   Best practice is to normalize outputs of convolutional layers before applying activation.

2. Step 2: Evaluate options

   Adding BatchNormalization after each conv layer and before activation stabilizes training and improves performance.

3. Final Answer:

   Add BatchNormalization immediately after each convolutional layer and before activation -> Option A

4. Quick Check:

   BatchNormalization follows conv layer, precedes activation [OK]

Hint: Place BatchNormalization after conv layer, before activation [OK]

Common Mistakes:

• Adding BatchNormalization only once at end
• Placing BatchNormalization before conv layers
• Adding BatchNormalization after activation