L1 and L2 regularization in TensorFlow - Interactive Code Practice

Practice - 5 Tasks

1fill in blank

easy

Complete the code to add L2 regularization to a Dense layer in TensorFlow.

TensorFlow

layer = tf.keras.layers.Dense(64, kernel_regularizer=tf.keras.regularizers.[1](0.01))

Drag options to blanks, or click blank then click option'

Al2

Bl1_l2

Cl1

Dnone

Attempts:

3 left

💡 Hint

Common Mistakes

Using 'l1' instead of 'l2' for L2 regularization.

Forgetting to specify the regularization parameter inside the parentheses.

The l2 function applies L2 regularization, which adds a penalty proportional to the square of the weights.

2fill in blank

medium

Complete the code to add L1 regularization with strength 0.005 to a Dense layer.

TensorFlow

layer = tf.keras.layers.Dense(32, kernel_regularizer=tf.keras.regularizers.[1](0.005))

Drag options to blanks, or click blank then click option'

Al1

Bl2

Cl1_l2

Dnone

Attempts:

3 left

💡 Hint

Common Mistakes

Using 'l2' instead of 'l1' for L1 regularization.

Not specifying the regularization strength correctly.

The l1 function applies L1 regularization, which adds a penalty proportional to the absolute value of the weights.

3fill in blank

hard

Fix the error in the code to correctly apply both L1 and L2 regularization.

TensorFlow

layer = tf.keras.layers.Dense(128, kernel_regularizer=tf.keras.regularizers.[1](l1=0.001, l2=0.002))

Drag options to blanks, or click blank then click option'

Al2

Bl1_l2

Cnone

Dl1

Attempts:

3 left

💡 Hint

Common Mistakes

Using only 'l1' or 'l2' when both are needed.

Not passing both l1 and l2 parameters.

The l1_l2 function allows applying both L1 and L2 regularization together by specifying both penalties.

4fill in blank

hard

Fill both blanks to create a model with L1 regularization on the first layer and L2 on the second.

TensorFlow

model = tf.keras.Sequential([
    tf.keras.layers.Dense(64, kernel_regularizer=tf.keras.regularizers.[1](0.01), activation='relu'),
    tf.keras.layers.Dense(10, kernel_regularizer=tf.keras.regularizers.[2](0.02), activation='softmax')
])

Drag options to blanks, or click blank then click option'

Al1

Bl2

Cl1_l2

Dnone

Attempts:

3 left

💡 Hint

Common Mistakes

Mixing up L1 and L2 regularizers.

Using the same regularizer for both layers.

The first layer uses l1 regularization to encourage sparsity, and the second uses l2 to penalize large weights.

5fill in blank

hard

Fill all three blanks to create a dictionary comprehension that maps layer names to their regularization type strings ('L1', 'L2', or 'L1_L2').

TensorFlow

layers = {'dense1': tf.keras.regularizers.l1(0.01), 'dense2': tf.keras.regularizers.l2(0.02), 'dense3': tf.keras.regularizers.l1_l2(l1=0.001, l2=0.002)}
reg_types = {name: [1] for name, reg in layers.items() if isinstance(reg, [2])}
reg_types.update({name: 'L2' for name, reg in layers.items() if isinstance(reg, [3])})

Drag options to blanks, or click blank then click option'

A'L1'

Btf.keras.regularizers.L1

Ctf.keras.regularizers.L2

Dtf.keras.regularizers.L1L2

Attempts:

3 left

💡 Hint

Common Mistakes

Using incorrect class names for isinstance checks.

Putting the string label in the wrong blank.

This code maps layer names to strings describing their regularization type by checking the instance type of each regularizer.

Practice

1. What is the main purpose of using L1 and L2 regularization in TensorFlow models?

easy

A. To change the activation function of the model

B. To increase the size of the model weights

C. To prevent overfitting by adding penalties to large weights

D. To speed up the training process by skipping layers

Solution

1. Step 1: Understand the role of regularization

   L1 and L2 regularization add penalties to the loss function based on the size of the weights to reduce overfitting.

2. Step 2: Identify the effect on model weights

   These penalties discourage large weights, helping the model generalize better on new data.

3. Final Answer:

   To prevent overfitting by adding penalties to large weights -> Option C

4. Quick Check:

   Regularization = Prevent overfitting [OK]

Hint: Regularization adds penalty to weights to avoid overfitting [OK]

Common Mistakes:

• Thinking regularization speeds up training
• Confusing regularization with activation functions
• Believing regularization increases weight size

2. Which of the following is the correct way to add L2 regularization to a Dense layer in TensorFlow?

easy

A. Dense(64, kernel_regularizer=tf.keras.regularizers.l2(0.01))

B. Dense(64, regularizer=tf.keras.l2(0.01))

C. Dense(64, kernel_regularizer=tf.keras.regularizers.l1(0.01))

D. Dense(64, kernel_regularizer=tf.keras.regularizers.l1_l2(0.01))

Solution

1. Step 1: Recall the correct argument name

   The argument to add regularization in a Dense layer is kernel_regularizer.

2. Step 2: Use the correct regularizer function

   For L2 regularization, use tf.keras.regularizers.l2() with the penalty value.

3. Final Answer:

   Dense(64, kernel_regularizer=tf.keras.regularizers.l2(0.01)) -> Option A

4. Quick Check:

   Correct syntax = Dense(64, kernel_regularizer=tf.keras.regularizers.l2(0.01)) [OK]

Hint: Use kernel_regularizer with tf.keras.regularizers.l2 for L2 [OK]

Common Mistakes:

• Using 'regularizer' instead of 'kernel_regularizer'
• Using L1 regularizer when L2 is needed
• Wrong function call like tf.keras.l2 instead of tf.keras.regularizers.l2

3. What will be the effect on the weights after training a model with L1 regularization compared to no regularization?

medium

A. Weights will be larger to fit the training data better

B. Weights will not change compared to no regularization

C. Weights will be small but rarely zero

D. Weights will be exactly zero for some features, creating sparsity

Solution

1. Step 1: Understand L1 regularization effect

   L1 regularization adds absolute value penalties, encouraging some weights to become zero.

2. Step 2: Compare with no regularization

   Without regularization, weights can be large and non-sparse; L1 creates sparsity by zeroing some weights.

3. Final Answer:

   Weights will be exactly zero for some features, creating sparsity -> Option D

4. Quick Check:

   L1 = Sparsity with zero weights [OK]

Hint: L1 makes some weights zero, creating sparse models [OK]

Common Mistakes:

• Confusing L1 with L2 effects
• Thinking weights get larger with regularization
• Assuming weights remain unchanged

4. You wrote this code to add L2 regularization but get an error:

model.add(Dense(32, kernel_regularizer=tf.keras.regularizers.l2))

What is the error and how to fix it?

medium

A. Missing parentheses after l2; fix with l2(0.01)

B. kernel_regularizer should be regularizer

C. Dense layer does not support regularization

D. l2 should be replaced with l1

Solution

1. Step 1: Identify the error in regularizer usage

   The code uses tf.keras.regularizers.l2 without calling it as a function.

2. Step 2: Correct the syntax by adding parentheses and penalty value

   Use tf.keras.regularizers.l2(0.01) to create the regularizer instance.

3. Final Answer:

   Missing parentheses after l2; fix with l2(0.01) -> Option A

4. Quick Check:

   Call l2() with penalty value [OK]

Hint: Always call l1 or l2 with parentheses and penalty [OK]

Common Mistakes:

• Forgetting parentheses after l1 or l2
• Using 'regularizer' instead of 'kernel_regularizer'
• Thinking Dense layer can't have regularization

5. You want to build a TensorFlow model that uses both L1 and L2 regularization on the kernel weights of a Dense layer with penalties 0.001 and 0.01 respectively. Which code snippet correctly applies this?

hard

A. Dense(128, kernel_regularizer=tf.keras.regularizers.l2(0.01), kernel_regularizer=tf.keras.regularizers.l1(0.001))

B. Dense(128, kernel_regularizer=tf.keras.regularizers.l1_l2(l1=0.001, l2=0.01))

C. Dense(128, kernel_regularizer=tf.keras.regularizers.l1(0.001), bias_regularizer=tf.keras.regularizers.l2(0.01))

D. Dense(128, kernel_regularizer=tf.keras.regularizers.l1(0.001) + tf.keras.regularizers.l2(0.01))

Solution

1. Step 1: Understand how to combine L1 and L2 regularization

   TensorFlow provides l1_l2 regularizer to apply both penalties together on the same weights.

2. Step 2: Check the correct syntax for l1_l2

   Use tf.keras.regularizers.l1_l2(l1=0.001, l2=0.01) as the kernel_regularizer argument.

3. Final Answer:

   Dense(128, kernel_regularizer=tf.keras.regularizers.l1_l2(l1=0.001, l2=0.01)) -> Option B

4. Quick Check:

   Use l1_l2() for combined regularization [OK]

Hint: Use l1_l2() to combine L1 and L2 penalties [OK]

Common Mistakes:

• Trying to add two kernel_regularizer arguments
• Adding regularizers instead of using l1_l2
• Applying L1 to kernel and L2 to bias instead