L1 and L2 regularization in TensorFlow

Introduction

L1 and L2 regularization help a model avoid overfitting by keeping its weights small and simple.

When your model learns the training data too well but performs poorly on new data.

When you want to make your model simpler and easier to understand.

When you want to reduce the effect of noisy or irrelevant features in your data.

When training deep neural networks to improve generalization.

When you want to prevent very large weights that can cause unstable training.

Syntax

TensorFlow

tf.keras.regularizers.l1(l1=0.01)
tf.keras.regularizers.l2(l2=0.01)
tf.keras.regularizers.l1_l2(l1=0.01, l2=0.01)

Use these regularizers inside layers by setting the 'kernel_regularizer' argument.

The l1 and l2 parameters control how strong the regularization is; higher means more penalty.

Examples

This adds L1 regularization to the weights of a Dense layer.

TensorFlow

layer = tf.keras.layers.Dense(64, kernel_regularizer=tf.keras.regularizers.l1(0.01))

This adds L2 regularization to the weights of a Dense layer.

TensorFlow

layer = tf.keras.layers.Dense(64, kernel_regularizer=tf.keras.regularizers.l2(0.01))

This adds both L1 and L2 regularization to the weights.

TensorFlow

layer = tf.keras.layers.Dense(64, kernel_regularizer=tf.keras.regularizers.l1_l2(l1=0.01, l2=0.01))

Sample Model

This code creates a simple model with L2 regularization on the first Dense layer. It trains on random data and prints the final loss and accuracy.

TensorFlow

import tensorflow as tf
from tensorflow.keras import layers, models, regularizers

# Create simple dataset
import numpy as np
x_train = np.random.rand(100, 10)
y_train = (np.sum(x_train, axis=1) > 5).astype(int)

# Build model with L2 regularization
model = models.Sequential([
    layers.Dense(32, activation='relu', kernel_regularizer=regularizers.l2(0.01), input_shape=(10,)),
    layers.Dense(1, activation='sigmoid')
])

model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])

history = model.fit(x_train, y_train, epochs=5, verbose=0)

# Print training loss and accuracy
print(f"Loss: {history.history['loss'][-1]:.4f}")
print(f"Accuracy: {history.history['accuracy'][-1]:.4f}")

Important Notes

L1 regularization tends to make some weights exactly zero, which can help with feature selection.

L2 regularization makes weights smaller but usually not zero, helping to keep the model smooth.

You can combine L1 and L2 to get benefits of both.

Summary

L1 and L2 regularization help prevent overfitting by adding a penalty to large weights.

Use 'kernel_regularizer' in layers to apply these penalties.

L1 can create sparse models; L2 keeps weights small but non-zero.