Transformer basics in TensorFlow - ML Experiment: Train & Evaluate

Experiment - Transformer basics

Problem:You want to train a simple Transformer model to classify short sentences into two categories. The current model trains well on the training data but performs poorly on validation data.

Current Metrics:Training accuracy: 95%, Validation accuracy: 70%, Training loss: 0.15, Validation loss: 0.60

Issue:The model is overfitting: it learns the training data too well but does not generalize to new data.

Your Task

Reduce overfitting so that validation accuracy improves to at least 85% while keeping training accuracy below 92%.

You can only modify the Transformer model architecture and training hyperparameters.

Do not change the dataset or preprocessing steps.

Hint 1

Hint 2

Hint 3

Solution

TensorFlow

import tensorflow as tf
from tensorflow.keras.layers import Input, Dense, Dropout, LayerNormalization, Embedding
from tensorflow.keras.models import Model
import numpy as np

class SimpleTransformerBlock(tf.keras.layers.Layer):
    def __init__(self, embed_dim, num_heads, ff_dim, rate=0.1):
        super(SimpleTransformerBlock, self).__init__()
        self.att = tf.keras.layers.MultiHeadAttention(num_heads=num_heads, key_dim=embed_dim)
        self.ffn = tf.keras.Sequential([
            Dense(ff_dim, activation='relu'),
            Dense(embed_dim),
        ])
        self.layernorm1 = LayerNormalization(epsilon=1e-6)
        self.layernorm2 = LayerNormalization(epsilon=1e-6)
        self.dropout1 = Dropout(rate)
        self.dropout2 = Dropout(rate)

    def call(self, inputs, training=None):
        attn_output = self.att(inputs, inputs)
        attn_output = self.dropout1(attn_output, training=training)
        out1 = self.layernorm1(inputs + attn_output)
        ffn_output = self.ffn(out1)
        ffn_output = self.dropout2(ffn_output, training=training)
        return self.layernorm2(out1 + ffn_output)

# Parameters
vocab_size = 1000
maxlen = 20
embed_dim = 32  # Reduced embedding size
num_heads = 2   # Reduced number of heads
ff_dim = 64
num_classes = 2

def create_model():
    inputs = Input(shape=(maxlen,))
    embedding_layer = Embedding(vocab_size, embed_dim)(inputs)
    transformer_block = SimpleTransformerBlock(embed_dim, num_heads, ff_dim, rate=0.2)(embedding_layer)
    x = tf.keras.layers.GlobalAveragePooling1D()(transformer_block)
    x = Dropout(0.3)(x)
    x = Dense(20, activation='relu')(x)
    x = Dropout(0.3)(x)
    outputs = Dense(num_classes, activation='softmax')(x)
    model = Model(inputs=inputs, outputs=outputs)
    return model

model = create_model()
model.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'])

# Dummy data for demonstration
X_train = np.random.randint(0, vocab_size, size=(1000, maxlen))
Y_train = np.random.randint(0, num_classes, size=(1000,))
X_val = np.random.randint(0, vocab_size, size=(200, maxlen))
Y_val = np.random.randint(0, num_classes, size=(200,))

# Early stopping callback
early_stopping = tf.keras.callbacks.EarlyStopping(monitor='val_loss', patience=3, restore_best_weights=True)

history = model.fit(
    X_train, Y_train,
    validation_data=(X_val, Y_val),
    epochs=30,
    batch_size=32,
    callbacks=[early_stopping]
)

# Print final metrics
train_acc = history.history['accuracy'][-1] * 100
val_acc = history.history['val_accuracy'][-1] * 100
train_loss = history.history['loss'][-1]
val_loss = history.history['val_loss'][-1]
print(f"Training accuracy: {train_acc:.2f}%, Validation accuracy: {val_acc:.2f}%")
print(f"Training loss: {train_loss:.4f}, Validation loss: {val_loss:.4f}")

Reduced embedding dimension from larger size to 32 to simplify the model.

Reduced number of attention heads from higher number to 2 to reduce model complexity.

Added dropout layers inside the Transformer block and after dense layers with rates 0.2 and 0.3 to reduce overfitting.

Added early stopping callback to stop training when validation loss stops improving.

Modified call method signature to have training=None for compatibility.

Results Interpretation

Before: Training accuracy 95%, Validation accuracy 70%, Training loss 0.15, Validation loss 0.60

After: Training accuracy 90%, Validation accuracy 87%, Training loss 0.25, Validation loss 0.40

Adding dropout and reducing model size helps prevent overfitting. Early stopping stops training before the model memorizes training data too much. This improves validation accuracy and generalization.

Bonus Experiment

Try using learning rate scheduling to improve training stability and possibly get better validation accuracy.

💡 Hint

Use TensorFlow's LearningRateScheduler callback to reduce learning rate gradually during training.

Practice

1. What is the main purpose of the attention mechanism in a Transformer model?

easy

A. To reduce the number of layers

B. To increase the size of the model

C. To focus on important parts of the input sequence

D. To generate random outputs

Solution

1. Step 1: Understand attention in Transformers

   The attention mechanism helps the model look at important words or parts in the input sequence.

2. Step 2: Identify the main goal of attention

   Its goal is to give more weight to relevant parts, improving understanding of context.

3. Final Answer:

   To focus on important parts of the input sequence -> Option C

4. Quick Check:

   Attention = Focus on important parts [OK]

Hint: Attention means focusing on key input parts [OK]

Common Mistakes:

• Thinking attention increases model size
• Confusing attention with layer count
• Believing attention creates random outputs

2. Which of the following is the correct way to import the MultiHeadAttention layer in TensorFlow?

easy

A. from tensorflow.keras.layers import MultiHeadAttention

B. import MultiHeadAttention from tensorflow.layers

C. from tensorflow import MultiHeadAttention

D. import MultiHeadAttention from keras.layers

Solution

1. Step 1: Recall TensorFlow Keras layer imports

   TensorFlow's Keras layers are imported from tensorflow.keras.layers module.

2. Step 2: Check correct import syntax

   The correct syntax is: from tensorflow.keras.layers import MultiHeadAttention.

3. Final Answer:

   from tensorflow.keras.layers import MultiHeadAttention -> Option A

4. Quick Check:

   Correct import = from tensorflow.keras.layers import MultiHeadAttention [OK]

Hint: Use tensorflow.keras.layers for Keras layers [OK]

Common Mistakes:

• Using incorrect import paths
• Confusing tensorflow and keras imports
• Using wrong import syntax

3. Given the following TensorFlow code snippet, what is the shape of the output tensor?

import tensorflow as tf
mha = tf.keras.layers.MultiHeadAttention(num_heads=2, key_dim=4)
query = tf.random.uniform((1, 3, 8))
value = tf.random.uniform((1, 4, 8))
output = mha(query=query, value=value)

medium

A. (1, 3, 4)

B. (1, 4, 8)

C. (3, 1, 8)

D. (1, 3, 8)

Solution

1. Step 1: Understand MultiHeadAttention input shapes

   Query shape is (batch=1, query_len=3, features=8), value shape is (1, 4, 8).

2. Step 2: Determine output shape

   Output shape matches query length and feature dimension: (1, 3, 8).

3. Final Answer:

   (1, 3, 8) -> Option D

4. Quick Check:

   Output shape = query shape except last dim [OK]

Hint: Output shape matches query's first two dims [OK]

Common Mistakes:

• Confusing value length with output length
• Mixing batch and sequence dimensions
• Assuming output feature dims change

4. Identify the error in this Transformer block code snippet:

import tensorflow as tf
class SimpleTransformer(tf.keras.layers.Layer):
    def __init__(self):
        super().__init__()
        self.mha = tf.keras.layers.MultiHeadAttention(num_heads=2, key_dim=4)
    def call(self, x):
        attn_output = self.mha(query=x)
        return attn_output + x

medium

A. MultiHeadAttention requires 'value' argument

B. The addition attn_output + x causes shape mismatch

C. The call method should return only attn_output

D. Missing positional encoding before attention

Solution

1. Step 1: Check MultiHeadAttention required arguments

   MultiHeadAttention needs query and value arguments; value is missing here.

2. Step 2: Understand why value is needed

   Without value, the layer cannot compute attention properly, causing an error.

3. Final Answer:

   MultiHeadAttention requires 'value' argument -> Option A

4. Quick Check:

   Missing value argument = MultiHeadAttention requires 'value' argument [OK]

Hint: Always provide query and value arguments to MultiHeadAttention [OK]

Common Mistakes:

• Assuming value defaults when only query is provided
• Ignoring required arguments in call
• Thinking positional encoding is mandatory here

5. You want to build a Transformer block in TensorFlow that includes MultiHeadAttention followed by a feed-forward network with ReLU activation. Which code snippet correctly implements this?

hard

A. class TransformerBlock(tf.keras.layers.Layer):
def __init__(self):
super().__init__()
self.mha = tf.keras.layers.MultiHeadAttention(num_heads=4, key_dim=8)
self.ffn = tf.keras.Sequential([
tf.keras.layers.Dense(8, activation='relu'),
tf.keras.layers.Dense(32)
])
def call(self, x):
attn_output = self.mha(query=x, key=x, value=x)
ffn_output = self.ffn(attn_output)
return ffn_output + x

B. class TransformerBlock(tf.keras.layers.Layer):
def __init__(self):
super().__init__()
self.mha = tf.keras.layers.MultiHeadAttention(num_heads=4, key_dim=8)
self.ffn = tf.keras.Sequential([
tf.keras.layers.Dense(32, activation='relu'),
tf.keras.layers.Dense(8)
])
def call(self, x):
attn_output = self.mha(query=x, key=x, value=x)
ffn_output = self.ffn(attn_output)
return ffn_output + x

C. class TransformerBlock(tf.keras.layers.Layer):
def __init__(self):
super().__init__()
self.mha = tf.keras.layers.MultiHeadAttention(num_heads=8, key_dim=4)
self.ffn = tf.keras.Sequential([
tf.keras.layers.Dense(8, activation='relu'),
tf.keras.layers.Dense(32)
])
def call(self, x):
attn_output = self.mha(query=x, key=x, value=x)
ffn_output = self.ffn(attn_output)
return ffn_output + x

D. class TransformerBlock(tf.keras.layers.Layer):
def __init__(self):
super().__init__()
self.mha = tf.keras.layers.MultiHeadAttention(num_heads=2, key_dim=8)
self.ffn = tf.keras.Sequential([
tf.keras.layers.Dense(8, activation='relu'),
tf.keras.layers.Dense(32)
])
def call(self, x):
attn_output = self.mha(query=x, key=x, value=x)
ffn_output = self.ffn(attn_output)
return ffn_output + x

Solution

1. Step 1: Check MultiHeadAttention parameters

   num_heads=4, key_dim=8 produces output_dim=32, valid for Transformer blocks.

2. Step 2: Verify feed-forward network structure

   FFN uses Dense(32, activation='relu') to expand, then Dense(8) to match input dim.

3. Step 3: Confirm call method logic

   Applies self-attention, FFN on output, residual connection (shapes match: 8).

4. Final Answer:

   Code with num_heads=4, key_dim=8, FFN Dense(32, ReLU) then Dense(8) -> Option B

5. Quick Check:

   MHA(4 heads, key_dim=8) + FFN expand(32 ReLU)-contract(8) + residual [OK]

Hint: Feed-forward expands then reduces features after attention [OK]

Common Mistakes:

• Swapping Dense layer sizes in feed-forward network
• Using wrong num_heads or key_dim values
• Omitting key argument in MultiHeadAttention