Transformer basics in TensorFlow - Interactive Code Practice

Practice - 5 Tasks

1fill in blank

easy

Complete the code to create a simple Transformer encoder layer using TensorFlow.

TensorFlow

import tensorflow as tf

encoder_layer = tf.keras.layers.TransformerEncoderLayer(
    num_heads=8,
    key_dim=64,
    dropout=[1]
)

Drag options to blanks, or click blank then click option'

A1.0

B-0.1

C10

D0.1

Attempts:

3 left

💡 Hint

Common Mistakes

Using dropout values greater than 1 or negative values.

Confusing dropout with number of heads.

The dropout rate should be a small positive float like 0.1 to prevent overfitting.

2fill in blank

medium

Complete the code to apply multi-head attention in TensorFlow.

TensorFlow

mha = tf.keras.layers.MultiHeadAttention(num_heads=4, key_dim=32)
query = tf.random.uniform((1, 60, 32))
value = tf.random.uniform((1, 60, 32))
output = mha(query=[1], value=value, key=value)

Drag options to blanks, or click blank then click option'

Aquery

Bvalue

Coutput

DNone

Attempts:

3 left

💡 Hint

Common Mistakes

Passing value as query instead of query tensor.

Passing output as query which is undefined yet.

The query argument should be the query tensor, here named 'query'.

3fill in blank

hard

Fix the error in the Transformer model definition by filling the correct activation function.

TensorFlow

from tensorflow.keras import layers

class SimpleTransformer(tf.keras.Model):
    def __init__(self):
        super().__init__()
        self.dense = layers.Dense(64, activation=[1])

    def call(self, inputs):
        return self.dense(inputs)

Drag options to blanks, or click blank then click option'

A'sigmoid'

B'softmax'

C'relu'

D'linear'

Attempts:

3 left

💡 Hint

Common Mistakes

Using softmax activation in dense layer causing wrong output shape.

Using linear activation which disables nonlinearity.

ReLU is commonly used as activation in Transformer feed-forward layers.

4fill in blank

hard

Fill both blanks to create a positional encoding matrix for Transformer input.

TensorFlow

import numpy as np

def positional_encoding(position, d_model):
    angle_rads = np.arange(position)[:, np.newaxis] / np.power(10000, (2 * (np.arange(d_model)[1])) / d_model)
    pos_encoding = np.zeros((position, d_model))
    pos_encoding[:, 0::2] = np.sin(angle_rads[:, 0::2])
    pos_encoding[:, 1::2] = np.cos(angle_rads[:, [2]])
    return pos_encoding

Drag options to blanks, or click blank then click option'

A% 2

B+ 2

C- 2

D* 2

Attempts:

3 left

💡 Hint

Common Mistakes

Using addition or multiplication instead of modulo causing indexing errors.

Using the same index slice for sine and cosine.

The modulo 2 operation selects even and odd indices for sine and cosine respectively.

5fill in blank

hard

Fill all three blanks to build a Transformer encoder block with normalization and dropout.

TensorFlow

from tensorflow.keras import layers

class EncoderBlock(layers.Layer):
    def __init__(self, d_model, num_heads, dff, rate=0.1):
        super().__init__()
        self.mha = layers.MultiHeadAttention(num_heads=[1], key_dim=d_model)
        self.ffn = tf.keras.Sequential([
            layers.Dense(dff, activation=[2]),
            layers.Dense(d_model)
        ])
        self.layernorm1 = layers.LayerNormalization(epsilon=1e-6)
        self.dropout1 = layers.Dropout(rate=[3])

    def call(self, x, training):
        attn_output = self.mha(x, x)
        attn_output = self.dropout1(attn_output, training=training)
        out1 = self.layernorm1(x + attn_output)
        ffn_output = self.ffn(out1)
        return ffn_output

Drag options to blanks, or click blank then click option'

A8

Brelu

C0.1

Dsoftmax

Attempts:

3 left

💡 Hint

Common Mistakes

Using softmax as activation in feed-forward network.

Setting dropout rate too high or zero.

Using wrong number of heads causing shape mismatch.

Number of heads is 8, activation is relu, and dropout rate is 0.1 for typical Transformer encoder.

Practice

1. What is the main purpose of the attention mechanism in a Transformer model?

easy

A. To reduce the number of layers

B. To increase the size of the model

C. To focus on important parts of the input sequence

D. To generate random outputs

Solution

1. Step 1: Understand attention in Transformers

   The attention mechanism helps the model look at important words or parts in the input sequence.

2. Step 2: Identify the main goal of attention

   Its goal is to give more weight to relevant parts, improving understanding of context.

3. Final Answer:

   To focus on important parts of the input sequence -> Option C

4. Quick Check:

   Attention = Focus on important parts [OK]

Hint: Attention means focusing on key input parts [OK]

Common Mistakes:

• Thinking attention increases model size
• Confusing attention with layer count
• Believing attention creates random outputs

2. Which of the following is the correct way to import the MultiHeadAttention layer in TensorFlow?

easy

A. from tensorflow.keras.layers import MultiHeadAttention

B. import MultiHeadAttention from tensorflow.layers

C. from tensorflow import MultiHeadAttention

D. import MultiHeadAttention from keras.layers

Solution

1. Step 1: Recall TensorFlow Keras layer imports

   TensorFlow's Keras layers are imported from tensorflow.keras.layers module.

2. Step 2: Check correct import syntax

   The correct syntax is: from tensorflow.keras.layers import MultiHeadAttention.

3. Final Answer:

   from tensorflow.keras.layers import MultiHeadAttention -> Option A

4. Quick Check:

   Correct import = from tensorflow.keras.layers import MultiHeadAttention [OK]

Hint: Use tensorflow.keras.layers for Keras layers [OK]

Common Mistakes:

• Using incorrect import paths
• Confusing tensorflow and keras imports
• Using wrong import syntax

3. Given the following TensorFlow code snippet, what is the shape of the output tensor?

import tensorflow as tf
mha = tf.keras.layers.MultiHeadAttention(num_heads=2, key_dim=4)
query = tf.random.uniform((1, 3, 8))
value = tf.random.uniform((1, 4, 8))
output = mha(query=query, value=value)

medium

A. (1, 3, 4)

B. (1, 4, 8)

C. (3, 1, 8)

D. (1, 3, 8)

Solution

1. Step 1: Understand MultiHeadAttention input shapes

   Query shape is (batch=1, query_len=3, features=8), value shape is (1, 4, 8).

2. Step 2: Determine output shape

   Output shape matches query length and feature dimension: (1, 3, 8).

3. Final Answer:

   (1, 3, 8) -> Option D

4. Quick Check:

   Output shape = query shape except last dim [OK]

Hint: Output shape matches query's first two dims [OK]

Common Mistakes:

• Confusing value length with output length
• Mixing batch and sequence dimensions
• Assuming output feature dims change

4. Identify the error in this Transformer block code snippet:

import tensorflow as tf
class SimpleTransformer(tf.keras.layers.Layer):
    def __init__(self):
        super().__init__()
        self.mha = tf.keras.layers.MultiHeadAttention(num_heads=2, key_dim=4)
    def call(self, x):
        attn_output = self.mha(query=x)
        return attn_output + x

medium

A. MultiHeadAttention requires 'value' argument

B. The addition attn_output + x causes shape mismatch

C. The call method should return only attn_output

D. Missing positional encoding before attention

Solution

1. Step 1: Check MultiHeadAttention required arguments

   MultiHeadAttention needs query and value arguments; value is missing here.

2. Step 2: Understand why value is needed

   Without value, the layer cannot compute attention properly, causing an error.

3. Final Answer:

   MultiHeadAttention requires 'value' argument -> Option A

4. Quick Check:

   Missing value argument = MultiHeadAttention requires 'value' argument [OK]

Hint: Always provide query and value arguments to MultiHeadAttention [OK]

Common Mistakes:

• Assuming value defaults when only query is provided
• Ignoring required arguments in call
• Thinking positional encoding is mandatory here

5. You want to build a Transformer block in TensorFlow that includes MultiHeadAttention followed by a feed-forward network with ReLU activation. Which code snippet correctly implements this?

hard

A. class TransformerBlock(tf.keras.layers.Layer):
def __init__(self):
super().__init__()
self.mha = tf.keras.layers.MultiHeadAttention(num_heads=4, key_dim=8)
self.ffn = tf.keras.Sequential([
tf.keras.layers.Dense(8, activation='relu'),
tf.keras.layers.Dense(32)
])
def call(self, x):
attn_output = self.mha(query=x, key=x, value=x)
ffn_output = self.ffn(attn_output)
return ffn_output + x

B. class TransformerBlock(tf.keras.layers.Layer):
def __init__(self):
super().__init__()
self.mha = tf.keras.layers.MultiHeadAttention(num_heads=4, key_dim=8)
self.ffn = tf.keras.Sequential([
tf.keras.layers.Dense(32, activation='relu'),
tf.keras.layers.Dense(8)
])
def call(self, x):
attn_output = self.mha(query=x, key=x, value=x)
ffn_output = self.ffn(attn_output)
return ffn_output + x

C. class TransformerBlock(tf.keras.layers.Layer):
def __init__(self):
super().__init__()
self.mha = tf.keras.layers.MultiHeadAttention(num_heads=8, key_dim=4)
self.ffn = tf.keras.Sequential([
tf.keras.layers.Dense(8, activation='relu'),
tf.keras.layers.Dense(32)
])
def call(self, x):
attn_output = self.mha(query=x, key=x, value=x)
ffn_output = self.ffn(attn_output)
return ffn_output + x

D. class TransformerBlock(tf.keras.layers.Layer):
def __init__(self):
super().__init__()
self.mha = tf.keras.layers.MultiHeadAttention(num_heads=2, key_dim=8)
self.ffn = tf.keras.Sequential([
tf.keras.layers.Dense(8, activation='relu'),
tf.keras.layers.Dense(32)
])
def call(self, x):
attn_output = self.mha(query=x, key=x, value=x)
ffn_output = self.ffn(attn_output)
return ffn_output + x

Solution

1. Step 1: Check MultiHeadAttention parameters

   num_heads=4, key_dim=8 produces output_dim=32, valid for Transformer blocks.

2. Step 2: Verify feed-forward network structure

   FFN uses Dense(32, activation='relu') to expand, then Dense(8) to match input dim.

3. Step 3: Confirm call method logic

   Applies self-attention, FFN on output, residual connection (shapes match: 8).

4. Final Answer:

   Code with num_heads=4, key_dim=8, FFN Dense(32, ReLU) then Dense(8) -> Option B

5. Quick Check:

   MHA(4 heads, key_dim=8) + FFN expand(32 ReLU)-contract(8) + residual [OK]

Hint: Feed-forward expands then reduces features after attention [OK]

Common Mistakes:

• Swapping Dense layer sizes in feed-forward network
• Using wrong num_heads or key_dim values
• Omitting key argument in MultiHeadAttention