Transformer modeling in Simulink - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

Understanding the Attention Mechanism in Transformers

Which statement best describes the role of the attention mechanism in a Transformer model?

A It reduces the size of the input data by compressing it into a fixed-length vector.

B It applies convolutional filters to extract local features from the input sequence.

C It normalizes the input data to have zero mean and unit variance before processing.

D It allows the model to focus on different parts of the input sequence when producing each output element.

💡 Hint

Think about how the model decides which words to pay attention to when translating a sentence.

The attention mechanism helps the Transformer model weigh the importance of different input tokens dynamically for each output token, enabling better context understanding.

❓ Predict Output

intermediate

Output Shape of Transformer Encoder Layer

Given an input tensor of shape (batch_size=4, sequence_length=10, embedding_dim=64) passed through a Transformer encoder layer with the same embedding dimension, what will be the shape of the output tensor?

Simulink

input_shape = (4, 10, 64)
# Transformer encoder layer with embedding_dim=64
output_shape = (4, 10, 64)

A (10, 4, 64)

B (4, 64, 10)

C (4, 10, 64)

D (4, 10)

💡 Hint

The Transformer encoder preserves the sequence length and embedding dimension in its output.

The Transformer encoder layer outputs a tensor with the same batch size, sequence length, and embedding dimension as the input.

❓ Hyperparameter

advanced

Choosing the Number of Attention Heads

In a Transformer model, if the embedding dimension is 128, which choice of number of attention heads is valid and why?

A 5, because it is the default number in most libraries.

B 8, because 128 divided by 8 equals 16, which is an integer dimension per head.

C 10, because more heads always improve performance regardless of embedding size.

D 7, because prime numbers improve model generalization.

💡 Hint

Each attention head processes a slice of the embedding dimension equally.

The embedding dimension must be divisible by the number of heads so each head gets an equal-sized slice. 128 divided by 8 is 16, which is valid.

❓ Metrics

advanced

Evaluating Transformer Model Performance

Which metric is most appropriate to evaluate a Transformer model trained for a multi-class text classification task?

A Accuracy, because it measures the proportion of correctly predicted classes.

B Mean Squared Error, because it measures the average squared difference between predicted and true values.

C BLEU score, because it evaluates the quality of generated text sequences.

D Perplexity, because it measures the uncertainty of a language model.

💡 Hint

Think about a task where the model picks one class label from many possible classes.

Accuracy is suitable for classification tasks as it shows how many predictions match the true labels.

🔧 Debug

expert

Identifying the Cause of Training Instability in a Transformer

A Transformer model training suddenly diverges with loss becoming NaN after a few epochs. Which of the following is the most likely cause?

A The learning rate is too high, causing gradient explosion.

B The batch size is too small, causing underfitting.

C The embedding dimension is too low, causing poor representation.

D The number of attention heads is too large, causing overfitting.

💡 Hint

Consider what causes gradients to become unstable during training.

A very high learning rate can cause gradients to explode, making loss values become NaN and training unstable.

Practice

1. What is the main purpose of the attention mechanism in a Transformer model in Simulink?

easy

A. To focus on important parts of the input sequence

B. To increase the size of the input data

C. To reduce the number of layers in the model

D. To convert sequence data into images

Solution

1. Step 1: Understand the role of attention in Transformers

   The attention mechanism helps the model decide which parts of the input sequence are most important for making predictions.

2. Step 2: Relate attention to sequence processing in Simulink

   In Simulink, attention blocks highlight key sequence elements, improving model focus and accuracy.

3. Final Answer:

   To focus on important parts of the input sequence -> Option A

4. Quick Check:

   Attention = Focus on important parts [OK]

Hint: Attention means focusing on key input parts [OK]

Common Mistakes:

• Thinking attention changes input size
• Confusing attention with layer count
• Assuming attention converts data types

2. Which Simulink block is typically used to implement the multi-head attention mechanism in a Transformer model?

easy

A. Multiport Switch block

B. Multi-Head Attention block

C. MATLAB Function block

D. Sum block

Solution

1. Step 1: Identify the block for multi-head attention

   Simulink provides a dedicated Multi-Head Attention block designed to implement this mechanism efficiently.

2. Step 2: Compare with other blocks

   Sum and Multiport Switch blocks do not perform attention; MATLAB Function block requires custom code.

3. Final Answer:

   Multi-Head Attention block -> Option B

4. Quick Check:

   Multi-Head Attention block = Correct block [OK]

Hint: Look for block named 'Multi-Head Attention' [OK]

Common Mistakes:

• Choosing generic blocks like Sum or Switch
• Assuming MATLAB Function block is default for attention
• Confusing block names

3. Given this Simulink Transformer model snippet, what will be the output shape after the multi-head attention block if the input sequence has shape (batch=2, length=10, features=64) and the block uses 8 heads each with 8 features?

medium

A. (2, 10, 64)

B. (2, 8, 64)

C. (2, 10, 8)

D. (2, 64, 10)

Solution

1. Step 1: Understand multi-head attention output shape

   The output shape matches the input sequence shape: batch size and sequence length stay the same, features remain 64.

2. Step 2: Confirm feature dimension calculation

   8 heads x 8 features per head = 64 features total, so output features = 64.

3. Final Answer:

   (2, 10, 64) -> Option A

4. Quick Check:

   Output shape = Input shape [OK]

Hint: Output shape matches input shape in multi-head attention [OK]

Common Mistakes:

• Multiplying batch or length by heads
• Confusing feature dimension with head count
• Swapping sequence length and feature dimensions

4. You built a Transformer model in Simulink but get an error: 'Dimension mismatch in matrix multiplication'. What is the most likely cause?

medium

A. Input sequence length is zero

B. Batch size is too large

C. Feature dimensions between layers do not match

D. Using wrong activation function

Solution

1. Step 1: Analyze the error message

   Dimension mismatch in matrix multiplication usually means the number of columns in one matrix does not match the number of rows in the next.

2. Step 2: Relate to Transformer layers in Simulink

   If feature dimensions between connected layers differ, matrix multiplication fails causing this error.

3. Final Answer:

   Feature dimensions between layers do not match -> Option C

4. Quick Check:

   Dimension mismatch = Feature size mismatch [OK]

Hint: Check feature sizes between connected blocks [OK]

Common Mistakes:

• Blaming batch size for dimension errors
• Ignoring feature dimension compatibility
• Assuming activation function causes dimension errors

5. You want to improve your Transformer model in Simulink to better predict sequences with long-range dependencies. Which change is most effective?

hard

A. Use fewer Transformer layers

B. Reduce the sequence length input

C. Replace attention with convolution blocks

D. Increase the number of attention heads

Solution

1. Step 1: Understand long-range dependency handling

   More attention heads allow the model to focus on multiple parts of the sequence simultaneously, capturing long-range relationships better.

2. Step 2: Evaluate other options

   Reducing sequence length or layers limits model capacity; replacing attention with convolution loses global context.

3. Final Answer:

   Increase the number of attention heads -> Option D

4. Quick Check:

   More heads = Better long-range focus [OK]

Hint: More attention heads capture more sequence details [OK]

Common Mistakes:

• Thinking fewer layers improve performance
• Reducing sequence length helps long-range learning
• Replacing attention with convolution is better