Transformer modeling in Simulink - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

Understanding the Attention Mechanism in Transformers

Which statement best describes the role of the attention mechanism in a Transformer model?

A It reduces the size of the input data by compressing it into a fixed-length vector.

B It applies convolutional filters to extract local features from the input sequence.

C It normalizes the input data to have zero mean and unit variance before processing.

D It allows the model to focus on different parts of the input sequence when producing each output element.

💡 Hint

Think about how the model decides which words to pay attention to when translating a sentence.

The attention mechanism helps the Transformer model weigh the importance of different input tokens dynamically for each output token, enabling better context understanding.

❓ Predict Output

intermediate

Output Shape of Transformer Encoder Layer

Given an input tensor of shape (batch_size=4, sequence_length=10, embedding_dim=64) passed through a Transformer encoder layer with the same embedding dimension, what will be the shape of the output tensor?

Simulink

input_shape = (4, 10, 64)
# Transformer encoder layer with embedding_dim=64
output_shape = (4, 10, 64)

A (10, 4, 64)

B (4, 64, 10)

C (4, 10, 64)

D (4, 10)

💡 Hint

The Transformer encoder preserves the sequence length and embedding dimension in its output.

The Transformer encoder layer outputs a tensor with the same batch size, sequence length, and embedding dimension as the input.

❓ Hyperparameter

advanced

Choosing the Number of Attention Heads

In a Transformer model, if the embedding dimension is 128, which choice of number of attention heads is valid and why?

A 5, because it is the default number in most libraries.

B 8, because 128 divided by 8 equals 16, which is an integer dimension per head.

C 10, because more heads always improve performance regardless of embedding size.

D 7, because prime numbers improve model generalization.

💡 Hint

Each attention head processes a slice of the embedding dimension equally.

The embedding dimension must be divisible by the number of heads so each head gets an equal-sized slice. 128 divided by 8 is 16, which is valid.

❓ Metrics

advanced

Evaluating Transformer Model Performance

Which metric is most appropriate to evaluate a Transformer model trained for a multi-class text classification task?

A Accuracy, because it measures the proportion of correctly predicted classes.

B Mean Squared Error, because it measures the average squared difference between predicted and true values.

C BLEU score, because it evaluates the quality of generated text sequences.

D Perplexity, because it measures the uncertainty of a language model.

💡 Hint

Think about a task where the model picks one class label from many possible classes.

Accuracy is suitable for classification tasks as it shows how many predictions match the true labels.

🔧 Debug

expert

Identifying the Cause of Training Instability in a Transformer

A Transformer model training suddenly diverges with loss becoming NaN after a few epochs. Which of the following is the most likely cause?

A The learning rate is too high, causing gradient explosion.

B The batch size is too small, causing underfitting.

C The embedding dimension is too low, causing poor representation.

D The number of attention heads is too large, causing overfitting.

💡 Hint

Consider what causes gradients to become unstable during training.

A very high learning rate can cause gradients to explode, making loss values become NaN and training unstable.