Self-attention and multi-head attention in NLP

Introduction

Self-attention helps a model focus on important parts of a sentence when understanding language. Multi-head attention lets the model look at the sentence from different views at the same time.

When translating a sentence from one language to another.

When summarizing a long article into a short paragraph.

When answering questions based on a paragraph of text.

When recognizing the meaning of words depending on context.

When building chatbots that understand user messages.

Syntax

NLP

Attention(Q, K, V) = softmax((Q * K^T) / sqrt(d_k)) * V

MultiHead(Q, K, V) = Concat(head_1, ..., head_h) * W_O
where head_i = Attention(Q * W_Qi, K * W_Ki, V * W_Vi)

Q, K, V stand for Query, Key, and Value matrices derived from the input.

Multi-head attention runs several attention calculations in parallel, then combines their results.

Examples

This is self-attention where the input attends to itself.

NLP

Q = input_embeddings
K = input_embeddings
V = input_embeddings
output = Attention(Q, K, V)

This shows multi-head attention with two heads looking at the input differently.

NLP

head_1 = Attention(Q * W_Q1, K * W_K1, V * W_V1)
head_2 = Attention(Q * W_Q2, K * W_K2, V * W_V2)
output = Concat(head_1, head_2) * W_O

Sample Model

This code creates a simple self-attention layer with two heads. It takes a small input tensor and computes the self-attention output.

NLP

import torch
import torch.nn as nn
import torch.nn.functional as F

class SelfAttention(nn.Module):
    def __init__(self, embed_size, heads):
        super(SelfAttention, self).__init__()
        self.embed_size = embed_size
        self.heads = heads
        self.head_dim = embed_size // heads

        assert (
            self.head_dim * heads == embed_size
        ), "Embedding size needs to be divisible by heads"

        self.values = nn.Linear(self.head_dim, self.head_dim, bias=False)
        self.keys = nn.Linear(self.head_dim, self.head_dim, bias=False)
        self.queries = nn.Linear(self.head_dim, self.head_dim, bias=False)
        self.fc_out = nn.Linear(heads * self.head_dim, embed_size)

    def forward(self, values, keys, queries):
        N = queries.shape[0]
        value_len, key_len, query_len = values.shape[1], keys.shape[1], queries.shape[1]

        # Split embedding into self.heads pieces
        values = values.reshape(N, value_len, self.heads, self.head_dim)
        keys = keys.reshape(N, key_len, self.heads, self.head_dim)
        queries = queries.reshape(N, query_len, self.heads, self.head_dim)

        values = self.values(values)
        keys = self.keys(keys)
        queries = self.queries(queries)

        # Einsum does batch matrix multiplication for query*keys for each training example
        energy = torch.einsum("nqhd,nkhd->nhqk", [queries, keys])

        # Scale energy
        energy = energy / (self.head_dim ** 0.5)

        attention = torch.softmax(energy, dim=3)

        out = torch.einsum("nhql,nlhd->nqhd", [attention, values]).reshape(
            N, query_len, self.heads * self.head_dim
        )

        out = self.fc_out(out)
        return out

# Example usage
embed_size = 8
heads = 2
self_attention = SelfAttention(embed_size, heads)

# Batch size 1, sequence length 3, embedding size 8
x = torch.tensor([[[1., 0., 1., 0., 1., 0., 1., 0.],
                   [0., 1., 0., 1., 0., 1., 0., 1.],
                   [1., 1., 1., 1., 1., 1., 1., 1.]]])

output = self_attention(x, x, x)
print(output)

Important Notes

Self-attention helps the model understand relationships between words regardless of their position.

Multi-head attention allows the model to capture different types of relationships at once.

Embedding size must be divisible by the number of heads for splitting.

Summary

Self-attention lets a model focus on important words in a sentence by comparing all words to each other.

Multi-head attention runs several self-attention processes in parallel to get richer understanding.

This technique is key in modern language models like Transformers.

Practice

1. What is the main purpose of self-attention in natural language processing?

easy

A. To reduce the size of the input data by removing words

B. To generate random sentences without context

C. To translate text from one language to another

D. To let the model focus on important words by comparing all words to each other

Solution

1. Step 1: Understand self-attention's role

   Self-attention helps the model look at all words in a sentence and decide which ones are important by comparing them to each other.

2. Step 2: Match purpose with options

   To let the model focus on important words by comparing all words to each other correctly describes this focus mechanism, while others describe unrelated tasks.

3. Final Answer:

   To let the model focus on important words by comparing all words to each other -> Option D

4. Quick Check:

   Self-attention = focus on important words [OK]

Hint: Self-attention means comparing words to find importance [OK]

Common Mistakes:

• Confusing self-attention with translation
• Thinking self-attention removes words
• Assuming it generates random text

2. Which of the following is the correct way to describe multi-head attention?

easy

A. Running several self-attention processes in parallel to get richer understanding

B. Applying self-attention only once on the input

C. Using attention only on the first word of a sentence

D. Ignoring word relationships and focusing on word order only

Solution

1. Step 1: Recall multi-head attention definition

   Multi-head attention means running multiple self-attention operations at the same time to capture different aspects of word relationships.

2. Step 2: Compare options to definition

   Running several self-attention processes in parallel to get richer understanding matches this exactly; others describe incomplete or incorrect ideas.

3. Final Answer:

   Running several self-attention processes in parallel to get richer understanding -> Option A

4. Quick Check:

   Multi-head attention = multiple self-attentions [OK]

Hint: Multi-head means many self-attentions at once [OK]

Common Mistakes:

• Thinking multi-head means single attention
• Believing it focuses only on first word
• Ignoring word relationships

3. Given the following simplified self-attention scores matrix for a 3-word sentence:
Scores = [[1, 0.5, 0], [0.5, 1, 0.2], [0, 0.2, 1]]
What is the attention weight for the second word attending to the third word after applying softmax on its row?

medium

A. Approximately 0.21

B. Approximately 0.50

C. Approximately 0.29

D. Approximately 0.70

Solution

1. Step 1: Extract the second row scores

   The second word's scores are [0.5, 1, 0.2].

2. Step 2: Apply softmax to these scores

   Softmax formula: exp(score) / sum(exp(all scores)). Calculate exp(0.5)=1.65, exp(1)=2.72, exp(0.2)=1.22. Sum = 1.65+2.72+1.22=5.59. Attention weight for third word = 1.22/5.59 ≈ 0.218.

3. Final Answer:

   Approximately 0.21 -> Option A

4. Quick Check:

   Softmax normalizes scores to probabilities [OK]

Hint: Softmax turns scores into probabilities summing to 1 [OK]

Common Mistakes:

• Forgetting to exponentiate scores
• Dividing by wrong sum
• Mixing row and column values

4. Consider this Python code snippet for multi-head attention weights calculation:

import numpy as np

def multi_head_attention(scores_list):
    heads = []
    for scores in scores_list:
        weights = np.exp(scores) / np.sum(np.exp(scores))
        heads.append(weights)
    return np.mean(heads, axis=0)

scores_list = [np.array([1, 0, 2]), np.array([0, 1, 1])]
print(multi_head_attention(scores_list))

What is the main bug in this code?

medium

A. Softmax is applied incorrectly; denominator should sum over exp(scores) per head

B. The function returns mean of weights instead of concatenating heads

C. The code uses np.exp twice causing overflow

D. Scores_list should be a 2D array, not a list of arrays

Solution

1. Step 1: Analyze softmax calculation

   Softmax is correctly applied per head by dividing exp(scores) by sum of exp(scores).

2. Step 2: Check output aggregation

   The function averages the weights from each head, but multi-head attention should concatenate or combine heads differently, not average weights element-wise.

3. Final Answer:

   The function returns mean of weights instead of concatenating heads -> Option B

4. Quick Check:

   Multi-head attention combines heads, not averages weights [OK]

Hint: Multi-head attention concatenates heads, not averages weights [OK]

Common Mistakes:

• Thinking averaging weights is correct
• Confusing softmax denominator
• Assuming input format is wrong

5. You want to improve a Transformer model's ability to understand complex sentences by increasing the number of attention heads from 4 to 8. What is the most likely effect of this change?

hard

A. The model will ignore word order completely

B. The model will run faster but lose accuracy

C. The model can capture more diverse word relationships but may require more computation

D. The model will only focus on the first half of the sentence

Solution

1. Step 1: Understand effect of increasing attention heads

   More heads mean the model can look at different parts of the sentence simultaneously, capturing richer relationships.

2. Step 2: Consider computational cost and accuracy

   Increasing heads usually increases computation and memory needs but can improve understanding and accuracy.

3. Final Answer:

   The model can capture more diverse word relationships but may require more computation -> Option C

4. Quick Check:

   More heads = richer focus + more compute [OK]

Hint: More heads = better focus but slower model [OK]

Common Mistakes:

• Assuming more heads always make model faster
• Thinking word order is ignored
• Believing model focuses only on part of sentence