Embedding generation in Prompt Engineering / GenAI - Model Pipeline Trace

Model Pipeline - Embedding generation

This pipeline converts text data into numerical vectors called embeddings. These embeddings capture the meaning of the text in a way that machines can understand and use for tasks like search or recommendation.

Data Flow - 5 Stages

1Raw Text Input

1000 rows x 1 column→Receive raw text sentences→1000 rows x 1 column

"I love sunny days"

↓

2Text Preprocessing

1000 rows x 1 column→Lowercase and remove punctuation→1000 rows x 1 column

"i love sunny days"

↓

3Tokenization

1000 rows x 1 column→Split sentences into words (tokens)→1000 rows x variable tokens

["i", "love", "sunny", "days"]

↓

4Embedding Lookup

1000 rows x variable tokens→Convert tokens to fixed-size vectors→1000 rows x tokens x 50 dimensions

[[0.12, -0.05, ..., 0.33], [0.45, 0.10, ..., -0.22], ...]

↓

5Pooling

1000 rows x tokens x 50 dimensions→Average token vectors to get sentence embedding→1000 rows x 50 dimensions

[0.23, 0.01, ..., -0.05]

Training Trace - Epoch by Epoch

Loss
1.0 |****
0.8 |****
0.6 |***
0.4 |**
0.2 |*
0.0 +---------
     1 2 3 4 5
     Epochs

Epoch	Loss ↓	Accuracy ↑	Observation
1	0.85	0.40	Model starts learning basic word relationships.
2	0.60	0.55	Embeddings begin to capture semantic similarity.
3	0.45	0.68	Improved representation of sentence meaning.
4	0.35	0.75	Embeddings show better clustering of similar texts.
5	0.28	0.80	Model converges with stable embeddings.

Prediction Trace - 4 Layers

Layer 1: Input Text

Layer 2: Tokenization

Layer 3: Embedding Lookup

Layer 4: Pooling

Model Quiz - 3 Questions

What happens to the text during the 'Text Preprocessing' stage?

A Tokens are averaged

B Text is converted into vectors

C Text is lowercased and punctuation removed

D Model weights are updated

Key Insight

Embedding generation transforms text into meaningful vectors that machines can use. Training improves these vectors so similar texts have similar embeddings, helping many AI tasks.

Practice

1. What is the main purpose of embedding generation in AI?

easy

A. To convert text or items into number vectors for easier comparison

B. To translate text from one language to another

C. To generate random numbers for encryption

D. To create images from text descriptions

Solution

1. Step 1: Understand embedding generation

   Embedding generation transforms text or items into number vectors that computers can process.

2. Step 2: Identify the main purpose

   This transformation helps in comparing meanings and finding similarities between data.

3. Final Answer:

   To convert text or items into number vectors for easier comparison -> Option A

4. Quick Check:

   Embedding = number vectors [OK]

Hint: Embeddings turn words into numbers for comparison [OK]

Common Mistakes:

• Confusing embeddings with translation
• Thinking embeddings generate images
• Believing embeddings create random numbers

2. Which of the following is the correct way to represent an embedding vector in Python?

easy

A. embedding = {0.1, 0.5, 0.3, 0.9}

B. embedding = '0.1, 0.5, 0.3, 0.9'

C. embedding = [0.1, 0.5, 0.3, 0.9]

D. embedding = (0.1 0.5 0.3 0.9)

Solution

1. Step 1: Identify valid Python data structures for vectors

   Embedding vectors are usually lists or arrays of numbers in Python.

2. Step 2: Check each option

   embedding = [0.1, 0.5, 0.3, 0.9] uses a list with commas, which is correct. embedding = '0.1, 0.5, 0.3, 0.9' is a string, C is a set (unordered), and D has invalid syntax.

3. Final Answer:

   embedding = [0.1, 0.5, 0.3, 0.9] -> Option C

4. Quick Check:

   Embedding vector = list of numbers [OK]

Hint: Embedding vectors are lists of numbers in Python [OK]

Common Mistakes:

• Using strings instead of lists
• Using sets which are unordered
• Incorrect tuple syntax without commas

3. Given the following code snippet, what will be the output?

import numpy as np
text_embedding = np.array([0.2, 0.4, 0.6])
query_embedding = np.array([0.1, 0.3, 0.5])
similarity = np.dot(text_embedding, query_embedding)
print(round(similarity, 2))

medium

A. 0.44

B. 0.28

C. 0.36

D. 0.52

Solution

1. Step 1: Calculate the dot product of the two vectors

   Dot product = (0.2*0.1) + (0.4*0.3) + (0.6*0.5) = 0.02 + 0.12 + 0.30 = 0.44

2. Step 2: Round the result to 2 decimal places

   Rounded value = 0.44

3. Final Answer:

   0.44 -> Option A

4. Quick Check:

   Dot product = 0.44 [OK]

Hint: Dot product sums element-wise products [OK]

Common Mistakes:

• Multiplying vectors element-wise without summing
• Rounding before summing
• Confusing dot product with vector length

4. The following code is intended to compute cosine similarity between two embeddings but has an error. What is the error?

import numpy as np
def cosine_similarity(a, b):
    return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))

vec1 = np.array([1, 0, 0])
vec2 = np.array([0, 1, 0])
print(cosine_similarity(vec1, vec2))

medium

A. Division by zero error when vectors are zero

B. No error; code works correctly

C. Using lists instead of numpy arrays

D. Incorrect use of np.dot instead of np.cross

Solution

1. Step 1: Analyze the cosine similarity function

   The function correctly computes dot product divided by product of norms.

2. Step 2: Check the example vectors and output

   Vectors are numpy arrays and non-zero, so no division by zero occurs. The code runs correctly and prints 0.0.

3. Final Answer:

   No error; code works correctly -> Option B

4. Quick Check:

   Cosine similarity code = correct [OK]

Hint: Check for zero vectors to avoid division errors [OK]

Common Mistakes:

• Confusing dot product with cross product
• Forgetting to use numpy arrays
• Not handling zero vectors causing division errors

5. You have a list of product descriptions and want to group similar products using embeddings. Which approach best helps you achieve this?

hard

A. Manually read and group descriptions without embeddings

B. Translate descriptions to another language before clustering

C. Use embeddings only for images, not text

D. Generate embeddings for each description, then use clustering on these vectors

Solution

1. Step 1: Understand the goal of grouping similar products

   Grouping similar products means finding which descriptions are close in meaning.

2. Step 2: Use embeddings and clustering

   Generating embeddings converts descriptions into vectors. Clustering groups vectors close in space, thus grouping similar products.

3. Final Answer:

   Generate embeddings for each description, then use clustering on these vectors -> Option D

4. Quick Check:

   Embedding + clustering = grouping similar items [OK]

Hint: Cluster embedding vectors to group similar items [OK]

Common Mistakes:

• Thinking translation helps grouping
• Assuming embeddings only work for images
• Ignoring embeddings and grouping manually