Vector similarity metrics in Prompt Engineering / GenAI - Deep Dive

Overview - Vector similarity metrics

What is it?

Vector similarity metrics are ways to measure how alike two lists of numbers are. These lists, called vectors, represent things like words, images, or sounds in a way a computer can understand. By comparing vectors, we can find out if two things are similar or different. This helps computers make decisions like finding similar pictures or understanding language.

Why it matters

Without vector similarity metrics, computers would struggle to compare complex data like images or text. These metrics let machines find patterns and connections in data, making technologies like search engines, recommendation systems, and voice assistants work well. Without them, many smart applications would be slow, inaccurate, or impossible.

Where it fits

Before learning vector similarity metrics, you should understand what vectors are and how data can be represented as numbers. After this, you can learn about machine learning models that use these metrics to find patterns or make predictions, like clustering or nearest neighbor search.

Mental Model

Core Idea

Vector similarity metrics measure how close or aligned two sets of numbers are to tell how alike the things they represent are.

Think of it like...

It's like comparing two arrows on a map: if they point in the same direction and have similar length, they represent similar things; if they point differently or have different lengths, they are less alike.

Vectors A and B:
  A → (x1, y1, z1)
  B → (x2, y2, z2)

Similarity measures:
  ┌───────────────┐
  │ Cosine Similarity │ Measures angle between A and B
  │ Euclidean Distance │ Measures straight-line distance
  │ Manhattan Distance │ Measures grid-like path distance
  │ Jaccard Similarity │ Measures overlap in sets
  └───────────────┘

Build-Up - 7 Steps

1

Foundation

Understanding vectors as number lists

Concept: Vectors are lists of numbers that represent data points in space.

Imagine a point in 2D space like (3, 4). This point can be written as a vector [3, 4]. In machine learning, data like words or images are turned into vectors with many numbers. Each number captures some feature or detail about the data.

Result

You can represent complex data as simple lists of numbers called vectors.

Understanding vectors as number lists is the base for comparing data mathematically.

2

Foundation

Why compare vectors? Measuring similarity

Concept: We compare vectors to find out how alike the things they represent are.

If two vectors are close or point in the same direction, their data is similar. For example, two pictures of cats will have vectors that are close, while a cat and a car will be far apart. We need ways to measure this closeness or similarity.

Result

We see the need for metrics that tell us how close or similar two vectors are.

Knowing why we compare vectors helps us choose the right similarity metric.

3

Intermediate

Cosine similarity: angle between vectors

🤔Before reading on: do you think cosine similarity cares about vector length or just direction? Commit to your answer.

Concept: Cosine similarity measures the angle between two vectors, ignoring their length.

Cosine similarity calculates the cosine of the angle between two vectors. If the angle is 0 degrees (vectors point the same way), similarity is 1 (most similar). If they are at 90 degrees (perpendicular), similarity is 0 (no similarity). It is calculated as (A·B) / (||A|| * ||B||), where · is dot product and || || is length.

Result

Vectors pointing in the same direction have cosine similarity close to 1, even if their lengths differ.

Cosine similarity focuses on direction, making it great for text or data where magnitude varies but pattern matters.

4

Intermediate

Euclidean distance: straight-line gap

🤔Before reading on: does a smaller Euclidean distance mean more or less similarity? Commit to your answer.

Concept: Euclidean distance measures the straight-line distance between two vectors in space.

Euclidean distance is like measuring with a ruler between two points. For vectors A and B, it is the square root of the sum of squared differences of each coordinate: sqrt((x1-x2)^2 + (y1-y2)^2 + ...). Smaller distance means vectors are closer and more similar.

Result

Vectors close in space have small Euclidean distance, indicating similarity.

Euclidean distance captures absolute closeness, useful when magnitude and scale matter.

5

Intermediate

Manhattan distance: grid-like path length

Concept: Manhattan distance sums absolute differences along each dimension, like walking city blocks.

Instead of straight-line, Manhattan distance measures how far you'd travel if you could only move along grid lines. For vectors A and B, it is sum of |x1-x2| + |y1-y2| + ... . It can be more robust to outliers in some cases.

Result

Manhattan distance gives a different sense of closeness, useful in certain data shapes.

Knowing different distance types helps pick the best metric for your data shape.

6

Advanced

Jaccard similarity for sets and sparse vectors

🤔Before reading on: do you think Jaccard similarity works well with numeric vectors or sets? Commit to your answer.

Concept: Jaccard similarity measures overlap between sets, useful for sparse or binary data.

Jaccard similarity is the size of intersection divided by size of union of two sets. For example, if two documents share many words, their Jaccard similarity is high. It can be applied to vectors by treating non-zero elements as set members.

Result

Jaccard similarity helps compare data where presence or absence matters more than magnitude.

Understanding Jaccard opens doors to comparing sparse data like text or user preferences.

7

Expert

Choosing and combining metrics in production

🤔Before reading on: do you think one similarity metric fits all data types perfectly? Commit to your answer.

Concept: Real-world systems often combine or select metrics based on data type, scale, and task needs.

In practice, cosine similarity is popular for text embeddings, Euclidean for images, and Jaccard for sets. Sometimes, systems combine metrics or normalize data first. Choosing the right metric affects accuracy and speed. Also, approximate methods speed up similarity search in large datasets.

Result

Effective similarity measurement requires understanding data and task, not just applying one metric blindly.

Knowing metric strengths and tradeoffs is key to building robust, efficient AI systems.

Under the Hood

Vector similarity metrics work by applying mathematical formulas to the numbers in vectors. For cosine similarity, the dot product and vector lengths are computed to find the angle. For Euclidean and Manhattan distances, coordinate differences are calculated and combined. Internally, these operations use fast linear algebra routines optimized for speed and memory. Sparse data is handled by ignoring zero entries to save computation.

Why designed this way?

These metrics were designed to capture different notions of similarity: direction (cosine) for pattern matching, absolute distance (Euclidean) for closeness, and overlap (Jaccard) for shared features. Alternatives like correlation or Hamming distance exist but are less general. The chosen metrics balance mathematical simplicity, interpretability, and computational efficiency.

Input vectors A and B
  │
  ├─> Compute dot product (A·B)
  ├─> Compute lengths ||A|| and ||B||
  ├─> Calculate cosine similarity = (A·B) / (||A|| * ||B||)
  │
  ├─> Compute coordinate differences
  │     ├─> Euclidean: sqrt(sum of squares)
  │     └─> Manhattan: sum of absolutes
  │
  └─> For sets: find intersection and union sizes
        └─> Jaccard similarity = intersection / union

Myth Busters - 4 Common Misconceptions

Quick: Does cosine similarity consider vector length when measuring similarity? Commit to yes or no.

Common Belief: Cosine similarity measures how close two vectors are in space, including their length.

Reality: Cosine similarity only measures the angle between vectors, ignoring their length.

Why it matters: Confusing this leads to wrong similarity judgments, especially when vector magnitude carries important meaning.

Quick: Is a smaller Euclidean distance always better for similarity? Commit to yes or no.

Common Belief: Smaller Euclidean distance always means more similarity regardless of data context.

Reality: Euclidean distance can be misleading if data is not normalized or has different scales across dimensions.

Why it matters: Ignoring scale differences can cause wrong nearest neighbor matches and poor model performance.

Quick: Can Jaccard similarity be used directly on numeric vectors? Commit to yes or no.

Common Belief: Jaccard similarity works well on any numeric vectors like cosine or Euclidean.

Reality: Jaccard similarity is designed for sets or binary data, not raw numeric vectors.

Why it matters: Using Jaccard on numeric vectors without conversion leads to meaningless similarity scores.

Quick: Does one similarity metric fit all data types perfectly? Commit to yes or no.

Common Belief: One similarity metric can be used for all types of data and tasks.

Reality: Different data types and tasks require different similarity metrics for best results.

Why it matters: Using the wrong metric reduces accuracy and efficiency in real-world applications.

Expert Zone

1

Cosine similarity is sensitive to zero vectors and requires careful handling to avoid division by zero.

2

Euclidean distance can be dominated by dimensions with large scale unless data is normalized or weighted.

3

Approximate nearest neighbor algorithms often rely on specific similarity metrics for speed, limiting metric choice.

When NOT to use

Avoid cosine similarity when vector magnitude matters, such as in physical measurements. Euclidean distance is not ideal for high-dimensional sparse data due to the curse of dimensionality; use cosine or Jaccard instead. For categorical or binary data, prefer Jaccard or Hamming distance over numeric metrics.

Production Patterns

In production, vector similarity is used in search engines to find relevant documents, in recommendation systems to suggest similar items, and in clustering algorithms to group alike data. Systems often preprocess vectors by normalization or dimensionality reduction before similarity calculation. Approximate methods like locality-sensitive hashing speed up large-scale similarity search.

Connections

Nearest Neighbor Search

Vector similarity metrics are the core calculations used to find nearest neighbors in data.

Understanding similarity metrics helps grasp how nearest neighbor algorithms decide which data points are closest.

Cosine of Angle in Trigonometry

Cosine similarity directly uses the cosine function from trigonometry to measure vector alignment.

Knowing trigonometry basics clarifies why cosine similarity measures direction, not magnitude.

Set Theory

Jaccard similarity is based on set intersection and union concepts from set theory.

Understanding sets and their operations helps explain why Jaccard similarity measures shared features.

Common Pitfalls

#1 Using cosine similarity without normalizing vectors, causing division by zero errors.

Wrong approach:

def cosine_similarity(a, b):
return sum(x*y for x,y in zip(a,b)) / (len(a) * len(b)) # wrong length calculation

Correct approach:

import math

def cosine_similarity(a, b):
dot = sum(x*y for x,y in zip(a,b))
norm_a = math.sqrt(sum(x*x for x in a))
norm_b = math.sqrt(sum(y*y for y in b))
if norm_a == 0 or norm_b == 0:
return 0.0
return dot / (norm_a * norm_b)

Root cause:

Misunderstanding that vector length means Euclidean norm, not just number of elements.

#2 Applying Euclidean distance directly on unscaled data with mixed units.

Wrong approach:

def euclidean_distance(a, b):
return sum((x - y)**2 for x,y in zip(a,b))**0.5

# Using raw data with different scales

Correct approach:

def normalize(v):
min_v = min(v)
max_v = max(v)
return [(x - min_v) / (max_v - min_v) for x in v]

# Normalize before distance
na = normalize(a)
nb = normalize(b)

def euclidean_distance(a, b):
return sum((x - y)**2 for x,y in zip(a,b))**0.5

Root cause:

Ignoring that Euclidean distance is sensitive to scale differences across dimensions.

#3 Using Jaccard similarity on raw numeric vectors without converting to sets.

Wrong approach:

def jaccard_similarity(a, b):
intersection = sum(min(x,y) for x,y in zip(a,b))
union = sum(max(x,y) for x,y in zip(a,b))
return intersection / union

Correct approach:

def jaccard_similarity_sets(a, b):
set_a = set(i for i,x in enumerate(a) if x != 0)
set_b = set(i for i,x in enumerate(b) if x != 0)
intersection = len(set_a & set_b)
union = len(set_a | set_b)
if union == 0:
return 0.0
return intersection / union

Root cause:

Confusing numeric vector similarity with set-based similarity.

Key Takeaways

Vector similarity metrics let us measure how alike two pieces of data are by comparing their numeric representations.

Different metrics capture different ideas of similarity: direction, distance, or shared features.

Choosing the right metric depends on the data type and the problem you want to solve.

Understanding the math behind these metrics helps avoid common mistakes and improves model accuracy.

In real-world AI systems, combining and tuning similarity metrics is key to fast and reliable results.