Vector database operations (CRUD) in Prompt Engineering / GenAI - Full Explanation

Introduction

Imagine you have a huge collection of images, texts, or sounds, and you want to find, add, change, or remove items quickly based on their meaning or features. Vector databases help solve this by storing data as points in space, making it easy to search and manage complex information.

Explanation

Create Operation

This is the process of adding new data to the vector database. When you create, the data is first converted into a vector, which is a list of numbers representing its features. Then, this vector is stored in the database so it can be found later.

Create means turning data into vectors and saving them in the database.

Read Operation

Reading involves searching or retrieving data from the vector database. You can ask the database to find vectors similar to a given query vector, which helps find items that are close in meaning or features. This is often called similarity search.

Read means searching for vectors similar to a query to find related data.

Update Operation

Updating changes existing data in the vector database. This means replacing an old vector with a new one that better represents the updated information. The database must keep track of these changes to maintain accurate search results.

Update means replacing old vectors with new ones to reflect changes.

Delete Operation

Deleting removes vectors from the database when the data is no longer needed. This helps keep the database clean and efficient by getting rid of outdated or unwanted information.

Delete means removing vectors to keep the database current and efficient.

Real World Analogy

Imagine a huge library where each book is represented by a unique code based on its content. Adding a book means creating its code and placing it on the shelf. Finding a book means searching for codes similar to the one you have. Changing a book means updating its code, and removing a book means taking its code off the shelf.

Create Operation → Writing a new book's unique code and placing it on the shelf

Read Operation → Looking for books with codes similar to your query code

Update Operation → Changing a book's code when its content is updated

Delete Operation → Removing a book's code and taking it off the shelf

Diagram

Diagram

┌───────────┐      ┌───────────┐      ┌───────────┐      ┌───────────┐
│  Create   │─────▶│   Read    │─────▶│  Update   │─────▶│  Delete   │
└───────────┘      └───────────┘      └───────────┘      └───────────┘
      ▲                                                         │
      └─────────────────────────────────────────────────────────┘

This diagram shows the flow of vector database operations: Create adds data, Read searches data, Update modifies data, and Delete removes data.

Key Facts

Vector → A list of numbers representing the features of data in a vector database.

Similarity Search → Finding vectors in the database that are close to a query vector in meaning or features.

CRUD → The four basic operations to manage data: Create, Read, Update, and Delete.

Indexing → Organizing vectors in the database to make searching faster and more efficient.

Common Confusions

Thinking vectors store the original data like text or images.

Thinking vectors store the original data like text or images. Vectors are numeric representations of data features, not the original data itself.

Believing update means changing the original data directly in the database.

Believing update means changing the original data directly in the database. Update replaces the vector representing the data, not the raw data itself.

Summary

Vector databases store data as numeric vectors to help find similar items quickly.

CRUD operations let you add, search, change, and remove these vectors to manage your data.

Understanding how vectors represent data is key to using vector databases effectively.

Practice

1. What does the CRUD acronym stand for in vector database operations?

easy

A. Connect, Run, Undo, Deploy

B. Compute, Retrieve, Upload, Download

C. Create, Read, Update, Delete

D. Cache, Refresh, Use, Drop

Solution

1. Step 1: Understand CRUD basics

   CRUD is a common term in databases meaning the four basic operations you can do with data.

2. Step 2: Match each letter to its meaning

   C stands for Create (add new data), R for Read (get data), U for Update (change data), and D for Delete (remove data).

3. Final Answer:

   Create, Read, Update, Delete -> Option C

4. Quick Check:

   CRUD = Create, Read, Update, Delete [OK]

Hint: Remember CRUD as basic data actions: add, get, change, remove [OK]

Common Mistakes:

• Confusing CRUD with unrelated terms
• Mixing up the order of operations
• Thinking CRUD only applies to files, not vectors

2. Which of the following is the correct syntax to add a vector with ID 'vec1' and values [0.1, 0.2, 0.3] to a vector database named db?

easy

A. db.push_vector(['vec1', 0.1, 0.2, 0.3])

B. db.insert('vec1', [0.1, 0.2, 0.3])

C. db.create_vector('vec1', 0.1, 0.2, 0.3)

D. db.add_vector('vec1', [0.1, 0.2, 0.3])

Solution

1. Step 1: Identify the common method for adding vectors

   Most vector databases use a method like add_vector with an ID and a list of numbers.

2. Step 2: Check method parameters

   The method should take the vector ID as a string and the vector values as a list or array.

3. Final Answer:

   db.add_vector('vec1', [0.1, 0.2, 0.3]) -> Option D

4. Quick Check:

   Add vector syntax = db.add_vector(id, vector) [OK]

Hint: Add vectors with add_vector(id, vector_list) method [OK]

Common Mistakes:

• Using wrong method names like insert or push_vector
• Passing vector values as separate arguments instead of a list
• Mixing ID and vector in one list

3. Given the following code snippet, what will be the output?

db = VectorDB()
db.add_vector('v1', [1, 0, 0])
db.add_vector('v2', [0, 1, 0])
results = db.search([0.9, 0.1, 0], top_k=1)
print(results)

medium

A. [('v1', 0.9)]

B. [('v2', 0.9)]

C. [('v1', 0.1)]

D. [('v2', 0.1)]

Solution

1. Step 1: Understand the vectors and query

   Vectors 'v1' = [1,0,0], 'v2' = [0,1,0], query = [0.9,0.1,0].

2. Step 2: Calculate similarity or distance

   Assuming cosine similarity, 'v1' is closer to query (dot product ~0.9), 'v2' is less similar (~0.1).

3. Final Answer:

   [('v1', 0.9)] -> Option A

4. Quick Check:

   Closest vector = v1 with similarity 0.9 [OK]

Hint: Closest vector has highest dot product with query [OK]

Common Mistakes:

• Confusing similarity with distance
• Mixing up vector IDs in output
• Assuming lower score means closer

4. The following code tries to update a vector but throws an error. What is the likely cause?

db = VectorDB()
db.add_vector('v1', [0.5, 0.5, 0.5])
db.update_vector('v2', [0.1, 0.1, 0.1])

medium

A. Vector 'v2' does not exist, so update fails

B. The update_vector method requires 4 arguments

C. Vector values must be integers, not floats

D. The add_vector method was not called before update_vector

Solution

1. Step 1: Check vector existence before update

   Updating a vector requires it to exist in the database first.

2. Step 2: Identify the error cause

   Since 'v2' was never added, trying to update it causes an error.

3. Final Answer:

   Vector 'v2' does not exist, so update fails -> Option A

4. Quick Check:

   Update needs existing vector [OK]

Hint: Update only existing vectors, else error occurs [OK]

Common Mistakes:

• Assuming update creates new vectors
• Thinking data type mismatch causes error
• Ignoring vector existence check

5. You want to delete vectors with similarity less than 0.5 to a query vector [0, 1, 0] from your vector database. Which sequence of operations correctly achieves this?

hard

A. Delete all vectors, then add only those with similarity >= 0.5

B. Search vectors with similarity < 0.5, then delete each by ID

C. Update vectors with similarity < 0.5 to zero vectors

D. Add new vectors with similarity >= 0.5, ignoring deletion

Solution

1. Step 1: Find vectors below similarity threshold

   Use a search or filter operation to get IDs of vectors with similarity less than 0.5.

2. Step 2: Delete vectors by their IDs

   Use the delete operation on each vector ID found to remove them from the database.

3. Final Answer:

   Search vectors with similarity < 0.5, then delete each by ID -> Option B

4. Quick Check:

   Filter then delete unwanted vectors [OK]

Hint: Filter vectors first, then delete by ID [OK]

Common Mistakes:

• Deleting all vectors instead of selective ones
• Trying to update instead of delete
• Ignoring the similarity filter step