data/vector_databases.md

Vector Databases for RAG

What are Vector Databases?

Vector databases are specialized databases designed to store and search high-dimensional vectors efficiently. They are essential for RAG systems because they enable semantic search over document embeddings.

Why Use Vector Databases?

Traditional databases use exact matching or keyword search. Vector databases use:

• Similarity Search: Find semantically similar content
• Approximate Nearest Neighbors (ANN): Fast similarity search at scale
• Metadata Filtering: Combine vector search with traditional filters

Popular Vector Databases

FAISS (Facebook AI Similarity Search)

• Type: Library, not a full database
• Deployment: In-memory or on-disk
• Best For: Local development, research, small to medium datasets
• Pros: Fast, free, runs locally
• Cons: No built-in persistence, limited metadata support

Pinecone

• Type: Managed cloud service
• Deployment: Cloud-hosted
• Best For: Production applications, scalable systems
• Pros: Fully managed, auto-scaling, metadata filtering
• Cons: Paid service, cloud dependency

Weaviate

• Type: Open-source vector database
• Deployment: Self-hosted or cloud
• Best For: Production with control over infrastructure
• Pros: GraphQL API, hybrid search, modular
• Cons: More complex setup

Chroma

• Type: Open-source embedding database
• Deployment: Local or server
• Best For: Development and production
• Pros: Simple API, built-in embeddings, free
• Cons: Newer, smaller community

Qdrant

• Type: Open-source vector database
• Deployment: Self-hosted or cloud
• Best For: High-performance applications
• Pros: Fast, filtered search, production-ready
• Cons: Requires setup and management

Key Concepts

Embeddings

Vector representations of text that capture semantic meaning:

• 384 dimensions (MiniLM)
• 768 dimensions (BERT)
• 1536 dimensions (OpenAI ada-002)

Distance Metrics

Measure similarity between vectors:

• Cosine Similarity: Angle between vectors (most common)
• Euclidean Distance: Straight-line distance
• Dot Product: Inner product of vectors

Indexing Methods

Optimize search performance:

• Flat Index: Exact search, slower at scale
• IVF (Inverted File): Partition space into clusters
• HNSW: Hierarchical graph structure (fast and accurate)
• LSH (Locality-Sensitive Hashing): Hash similar items together

FAISS Examples

Basic Usage

from langchain.vectorstores import FAISS
from langchain.embeddings import HuggingFaceEmbeddings

embeddings = HuggingFaceEmbeddings()
vectorstore = FAISS.from_documents(documents, embeddings)

# Search
results = vectorstore.similarity_search("query", k=4)

Save and Load

# Save
vectorstore.save_local("faiss_index")

# Load
vectorstore = FAISS.load_local("faiss_index", embeddings)

With Metadata Filtering

# Add metadata
docs_with_metadata = [
    Document(page_content="...", metadata={"source": "doc1.txt", "date": "2024-01-01"})
]

# Search with filter
results = vectorstore.similarity_search(
    "query",
    k=4,
    filter={"source": "doc1.txt"}
)

Choosing a Vector Database

For Development/Prototyping

• FAISS: Fast local development
• Chroma: Easy setup with persistence

For Production

• Pinecone: Minimal ops, quick to market
• Weaviate: Control and flexibility
• Qdrant: High performance needs

Considerations

1. Scale: Expected data volume and query load
2. Budget: Managed vs self-hosted costs
3. Latency: Response time requirements
4. Features: Metadata filtering, hybrid search, etc.
5. Maintenance: Ops team capacity

Performance Optimization

1. Chunk Size

• Smaller chunks: More precise retrieval
• Larger chunks: More context per result
• Optimal: 500-800 tokens for most use cases

2. Index Configuration

• Use HNSW for production (fast + accurate)
• Tune ef_construction and M parameters
• Consider quantization for large datasets

3. Batch Processing

• Embed documents in batches
• Use parallel processing for large datasets
• Cache embeddings when possible

4. Query Optimization

• Adjust k (number of results) based on use case
• Use MMR (Maximum Marginal Relevance) for diversity
• Implement result reranking for better quality