Spaces:
Build error
Build error
|
Download docs/explanations/vectorstore.md from AIenthusSP/RAG-Documind-AI: direct link, hf CLI and curl.
- Browser
- Download file 11 kB
-
https://huggingface.co/spaces/AIenthusSP/RAG-Documind-AI/resolve/main/docs/explanations/vectorstore.md
- Command line
-
hf download hf://spaces/AIenthusSP/RAG-Documind-AI/docs/explanations/vectorstore.md
-
curl -L -o vectorstore.md https://huggingface.co/spaces/AIenthusSP/RAG-Documind-AI/resolve/main/docs/explanations/vectorstore.md
11 kB
| # π Deep Dive: ChromaDB Vector Store Architecture | |
| ## π§ Why Vector Stores Matter in RAG | |
| ``` | |
| βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ | |
| β THE RETRIEVAL PROBLEM β | |
| β β | |
| β User Query: "What is the refund policy?" β | |
| β β β | |
| β Need to find relevant documents from 10,000+ chunks β | |
| β β β | |
| β Can't do keyword search (user might say "return" vs "refund") β | |
| β β β | |
| β Solution: SEMANTIC SEARCH using vector embeddings β | |
| βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ | |
| ``` | |
| **Key Insight:** Vector stores enable **semantic similarity** β finding documents by meaning, not just keywords. | |
| --- | |
| ## π How Vector Search Works | |
| ``` | |
| EMBEDDING SPACE (384 dimensions) | |
| "refund policy" βββββββββββββββΊ [0.23, -0.45, 0.12, ...] | |
| β | |
| Find nearest neighbors | |
| β | |
| βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ | |
| β Stored Vectors: β | |
| β β | |
| β "Returns are processed in 7 days" β [0.21, -0.42, ...] β β CLOSE! | |
| β "Contact support for help" β [0.89, 0.12, ...] β β FAR | |
| β "Money back guarantee applies" β [0.24, -0.41, ...] β β CLOSE! | |
| βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ | |
| ``` | |
| **ChromaDB** is the database that stores these vectors and performs fast similarity search. | |
| --- | |
| ## ποΈ VectorStore Class Architecture | |
| ``` | |
| βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ | |
| β VectorStore β | |
| β β | |
| β βββββββββββββββββββ ββββββββββββββββββββ β | |
| β β embedding_fn β β ChromaDB β β | |
| β β (HuggingFace) βββββΊβ _store β β | |
| β βββββββββββββββββββ ββββββββββ¬ββββββββββ β | |
| β β β | |
| β ββββββββββββββββββββββββββΌβββββββββββββββββββββββββ β | |
| β βΌ βΌ βΌ β | |
| β βββββββββββββββ βββββββββββββββ ββββββββββββ β | |
| β βadd_documentsβ βsimilarity_ β βas_ β β | |
| β β β βsearch β βretriever β β | |
| β βββββββββββββββ βββββββββββββββ ββββββββββββ β | |
| βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ | |
| β | |
| βΌ | |
| ββββββββββββββββββββββββββ | |
| β Persistent Storage β | |
| β (chroma_db/ folder) β | |
| ββββββββββββββββββββββββββ | |
| ``` | |
| --- | |
| ## π― Constructor (`__init__`) | |
| ```python | |
| def __init__( | |
| self, | |
| embedding_function: Embeddings, # How to convert text β vectors | |
| collection_name: str = "documents", # Namespace for documents | |
| persist_directory: Optional[str] = None, # Where to save | |
| ): | |
| ``` | |
| ### Why These Parameters? | |
| | Parameter | Purpose | | |
| |-----------|---------| | |
| | `embedding_function` | **Dependency Injection** β allows swapping embedding models | | |
| | `collection_name` | **Multi-tenancy** β different projects in same DB | | |
| | `persist_directory` | **Durability** β survives application restarts | | |
| ### How ChromaDB Initializes: | |
| ```python | |
| self._store = Chroma( | |
| collection_name=self.collection_name, | |
| embedding_function=self.embedding_function, # Used automatically | |
| persist_directory=str(self.persist_directory), | |
| ) | |
| ``` | |
| ChromaDB automatically: | |
| 1. Creates the collection if it doesn't exist | |
| 2. Loads existing data if it does exist | |
| 3. Uses the embedding function for all operations | |
| --- | |
| ## π₯ The `add_documents` Method | |
| ```python | |
| def add_documents( | |
| self, | |
| documents: List[Document], | |
| batch_size: int = 100 # Process 100 at a time | |
| ) -> List[str]: | |
| ``` | |
| ### Why Batching? | |
| ``` | |
| β WITHOUT BATCHING: | |
| βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ | |
| β 10,000 documents β All loaded in memory β OOM ERROR! β | |
| βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ | |
| β WITH BATCHING: | |
| βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ | |
| β 10,000 documents β | |
| β β β | |
| β Batch 1: docs[0:100] β Embed β Store β Free memory β | |
| β Batch 2: docs[100:200] β Embed β Store β Free memory β | |
| β ... β | |
| β Batch 100: docs[9900:10000] β Embed β Store β Done! β | |
| βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ | |
| ``` | |
| ### What Happens Inside: | |
| ```python | |
| for i in range(0, len(documents), batch_size): | |
| batch = documents[i:i + batch_size] | |
| # ChromaDB does this internally: | |
| # 1. For each doc: vector = embedding_function.embed(doc.page_content) | |
| # 2. Store (id, vector, metadata, page_content) | |
| ids = self._store.add_documents(batch) | |
| ``` | |
| --- | |
| ## π The `similarity_search` Method | |
| ```python | |
| def similarity_search( | |
| self, | |
| query: str, | |
| k: int = None, # Number of results (default: 5 from settings) | |
| filter: Optional[Dict[str, Any]] = None # Metadata filtering | |
| ) -> List[Document]: | |
| ``` | |
| ### How It Works: | |
| ``` | |
| Step 1: Query β Embedding | |
| "What is the refund policy?" β [0.23, -0.45, 0.12, ...] | |
| Step 2: Find K Nearest Neighbors | |
| ChromaDB uses approximate nearest neighbor (ANN) algorithms | |
| Step 3: Return Documents | |
| ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ | |
| β Document 1: "Refunds are processed within 7 days..." β | |
| β Document 2: "Our return policy guarantees..." β | |
| β Document 3: "Money back if not satisfied..." β | |
| ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ | |
| ``` | |
| ### Metadata Filtering: | |
| ```python | |
| # Find only from specific source | |
| results = store.similarity_search( | |
| "refund policy", | |
| k=5, | |
| filter={"file_name": "policies.pdf"} # β Only from this file! | |
| ) | |
| ``` | |
| --- | |
| ## π The `similarity_search_with_score` Method | |
| ```python | |
| def similarity_search_with_score( | |
| self, | |
| query: str, | |
| k: int = None, | |
| filter: Optional[Dict[str, Any]] = None | |
| ) -> List[tuple]: # Returns (Document, score) pairs | |
| ``` | |
| ### Why Scores Matter: | |
| ```python | |
| results = store.similarity_search_with_score("refund policy", k=3) | |
| # Returns: | |
| [ | |
| (Document("Refunds within 7 days..."), 0.92), # β High confidence | |
| (Document("Return shipping free..."), 0.78), # β Medium confidence | |
| (Document("Contact us for help..."), 0.31), # β Low confidence β οΈ | |
| ] | |
| ``` | |
| **Use Cases:** | |
| 1. **Threshold filtering** β only use if score > 0.7 | |
| 2. **Confidence display** β show users "95% relevant" | |
| 3. **Debugging** β understand why wrong docs were retrieved | |
| --- | |
| ## π The `as_retriever` Method (LangChain Integration) | |
| ```python | |
| @property | |
| def retriever(self): | |
| """Default retriever with settings from config.""" | |
| return self._store.as_retriever( | |
| search_type="similarity", | |
| search_kwargs={"k": settings.top_k_results} | |
| ) | |
| ``` | |
| ### Why Retrievers? | |
| LangChain's **LCEL** uses retrievers in chains: | |
| ```python | |
| chain = ( | |
| {"context": store.retriever, "question": RunnablePassthrough()} | |
| | prompt | |
| | llm | |
| | output_parser | |
| ) | |
| answer = chain.invoke("What is the refund policy?") | |
| ``` | |
| --- | |
| ## π The `get_collection_stats` Method | |
| ```python | |
| def get_collection_stats(self) -> Dict[str, Any]: | |
| return { | |
| "collection_name": self.collection_name, | |
| "document_count": collection.count(), | |
| "persist_directory": str(self.persist_directory), | |
| } | |
| ``` | |
| ### Debugging in Production: | |
| ```python | |
| stats = store.get_collection_stats() | |
| # {"collection_name": "documents", "document_count": 1547, ...} | |
| ``` | |
| --- | |
| ## π Summary | |
| | Component | Pattern | Purpose | | |
| |-----------|---------|---------| | |
| | `VectorStore` | Wrapper/Facade | Simplify ChromaDB API | | |
| | `embedding_function` | Dependency Injection | Swap embedding models | | |
| | `add_documents()` | Batch Iterator | Memory-efficient ingestion | | |
| | `similarity_search()` | Query Interface | Semantic document retrieval | | |
| | `as_retriever()` | Adapter | LangChain compatibility | | |
| | `create_vector_store()` | Factory | Simplified instantiation | | |