| # Prism β Learning Concepts |
|
|
| ## Concept Index by Product Workflow |
|
|
| Every concept in this file maps to a specific stage in Prism's request lifecycle. Use this to understand *when* each idea becomes relevant, not just *what* it is. |
|
|
| ``` |
| βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ |
| β USER UPLOADS A DOCUMENT β |
| β β |
| β PDF/TXT/CSV β chunk β embed β store β briefing β |
| β β |
| β Concept 11 RecursiveCharacterTextSplitter β |
| β How text is split into chunks (paragraph β line β |
| β β word β char priority). Overlap bridges splits. β |
| β β |
| β Concept 1 Parent-Child Chunking β |
| β Design pattern: small child chunks for retrieval, β |
| β large parent chunks sent to LLM. β |
| β β |
| β Concept 2 InMemoryStore β |
| β Where parent chunks live (RAM only, dies on β |
| β restart). ChromaDB stores child vectors on disk. β |
| β β |
| β Concept 14 LLM at Ingest Time (Briefing) β |
| β After ingest, LLM auto-generates 5-bullet summary β |
| β + 3 suggested questions. Runs once, not per query. β |
| β β |
| β Concept 21 Contextual Retrieval β |
| β Prepend 2-sentence situating context to each chunk β |
| β before embedding. Fixes decontextualized chunks. β |
| β Measured: +18% recall (0.51β0.60), +6.6pp P@5. β |
| βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ |
| β |
| βΌ |
| βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ |
| β USER SENDS A QUESTION β |
| β β |
| β question β (optionally) multi-query expand β (optionally) β |
| β HyDE expand β retrieve β fuse β rerank β top-5 chunks β |
| β β |
| β Concept 17 Multi-Query Retrieval β |
| β LLM generates 3 phrasings of query β retrieve for β |
| β each β pool + deduplicate β RRF β rerank. β |
| β Widens candidate pool. ON by default. β |
| β β |
| β Concept 7 HyDE β |
| β LLM generates fake answer β embed fake answer β |
| β instead of query β closes question/answer vector β |
| β space gap β higher context recall. β |
| β Measured: +21pp recall (0.51β0.72). ON by default. β |
| β β |
| β Concept 4 BM25Okapi β |
| β Sparse keyword retrieval. Catches exact terms β |
| β (section numbers, βΉ amounts) that dense misses. β |
| β TF saturation prevents high-frequency term bias. β |
| β β |
| β Concept 3 RRF + Cross-Encoder Pipeline β |
| β Dense (0.7) + BM25 (0.3) merged via weighted RRF. β |
| β Cross-encoder reranks top-10 jointly β top-5. β |
| βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ |
| β |
| βΌ |
| βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ |
| β WEB SEARCH (always on) β |
| β β |
| β question β condense with history β Tavily β web results β |
| β β |
| β Concept 5 ConversationalRetrievalChain Condensation Trap β |
| β Chain's internal condense step strips prepended β |
| β web context. Fix: bypass chain entirely for web β |
| β queries. Direct LLM call preserves all context. β |
| β β |
| β Concept 12 ConversationBufferWindowMemory β |
| β Sliding k=10 window of chat history injected into β |
| β condense_question() before Tavily search, so β |
| β follow-up queries have full context. β |
| βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ |
| β |
| βΌ |
| βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ |
| β LLM GENERATES ANSWER β |
| β β |
| β [doc chunks] + [web results] + [chat history] β LLM β answer β |
| β β |
| β Concept 12 ConversationBufferWindowMemory (output_key trap) β |
| β Memory saves this turn for next question. β |
| β output_key="answer" required β chain returns β |
| β multiple keys, memory needs to know which to save. β |
| βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ |
| β |
| βΌ |
| β |
| βΌ |
| βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ |
| β EVALUATION (offline, run via scripts/run_eval_versioned.py) β |
| β β |
| β 50 eval pairs β retrieve β answer β score β versioned JSON β |
| β β |
| β Concept 8 Eval Metric Design (Faithfulness Is Circular) β |
| β Why faithfulness 1.0 was meaningless. How β |
| β answer_correctness (vs ground truth) is honest. β |
| β β |
| β Concept 9 Precision@K vs Recall Diagnostic β |
| β P@5=0.89 + recall=0.51 = pool too narrow. β |
| β The combination tells you exactly what to fix. β |
| β β |
| β Concept 15 All 7 Eval Metrics β Full Reference β |
| β Answer Correctness, Answer Relevancy, Context β |
| β Recall, Precision@5, Latency p50/p95/p99. β |
| β Includes computation steps, failure modes, β |
| β Prism v1.0.0 results, and metric interaction map. β |
| β β |
| β Concept 20 Semantic Chunking Tradeoff β |
| β Topic-boundary splits improve recall but hurt β |
| β precision when eval pairs are aligned to fixed β |
| β chunk boundaries. Measured: +9.3pp recall, β |
| β -27.3pp P@5, 5Γ latency (v1.4.0 ablation). β |
| βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ |
| ``` |
|
|
| --- |
|
|
| ## 1. Parent-Child Chunking β The Retrieval-Faithfulness Tradeoff |
|
|
| ### The Problem with One Chunk Size |
|
|
| | Chunk Size | Retrieval | LLM Answer Quality | |
| | --- | --- | --- | |
| | Small (200 chars) | Precise β matches exact phrase | Bad β too little context, answer is fragmented | |
| | Large (800 chars) | Imprecise β embedding averages over too much text | Good β LLM sees full context | |
|
|
| **Key insight:** Embeddings of large chunks get "diluted" β the vector represents the average meaning of 800 chars. Small chunks have sharper, more focused vectors that match queries better. |
|
|
| ### How ParentDocumentRetriever Works |
|
|
| **INGEST:** |
|
|
| - 800-char parent β stored in `InMemoryStore` (keyed by ID) |
| - 200-char children β embedded β stored in ChromaDB |
|
|
| **QUERY:** |
|
|
| 1. Query hits ChromaDB β finds best matching 200-char child chunk |
| 2. Look up `parent_id` from child metadata |
| 3. Return the full 800-char parent to LLM |
|
|
| ### Why This Works |
|
|
| - **Child chunk** = precise retrieval target (dense vector = focused meaning) |
| - **Parent chunk** = rich answer context (LLM gets surrounding sentences) |
|
|
| ### Concrete Example β RBI Circular PDF |
|
|
| **Raw text (one paragraph):** |
|
|
| > "The Reserve Bank of India has mandated that all UPI transactions above βΉ2,000 must undergo additional authentication from January 2025. This includes biometric verification or OTP-based second factor. Non-compliant PSPs will face penalties up to βΉ10 lakh per violation." |
| > |
| |
| **After chunking:** |
|
|
| - **Parent (800 chars) β InMemoryStore:** Full paragraph above |
| - **Child A (200 chars) β ChromaDB:** "The Reserve Bank of India has mandated that all UPI transactions above βΉ2,000 must undergo additional authentication from January 2025." |
| - **Child B:** "This includes biometric verification or OTP-based second factor." |
| - **Child C:** "Non-compliant PSPs will face penalties up to βΉ10 lakh per violation." |
|
|
| **Query:** `"What is the UPI transaction authentication limit?"` |
|
|
| β ChromaDB finds **Child A** (score: 0.91) β focused vector on UPI + βΉ2000 + authentication |
|
|
| β Fetches parent_id β returns **full parent paragraph** to LLM |
| |
| **Comparison Table:** |
| |
| | Approach | LLM Gets | Problem | |
| | --- | --- | --- | |
| | Dense-only, large chunks | Full paragraph (good) | Match was imprecise β wrong paragraph might score higher | |
| | Dense-only, small chunks | Child A only (38 words) | Misses penalty info β incomplete answer | |
| | ParentDocumentRetriever | Full paragraph (precise match + rich context) | β
Best of both | |
| |
| **LLM final answer:** "UPI transactions above βΉ2,000 require additional authentication (biometric or OTP) from Jan 2025. Non-compliant PSPs face up to βΉ10 lakh penalty." |
| |
| --- |
| |
| ## 2. InMemoryStore β What It Actually Is |
| |
| **Key point: InMemoryStore is NOT part of ChromaDB. It's a separate, RAM-only store.** |
| |
| Under the hood it's essentially a plain Python dict wrapped in a LangChain class: |
| |
| ```python |
| store = {} |
| store["parent_id_abc123"] = "Full 800-char parent chunk text..." |
| store["parent_id_def456"] = "Another parent chunk..." |
| ``` |
| |
| ### Two Separate Stores in Prism |
| |
| | ChromaDB (on disk) | InMemoryStore (RAM only) | |
| | --- | --- | |
| | Child chunk vectors | Parent chunk text | |
| | [vector, metadata] β find similar | parent_id β full text | |
| | β
Survives restart | β Dies on restart | |
|
|
| ### Flow |
|
|
| ``` |
| Child A metadata = { "parent_id": "abc123", "text": "short child..." } |
| β |
| InMemoryStore["abc123"] |
| β |
| "Full 800-char parent text" β LLM |
| ``` |
|
|
| ### β οΈ Known Limitation in Prism |
|
|
| Render free tier cold-starts β InMemoryStore is wiped β must **re-ingest PDFs on every cold start**. |
|
|
| ChromaDB persists to disk so child vectors survive, but parent text is gone. |
|
|
| ### β οΈ Architecture Note β Intended vs Current Implementation |
|
|
| `architecture.md` and Concept 1 describe ParentDocumentRetriever (child 200 / parent 800). The current `ingest.py` uses a single-pass `RecursiveCharacterTextSplitter` at 500 chars β no separate parent store. The ParentDocumentRetriever was the original design and is documented as such. Both chunking approaches teach the same tradeoff; the parent-child concept remains valid as a pattern even if the current code simplified to single-pass chunking. |
|
|
| --- |
|
|
| ## 3. RRF + Cross-Encoder Reranker Pipeline |
|
|
| ### 3a. RRF β Reciprocal Rank Fusion |
|
|
| **Problem:** Dense retrieval returns a ranked list. BM25 returns a ranked list. Scores are on different scales (BM25: 0β15, cosine: 0β1) β can't add them directly. |
|
|
| **RRF Solution:** Ignore raw scores. Use only rank position. |
|
|
| ``` |
| Standard RRF formula: |
| score(doc) = Ξ£ 1 / (k + rank_in_list) |
| k = 60 (constant, dampens top-rank advantage) |
| |
| Prism's weighted RRF (applied inside the formula per list): |
| score(doc) += dense_weight / (k + rank_in_dense_list) # 0.7 Γ contribution |
| score(doc) += sparse_weight / (k + rank_in_sparse_list) # 0.3 Γ contribution |
| ``` |
|
|
| **Example:** |
|
|
| | Doc | Dense (ChromaDB) | BM25 | RRF Score | |
| | --- | --- | --- | --- | |
| | Doc A | Rank 1 (0.91) | Rank 2 (9.1) | 1/61 + 1/62 = **0.0325** β
| |
| | Doc C | Rank 3 (0.71) | Rank 1 (12.3) | 1/63 + 1/61 = **0.0320** | |
| | Doc B | Rank 2 (0.87) | β | 1/62 = 0.0161 | |
| | Doc D | β | Rank 3 (7.4) | 1/63 = 0.0159 | |
|
|
| **Doc A wins** β appeared high in BOTH lists β signals true relevance. |
|
|
| > In Prism: `dense_weight=0.7`, `sparse_weight=0.3` applied **inside** the RRF formula β each list's contribution is multiplied by its weight before summing. Not "before RRF" as a pre-filter, but as a per-list scaling factor within fusion (see Concept 4 for the actual code). |
| > |
| |
| ### 3b. Cross-Encoder Reranker |
|
|
| **Problem:** RRF gives top-10 candidates. Bi-encoder (ChromaDB) encodes query and doc *separately* β approximate similarity. |
|
|
| **Cross-Encoder:** Feeds query + doc *together* into BERT β full attention across both β much more accurate relevance score. |
|
|
| | | Bi-Encoder (ChromaDB) | Cross-Encoder (TinyBERT) | |
| | --- | --- | --- | |
| | Method | embed(query) + embed(doc) β cosine(q,d) | BERT([query][SEP][doc]) β single score 0β1 | |
| | Speed | Fast | Slow | |
| | Accuracy | Approximate | Accurate | |
| | Encoding | Independent | Joint | |
|
|
| **Example β after RRF top-10:** |
|
|
| | Pair | Cross-Encoder Score | |
| | --- | --- | |
| | (query, Doc A) | 0.94 β
| |
| | (query, Doc B) | 0.88 β
| |
| | (query, Doc C) | 0.61 | |
| | (query, Doc D) | 0.23 | |
|
|
| Top-5 by cross-encoder score β LLM |
|
|
| ### Full Retrieval Pipeline |
|
|
| ``` |
| Query |
| | |
| ββ ChromaDB dense β top-10 ranked docs |
| ββ BM25 sparse β top-10 ranked docs |
| | |
| β |
| RRF fusion β merged top-10 (rank-based, scale-agnostic) |
| | |
| β |
| Cross-encoder β re-scores all 10 jointly with query |
| | |
| β |
| Top-5 parent chunks β LLM |
| ``` |
|
|
| --- |
|
|
| --- |
|
|
| ## 4. BM25Okapi β Why Keywords Beat Embeddings for Exact Terms |
|
|
| ### The Problem with Dense Retrieval on Regulatory Text |
|
|
| Embeddings capture *meaning*. But regulatory text has exact identifiers β section numbers, policy codes, rupee amounts β where the exact token matters, not the meaning. |
|
|
| **Example:** |
|
|
| Query: `"What is the penalty for UPI non-compliance?"` |
|
|
| A dense embedding model reads this as: *"something about UPI and consequences"*. |
|
|
| Two chunks in corpus: |
| - Chunk A: `"Non-compliant PSPs will face penalties up to βΉ10 lakh per violation."` β exact answer |
| - Chunk B: `"UPI has transformed digital payments in India with over 100 billion transactions."` β semantically close (UPI topic) but wrong |
|
|
| Dense retrieval might rank Chunk B high because it's heavily UPI-themed. BM25 ranks Chunk A high because "penalt" and "non-compli" are rare, high-signal tokens. |
|
|
| ### How BM25Okapi Works |
|
|
| Plain TF-IDF problem: a doc that says "UPI" 50 times gets 50Γ the score. That's unfair β one mention of "βΉ10 lakh" in the right context should beat 50 mentions of "UPI" in a generic overview. |
|
|
| BM25 fixes this with **term frequency saturation**: |
|
|
| ``` |
| BM25 score = IDF(term) Γ [ tf Γ (k1 + 1) ] / [ tf + k1 Γ (1 - b + b Γ dl/avgdl) ] |
| |
| Where: |
| tf = how many times term appears in this chunk |
| IDF = how rare the term is across all chunks (log scale) |
| k1 = saturation constant (~1.5) β controls how fast TF saturates |
| b = length normalization (~0.75) |
| dl = this chunk's length |
| avgdl = average chunk length in corpus |
| ``` |
|
|
| **Saturation in plain English:** |
|
|
| | tf (term count in chunk) | TF-IDF score | BM25 score (k1=1.5) | |
| | --- | --- | --- | |
| | 1 | 1.0 | 1.0 | |
| | 5 | 5.0 | 1.56 β plateaus | |
| | 20 | 20.0 | 1.79 β barely grows | |
| | 50 | 50.0 | 1.88 β effectively capped | |
|
|
| A chunk mentioning "UPI" 50 times scores almost the same as one mentioning it 5 times. But "βΉ10 lakh" appearing once in a chunk that has it = high IDF (rare token) Γ full TF benefit. |
|
|
| ### In Prism |
|
|
| ```python |
| # bm25_index.py |
| from rank_bm25 import BM25Okapi |
| |
| # .lower() matters β "UPI" and "upi" are different tokens without it |
| corpus = [doc["content"].lower().split() for doc in all_chunks] |
| bm25 = BM25Okapi(corpus) |
| |
| # At query time β query also lowercased to match corpus tokenization: |
| scores = bm25.get_scores(query.lower().split()) |
| top_k_idx = sorted(range(len(scores)), key=lambda i: scores[i], reverse=True)[:k] |
| ``` |
|
|
| BM25 operates on raw tokens (word split). No embeddings. No GPU. Rebuilds in ~1s at startup from corpus. |
|
|
| **Weight in Prism:** `sparse_weight=0.3` in RRF β BM25 is complementary, not dominant. Dense handles semantic meaning; BM25 catches exact matches dense misses. |
|
|
| --- |
|
|
| ## 5. ConversationalRetrievalChain β The Silent Context Killer |
|
|
| ### What the Chain Does Internally |
|
|
| `ConversationalRetrievalChain` has two internal steps most people don't know about: |
|
|
| ``` |
| User question + chat history |
| β |
| [STEP 1] Condense question |
| LLM rewrites "Is the price level good?" |
| β "Is Bajaj Finance stock βΉ908-924 a good buy in 2026?" |
| (standalone question for retrieval) |
| β |
| [STEP 2] Retrieve + Answer |
| Standalone question β retriever β top-5 chunks β LLM answer |
| ``` |
|
|
| Step 1 exists so follow-up questions work without full history context. Good design for RAG. |
|
|
| ### The Bug: Web Context Gets Stripped |
|
|
| When Prism added Tavily web search, the naive approach was: |
|
|
| ```python |
| web_results = tavily.search(question) |
| augmented_question = f"Web context: {web_results}\n\nQuestion: {question}" |
| chain.invoke({"question": augmented_question}) # β WRONG |
| ``` |
|
|
| What actually happens: |
|
|
| ``` |
| augmented_question (with Tavily content prepended) |
| β |
| [STEP 1] Chain's condense LLM: |
| "Rewrite this as a standalone retrieval query." |
| Output: "What is Bajaj Finance stock price?" |
| β ALL TAVILY CONTENT STRIPPED. LLM never sees it. |
| β |
| [STEP 2] Answer LLM gets: corpus chunks only. No web context. |
| ``` |
|
|
| The chain's condensation step rewrites the question for retrieval quality β and throws away everything else. |
|
|
| ### The Fix in Prism |
|
|
| Bypass the chain entirely for web queries. Direct LLM call: |
|
|
| ```python |
| # chain.py |
| def run_query_with_web(question, rag_docs, web_sources, memory): |
| history = memory.load_memory_variables({})["history"] |
| |
| prompt = f""" |
| Chat history: {history} |
| |
| Web search results: |
| {web_sources} |
| |
| Document context: |
| {rag_docs} |
| |
| Question: {question} |
| Answer:""" |
| |
| answer = llm.invoke(prompt) |
| memory.save_context({"input": question}, {"answer": answer}) |
| return answer |
| ``` |
|
|
| No condensation step. Web context reaches the LLM guaranteed. |
|
|
| **Lesson:** LangChain abstractions are powerful but opaque. When something doesn't work, read what the chain actually does internally β don't assume the abstraction is transparent. |
|
|
| --- |
|
|
| ## 7. HyDE β Hypothetical Document Embeddings |
|
|
| ### The Problem with Embedding a Question |
|
|
| When you search ChromaDB, you embed the *query* and find similar vectors. But your corpus contains *answers*, not questions. They live in different regions of vector space. |
|
|
| **Concrete example:** |
|
|
| Query: `"What is the UPI transaction limit for P2P transfers?"` |
|
|
| This embeds as a *question vector* β the model has seen millions of questions, it knows this pattern. |
|
|
| The answer in your corpus: `"P2P UPI transfers are capped at βΉ1 lakh per transaction per day as per NPCI guidelines."` |
|
|
| This embeds as an *answer vector* β declarative sentence, factual tone, different region in 1536-dimensional space. |
|
|
| They're semantically related, but the vector distance is larger than it should be. |
|
|
| ### HyDE's Solution |
|
|
| Before searching, ask the LLM to hallucinate an answer: |
|
|
| ``` |
| Step 1: LLM generates a fake answer to the query |
| Query: "What is the UPI transaction limit for P2P transfers?" |
| Fake answer: "The UPI transaction limit for P2P transfers is typically |
| set by NPCI and varies by bank, generally around βΉ1 lakh |
| per day for most PSPs." |
| |
| Step 2: Embed the FAKE ANSWER (not the query) |
| vector = embed("The UPI transaction limit for P2P transfers is...") |
| |
| Step 3: Search ChromaDB with this vector |
| β Finds real answer chunks that are semantically close to a fake answer |
| β Much closer in vector space than the original question was |
| ``` |
|
|
| ### Why "Hypothetical" Works |
|
|
| The fake answer and the real corpus chunk are both declarative, factual, answer-shaped text. They live in the same region of vector space. The query (a question) lives elsewhere. |
|
|
| ``` |
| Vector space (simplified): |
| |
| [Question zone] [Answer zone] |
| "What is UPI limit?" --- ... --- "P2P capped at βΉ1 lakh..." |
| β β |
| far from corpus close to corpus chunk |
| |
| HyDE: |
| "UPI limit is ~βΉ1 lakh..." β fake answer β close to real chunk β
|
| ``` |
|
|
| ### In Prism |
|
|
| ```python |
| # retriever.py |
| def _hyde_expand(self, query: str) -> str: |
| prompt = f"Write a 2-sentence factual answer to: {query}" |
| return self.llm.invoke(prompt).content |
| |
| def _get_relevant_documents(self, query: str): |
| dense_query = self._hyde_expand(query) if self.use_hyde else query |
| |
| # Dense search uses fake answer embedding |
| dense_docs = self.vectorstore.similarity_search(dense_query, k=10) |
| |
| # BM25 still uses original query (keyword matching needs real terms) |
| sparse_docs = self.bm25_retrieve(query, k=10) |
| |
| return self.rrf_and_rerank(dense_docs, sparse_docs) |
| ``` |
|
|
| **ON by default** (`config.yaml` `hyde_enabled: true`) β adds one Groq call per query (~200ms). Latency cost accepted. |
|
|
| **Measured lift (v1.1.0, 18 samples):** context_recall 0.51 β 0.72 (+21pp). Latency 2029ms β 4018ms p50 (2Γ). HyDE specifically helps recall β it finds chunks that keyword/direct-embed matching misses. |
| |
| --- |
| |
| ## 8. Eval Metric Design β Why Faithfulness Is Circular |
| |
| ### Four RAGAS Metrics and What They Actually Measure |
| |
| | Metric | Judge compares | Question it answers | |
| | --- | --- | --- | |
| | **faithfulness** | LLM answer vs retrieved chunks | "Did the LLM stick to what was in the retrieved docs?" | |
| | **answer_correctness** | LLM answer vs ground_truth reference | "Is the answer actually right?" | |
| | **answer_relevancy** | LLM answer vs original question | "Did the answer address what was asked?" | |
| | **context_recall** | Retrieved chunks vs ground_truth | "Did retrieval find the chunks needed to answer?" | |
| |
| ### The Faithfulness Trap |
| |
| In Prism v1 eval, faithfulness scored 1.0. Seemed great. Was meaningless. |
| |
| Here's why: |
| |
| ``` |
| Eval pair designed alongside corpus: |
| Question: "What is the UPI P2P transaction limit?" |
| Ground truth: "βΉ1 lakh per day" |
| |
| Corpus chunk (also written by us): |
| "P2P UPI transfers are capped at βΉ1 lakh per day per NPCI guidelines." |
| |
| Flow: |
| 1. LLM retrieves that chunk (of course β it's perfectly matched) |
| 2. LLM answers: "The UPI P2P limit is βΉ1 lakh per day." |
| 3. RAGAS faithfulness judge: "Does answer match retrieved chunk?" β YES β score: 1.0 |
| ``` |
| |
| The judge is comparing the answer to the chunk that was *designed to produce that answer*. Circular. Score tells you the retrieval worked, not whether the answer is correct. |
| |
| ### answer_correctness Is the Honest Metric |
|
|
| ``` |
| Flow: |
| 1. LLM answers: "The UPI P2P limit is βΉ1 lakh per day." |
| 2. Ground truth (human-written reference): "βΉ1 lakh per transaction per day" |
| 3. RAGAS judge: "Does answer match ground truth?" β mostly yes β score: 0.82 |
| ``` |
|
|
| Judge compares to an *independent human reference*. No circular dependency on retrieved chunks. Harder to game. |
|
|
| **Prism v1.0.0 Violet results:** |
|
|
| | Metric | Score | Interpretation | |
| | --- | --- | --- | |
| | answer_correctness | 0.82 | 82% of answers match ground truth β honest signal | |
| | answer_relevancy | 0.62 | RAGAS penalizes verbosity β Groq 70B tends to over-explain | |
| | context_recall | 0.51 | Only 51% of needed chunks retrieved β target for Multi-Query | |
| | P@5 | 0.89 | When chunks are retrieved, 89% are correct β precision is fine | |
| |
| ### The P@5 + Recall Diagnostic |
| |
| P@5=0.89 + recall=0.51 tells you something specific: |
| |
| ``` |
| Retrieval pool is too narrow. |
| |
| "When we retrieve something, it's usually right (0.89)." |
| "But we're missing ~half the relevant chunks (0.51)." |
| |
| Root cause: single-phrasing query misses chunks phrased differently. |
| Fix: Multi-Query Retrieval β generate 3 phrasings, retrieve for each, pool candidates. |
| ``` |
| |
| This is exactly how production ML teams diagnose retrieval systems. The combination of metrics points to the specific fix. |
| |
| --- |
| |
| ## 9. Precision@K vs Recall β The Retrieval Diagnostic Pair |
| |
| ### Definitions in Plain English |
| |
| Imagine your corpus has **5 chunks** that are genuinely relevant to a query. Your retriever returns **5 chunks** (K=5). |
| |
| ``` |
| Ground truth relevant chunks: [A, B, C, D, E] |
| Retrieved chunks: [A, B, X, Y, Z] |
| |
| Precision@5 = correct retrieved / K = 2/5 = 0.40 |
| "Of what I returned, how much was right?" |
| |
| Recall@5 = correct retrieved / total relevant = 2/5 = 0.40 |
| "Of all the right chunks, how many did I find?" |
| ``` |
| |
| ### The Four Diagnostic Combinations |
| |
| | P@5 | Recall | Diagnosis | Fix | |
| | --- | --- | --- | --- | |
| | High | High | β
Retrieval working well | Ship it | |
| | High | Low | Pool too narrow β finding right chunks but missing others | Multi-Query Retrieval, HyDE | |
| | Low | High | Too much noise β finding relevant chunks but also junk | Better reranking, stricter K | |
| | Low | Low | Retrieval fundamentally broken | Check embeddings, chunking strategy | |
| |
| ### Prism v1.0.0 Violet: High P, Low Recall |
| |
| ``` |
| P@5 = 0.89 β "When Prism retrieves a chunk, 89% of the time it's relevant." |
| Recall = 0.51 β "But Prism only finds 51% of the relevant chunks total." |
| |
| Example query: "What are the RBI guidelines on UPI merchant limits?" |
| |
| Relevant chunks in corpus: [merchant_limit_2024, merchant_kyc, limit_circular_2023, payment_cap, psp_obligations] |
|
|
| Prism retrieved: [merchant_limit_2024, merchant_kyc, some_unrelated_chunk, another_unrelated, payment_cap] |
| |
| Precision@5 = 3/5 = 0.60 (found 3 right ones, 2 noise) |
| Recall = 3/5 = 0.60 (missed limit_circular_2023 and psp_obligations) |
| ``` |
| |
| Why did it miss them? `limit_circular_2023` might use different phrasing: *"The ceiling for merchant UPI collections was revised..."* β no overlap with "merchant limits". Single-phrasing retrieval misses it. |
| |
| Multi-Query generates: `"UPI merchant payment ceiling"`, `"RBI merchant collection limit"`, `"PSP merchant UPI cap"` β retrieves from all three β pools candidates β recall rises. |
| |
| --- |
| |
| --- |
| |
| ## 11. RecursiveCharacterTextSplitter β How Text Gets Chunked |
| |
| ### The Problem with Fixed Splits |
| |
| Splitting every 500 characters naively: |
| |
| ``` |
| "...comply with Section 7(b) of the Act. |SPLIT| Payment systems must maintain..." |
| ``` |
| |
| Cut mid-sentence. "Section 7(b)" separated from "Payment systems must maintain" β the context that makes "Section 7(b)" meaningful is now in a different chunk. The embedding for that chunk is weak. |
| |
| ### How RecursiveCharacterTextSplitter Works |
| |
| It tries a priority list of separators, from most preferred to least: |
| |
| ```python |
| separators = ["\n\n", "\n", " ", ""] |
| # Priority: paragraph break > line break > word break > character break |
| ``` |
| |
| For a given `chunk_size=500`: |
| |
| 1. **Try `\n\n` first** β split at paragraph boundaries. If resulting piece β€ 500 chars: done. |
| 2. **If piece still >500** β try `\n` (line breaks within the paragraph). |
| 3. **If still >500** β try spaces (word boundaries). |
| 4. **Last resort** β split at exact character position. |
| |
| **Result:** Chunks break at natural language boundaries when possible, hard character cuts only when forced. |
| |
| ### Overlap β Why Adjacent Chunks Share Text |
| |
| ```python |
| splitter = RecursiveCharacterTextSplitter( |
| chunk_size=500, |
| chunk_overlap=50, # last 50 chars of chunk N = first 50 chars of chunk N+1 |
| ) |
| ``` |
| |
| Without overlap: |
| ``` |
| Chunk 1: "...UPI transaction limits are set by NPCI guidelines for" |
| Chunk 2: "each payment service provider based on risk assessment." |
| ``` |
| Query `"UPI limit by PSP"` β the full sentence is split. Neither chunk alone has a strong embedding. |
|
|
| With overlap=50: |
| ``` |
| Chunk 1: "...UPI transaction limits are set by NPCI guidelines for each" |
| Chunk 2: "for each payment service provider based on risk assessment." |
| ``` |
| The bridging phrase appears in both chunks. One of them will have a strong vector for the full concept. |
|
|
| ### In Prism |
|
|
| ```python |
| # ingest.py |
| splitter = RecursiveCharacterTextSplitter( |
| chunk_size=500, # from config.yaml |
| chunk_overlap=50, |
| ) |
| chunks = splitter.split_documents(documents) |
| ``` |
|
|
| **Note:** Prism originally used ParentDocumentRetriever (child 200 / parent 800). The current code uses a single-pass splitter at 500 chars. The architecture.md documents the intended design; the code reflects the current implementation. Both teach the same concept β chunk size is a precision-vs-context tradeoff. |
|
|
| --- |
|
|
| ## 12. ConversationBufferWindowMemory β Sliding Window Chat History |
|
|
| ### Why You Need Memory in a Chat System |
|
|
| Each `POST /api/chat` is a stateless HTTP request. The LLM has no memory of what was said 10 seconds ago. Without memory: |
|
|
| ``` |
| User: "Tell me about UPI transaction limits." |
| Prism: "UPI P2P limit is βΉ1 lakh per day..." |
| |
| User: "What about merchant payments?" β no context |
| LLM sees: question = "What about merchant payments?" β what is "about"? who knows? |
| Prism: "Merchant payments are a type of..." β wrong, generic answer |
| ``` |
|
|
| ### How ConversationBufferWindowMemory Works |
|
|
| Keeps last `k` conversation turns (human + AI) as a rolling window: |
|
|
| ```python |
| # memory.py |
| memory = ConversationBufferWindowMemory( |
| memory_key="chat_history", |
| return_messages=True, |
| output_key="answer", # β critical β explained below |
| k=10, # keep last 10 turns |
| ) |
| ``` |
|
|
| Turn 1: `[Human: "Tell me about UPI limits", AI: "UPI P2P is βΉ1 lakh..."]` |
| Turn 2: `[Human: "Tell me about UPI limits", AI: "...", Human: "What about merchant payments?", AI: "Merchant UPI limit is βΉ5 lakh..."]` |
|
|
| At turn 11: Turn 1 is evicted. Window always has last 10. |
|
|
| The chain sees: |
| ``` |
| Chat history: |
| Human: Tell me about UPI limits |
| AI: UPI P2P is βΉ1 lakh per day... |
| |
| Current question: What about merchant payments? |
| ``` |
| β LLM understands "What about" refers to UPI limits. Answers correctly. |
|
|
| ### The output_key Trap |
| |
| `ConversationalRetrievalChain` returns a dict with multiple keys: |
| |
| ```python |
| result = chain.invoke({"question": "..."}) |
| # result = { |
| # "answer": "UPI P2P limit is βΉ1 lakh...", |
| # "source_documents": [...], |
| # "question": "..." |
| # } |
| ``` |
| |
| Memory's `save_context` call needs to know which key is the "output" to save: |
| |
| ```python |
| # WRONG β KeyError because chain returns multiple output keys |
| memory = ConversationBufferWindowMemory(memory_key="chat_history") |
|
|
| # RIGHT β tell memory exactly which key to save |
| memory = ConversationBufferWindowMemory( |
| memory_key="chat_history", |
| output_key="answer", # save result["answer"], not the whole dict |
| ) |
| ``` |
| |
| Without `output_key="answer"`: LangChain tries to infer the output key, sees multiple candidates, raises `ValueError: Multiple keys returned`. This was a real bug hit during Prism development β subtle because the chain runs fine; the crash happens on the `save_context` call after. |
|
|
| --- |
|
|
| ## 14. LLM at Ingest Time β The Briefing Pattern |
|
|
| ### Two Places LLMs Can Run in a RAG System |
|
|
| Most tutorials show LLM running only at **query time**: user asks β retrieve β LLM answers. |
|
|
| Prism also runs LLM at **ingest time**: document uploaded β LLM summarizes β shown to user immediately. |
|
|
| ``` |
| INGEST TIME (once, when doc uploaded): |
| PDF β chunks β embed β store |
| β (also) |
| LLM reads first 6 chunks |
| β generates 5-bullet summary |
| β generates 3 suggested questions |
| β returned in upload response |
| |
| QUERY TIME (every user message): |
| question β retrieve β LLM answers |
| ``` |
|
|
| ### Why This Is Useful |
|
|
| When a user uploads an unfamiliar document, they don't know what to ask. The briefing gives them: |
|
|
| 1. **Orientation** β "what is this document about?" |
| 2. **Starting questions** β clickable prompts that immediately work |
|
|
| ```python |
| # briefing.py |
| def generate_briefing(doc_name: str, text_sample: str) -> dict: |
| prompt = f""" |
| Analyze this document excerpt and respond with valid JSON only. |
| Document: {doc_name} |
| Content: {text_sample[:3000]} |
| |
| Return exactly: {{"summary": ["5 bullet strings"], "suggested_questions": ["3 question strings"]}} |
| """ |
| response = llm.invoke([HumanMessage(content=prompt)]) |
| # Parse JSON from response β return to frontend |
| ``` |
|
|
| ```python |
| # routes/upload.py β after ingestion completes: |
| briefing = generate_briefing(doc_name, sample_text) |
| return { |
| "uploaded": [filename], |
| "documents": docs, |
| "briefing": briefing, # β frontend shows this immediately |
| } |
| ``` |
|
|
| ### Design Decisions |
|
|
| **Non-critical path** β briefing failure doesn't block upload: |
| ```python |
| try: |
| briefing = generate_briefing(...) |
| except Exception as e: |
| logger.warning("Briefing skipped: %s", e) |
| briefing = None # upload still succeeds |
| ``` |
|
|
| **JSON extraction from LLM output** β LLMs often wrap JSON in markdown fences (` ```json`). Prism strips these with regex before `json.loads()`: |
| ```python |
| raw = re.sub(r"^```(?:json)?\s*", "", raw) # strip opening fence |
| raw = re.sub(r"\s*```$", "", raw) # strip closing fence |
| match = re.search(r"\{.*\}", raw, re.DOTALL) # extract JSON object |
| data = json.loads(match.group()) |
| ``` |
|
|
| **When to use LLM at ingest time vs query time:** |
|
|
| | | Ingest time | Query time | |
| | --- | --- | --- | |
| | Runs | Once per document | Once per user message | |
| | Cost | Fixed (paid once) | Scales with usage | |
| | Best for | Document-level metadata, summaries, question suggestions | Answering specific user questions | |
| | Examples | Briefing, contextual retrieval, chunk tagging | RAG answer, faithfulness eval, HyDE | |
|
|
| The most powerful use of ingest-time LLM is **contextual retrieval** (roadmap): for every chunk, ask LLM "given this full document, write 2 sentences situating this chunk" β then prepend that context before embedding. Anthropic reports ~49% reduction in retrieval failures. Same LLM-at-ingest-time pattern, much higher impact. |
|
|
| --- |
|
|
| --- |
|
|
| ## 15. Eval Dashboard Metrics β Full Reference |
|
|
| Prism's eval dashboard shows 7 numbers per run. Each answers a different question about the system. Understanding *what* each metric measures, *how* it's computed, and *what can fool it* is the difference between blindly running evals and actually improving the system. |
|
|
| --- |
|
|
| ### Overview β What Each Metric Diagnoses |
|
|
| ``` |
| Query β [Retrieval] β top-5 chunks β [LLM] β answer |
| β β β |
| Precision@5 Context Recall Answer Correctness |
| Answer Relevancy |
| Latency p50/p95/p99 |
| ``` |
|
|
| | Metric | What it measures | Requires ground truth? | LLM call? | |
| | --- | --- | --- | --- | |
| | Answer Correctness | Is the answer factually right? | Yes | Yes (judge) | |
| | Answer Relevancy | Does answer address the question? | No | Yes + embed | |
| | Context Recall | Did retrieval find all needed chunks? | Yes | Yes (RAGAS) | |
| | Precision@5 | Are the top-5 chunks relevant? | Yes (keywords/sources) | No | |
| | Latency p50/p95/p99 | How fast is the system? | No | No | |
|
|
| --- |
|
|
| ### Metric 1 β Answer Correctness (primary) |
|
|
| **Question it answers:** "Is Prism's answer actually correct?" |
|
|
| **Why it's primary:** The only metric that directly measures answer quality vs a human-authored reference. Everything else is a proxy. |
|
|
| **How it's computed:** |
|
|
| ```python |
| CORRECTNESS_PROMPT = """ |
| You are an evaluation judge. Score the generated answer against the reference answer. |
| |
| Use a 1β5 integer scale: |
| 5 β All key facts present and correct |
| 4 β Most key facts correct, minor omissions or imprecision |
| 3 β Some key facts correct, moderate gaps |
| 2 β Few facts correct, significant errors or hallucinations |
| 1 β Completely wrong, irrelevant, or contradicts reference |
| |
| Reference answer: {ground_truth} |
| Generated answer: {answer} |
| |
| Respond with JSON: {"score": 4, "reason": "one sentence"} |
| """ |
| |
| # Normalize 1β5 β 0β1 |
| score_normalized = (raw_score - 1) / 4 |
| ``` |
|
|
| **Example:** |
|
|
| ``` |
| Ground truth: "UPI P2P limit is βΉ1 lakh per transaction per day per NPCI guidelines." |
| Answer: "You can transfer up to βΉ1 lakh daily on UPI for person-to-person transfers." |
| |
| Judge: score=4 (correct amount + correct type, slight imprecision on "per transaction") |
| Normalized: (4-1)/4 = 0.75 |
| ``` |
|
|
| **What the judge model sees:** Only the answer and ground truth. NOT the retrieved chunks. This is intentional β if the answer is wrong but the chunks were correct, that's an LLM reasoning failure. If the answer is right but the chunks were irrelevant, that's lucky hallucination. |
|
|
| **What fools it:** |
| - 8B judge is lenient β "approximately correct" often scores 4/5 |
| - Rephrased correct answers may score 3 if judge doesn't recognize equivalence |
| - Very long answers may confuse the judge β score correct facts buried in padding lower |
|
|
| **Prism v1.0.0 result:** `0.82` β 82% of answers have most key facts correct. Target: >0.85 via better retrieval (Multi-Query β more context β better answers). |
|
|
| --- |
|
|
| ### Metric 2 β Answer Relevancy |
|
|
| **Question it answers:** "Did Prism answer the question that was actually asked, or did it go off on a tangent?" |
|
|
| **How it's computed (RAGAS):** |
|
|
| ``` |
| Step 1: Given the generated answer, LLM generates N reverse questions |
| Answer: "UPI P2P limit is βΉ1 lakh per day..." |
| Reverse Q1: "What is the UPI transaction limit?" |
| Reverse Q2: "How much can you transfer via UPI P2P daily?" |
| Reverse Q3: "What is the per-day UPI P2P cap?" |
| |
| Step 2: Embed original question + all N reverse questions |
| embed("What is the UPI transaction limit for P2P transfers?") β vector_q |
| embed("What is the UPI transaction limit?") β vector_r1 |
| embed("How much can you transfer via UPI P2P daily?") β vector_r2 |
| |
| Step 3: Cosine similarity between original question and each reverse question |
| sim(vector_q, vector_r1) = 0.94 |
| sim(vector_q, vector_r2) = 0.91 |
| sim(vector_q, vector_r3) = 0.89 |
| |
| Step 4: Answer relevancy = mean similarity = (0.94 + 0.91 + 0.89) / 3 = 0.91 |
| ``` |
|
|
| **Intuition:** If the answer actually addresses the question, reverse-engineered questions from that answer will be similar to the original question. If the answer went off-topic, the reverse questions will diverge. |
|
|
| **What fools it:** |
| - **Verbosity penalty:** Groq 70B tends to over-explain. A padded answer like "Great question! UPI P2P limit is βΉ1 lakh. UPI has transformed payments in India, with many PSPs offering..." generates reverse questions about UPI history, not the limit β lower similarity β lower score. |
| - Doesn't check factual correctness β a confident wrong answer can score high if it's "on topic". |
|
|
| **Prism v1.0.0 result:** `0.62` β lower than expected. Direct cause: Groq 70B verbose responses. Fix: tighter system prompt (`"Be concise. One paragraph maximum."`). |
|
|
| --- |
|
|
| ### Metric 3 β Context Recall |
|
|
| **Question it answers:** "Did the retriever surface all the chunks that were needed to construct a correct answer?" |
|
|
| **How it's computed (RAGAS):** |
|
|
| ``` |
| Step 1: Take ground truth reference answer |
| "UPI P2P limit is βΉ1 lakh per day. Merchant UPI payments are capped at βΉ5 lakh." |
| |
| Step 2: RAGAS LLM breaks it into individual factual sentences (claims) |
| Claim A: "UPI P2P limit is βΉ1 lakh per day." |
| Claim B: "Merchant UPI payments are capped at βΉ5 lakh." |
| |
| Step 3: For each claim, check: can this claim be attributed to any retrieved chunk? |
| Claim A β search retrieved chunks β Chunk 3 contains "P2P cap βΉ1 lakh" β ATTRIBUTED β |
| Claim B β search retrieved chunks β no chunk mentions merchant cap β NOT ATTRIBUTED β |
| |
| Step 4: Context recall = attributed claims / total claims = 1/2 = 0.50 |
| ``` |
|
|
| **Intuition:** Recall measures completeness of the retrieval pool. Low recall = the retriever missed relevant chunks β LLM can't answer completely even if it tries β answer will be incomplete or hallucinated. |
|
|
| **Why this is the most important metric to improve:** |
| - Low recall (0.51 in v1.0.0) means ~half the relevant information is missing from context |
| - LLM can only work with what it's given β perfect LLM + bad retrieval = bad answer |
| - Improving recall directly improves answer quality |
|
|
| **What fools it:** |
| - Ground truth phrasing matters β if GT says "βΉ1 lakh" and the chunk says "100,000 rupees", RAGAS LLM may or may not link them |
| - Long GT answers with many claims β denominator grows β harder to achieve high recall |
|
|
| **Prism v1.0.0 result:** `0.51` β retriever finds correct chunks (P@5=0.89) but misses ~half the relevant ones. **Directly motivates Multi-Query Retrieval.** |
|
|
| --- |
|
|
| ### Metric 4 β Precision@5 |
|
|
| **Question it answers:** "Of the 5 chunks Prism returned, how many were actually relevant?" |
|
|
| **How it's computed (custom, deterministic β no LLM):** |
|
|
| ```python |
| def compute_precision_at_k(query, retrieved_chunks, ground_truth, k=5): |
| relevant_sources = ground_truth.get("relevant_sources", []) # expected source filenames |
| keywords = ground_truth.get("relevant_chunk_keywords", []) # expected keywords |
| |
| top_k = retrieved_chunks[:k] |
| relevant_count = 0 |
| |
| for chunk in top_k: |
| source_match = chunk["source"] in relevant_sources |
| keyword_match = any(kw.lower() in chunk["content"].lower() for kw in keywords) |
| |
| if source_match or keyword_match: # OR β either condition counts |
| relevant_count += 1 |
| |
| return relevant_count / k # 0.0 β 1.0 |
| ``` |
|
|
| **Example eval pair:** |
| ```json |
| { |
| "query": "What is the UPI merchant payment limit?", |
| "relevant_sources": ["npci_merchant_guidelines.pdf"], |
| "relevant_chunk_keywords": ["merchant", "βΉ5 lakh", "payment cap", "PSP ceiling"] |
| } |
| ``` |
|
|
| Retrieved chunks: |
| ``` |
| Chunk 1 (source: npci_merchant_guidelines.pdf) β source_match β β relevant |
| Chunk 2 (content: "...merchant PSP ceiling is βΉ5 lakh...") β keyword_match β β relevant |
| Chunk 3 (source: rbi_circular.pdf, content: "UPI transaction monitoring...") β neither β irrelevant |
| Chunk 4 (source: npci_merchant_guidelines.pdf) β source_match β β relevant |
| Chunk 5 (content: "...digital payment systems in India...") β neither β irrelevant |
| |
| P@5 = 3/5 = 0.60 |
| ``` |
|
|
| **Why deterministic (no LLM):** |
| - Fast β runs per-query without API calls |
| - Reproducible β same query always same score |
| - No rate limits β can evaluate 50 pairs without hitting Groq TPD |
|
|
| **What fools it:** |
| - Source OR keyword: a chunk from the wrong document that happens to mention a keyword counts as relevant |
| - Keyword list quality matters β too broad β everything matches, score inflates β too narrow β correct chunks missed, score deflates |
|
|
| **Prism v1.0.0 result:** `0.89` β 89% of retrieved chunks are relevant. High precision + low recall = pool is accurate but too narrow. The retriever is finding the right kind of chunks, just not enough of them. |
|
|
| --- |
|
|
| ### Metric 5 β Latency p50 / p95 / p99 |
|
|
| **Question it answers:** "How fast is Prism in practice?" |
|
|
| **Why three numbers instead of average:** |
|
|
| ``` |
| Query latencies (ms): [1800, 1950, 2100, 1900, 2050, 2200, 1850, 3800, 2000, 1950] |
| β one slow outlier |
| |
| Mean: 2160ms β pulled up by the outlier, misleading |
| p50: 1975ms β 50% of queries faster than this (typical experience) |
| p95: 3230ms β 95% of queries faster than this (worst normal case) |
| p99: 3740ms β 99% of queries faster than this (absolute worst) |
| ``` |
|
|
| **How it's computed:** |
|
|
| ```python |
| import numpy as np |
| |
| # Measure per query: retrieval + LLM call (no Tavily, no web search) |
| t0 = time.time() |
| answer, docs, chunks = _answer_query(llm, retriever, query) |
| latency_ms = int((time.time() - t0) * 1000) |
| latencies.append(latency_ms) |
| |
| # After all queries: |
| p50 = int(np.percentile(latencies, 50)) # median |
| p95 = int(np.percentile(latencies, 95)) # 95th percentile |
| p99 = int(np.percentile(latencies, 99)) # 99th percentile |
| ``` |
|
|
| **What's included in the latency measurement:** |
| - ChromaDB dense retrieval (cosine ANN search) |
| - BM25 sparse retrieval (in-memory token scoring) |
| - RRF fusion (pure Python, negligible) |
| - TinyBERT cross-encoder reranking (BERT forward pass Γ 10 pairs) |
| - Groq LLM call (network + inference) |
|
|
| **What's NOT included:** |
| - Tavily web search (separate, always adds 500β1000ms) |
| - HyDE expansion (adds one extra Groq call ~200ms) |
| - Python GC (`gc.collect()` after response) |
|
|
| **Prism results across versions:** |
|
|
| | Version | Stack | p50 | p95 | p99 | |
| | --- | --- | --- | --- | --- | |
| | v1.0.0 | baseline | 2029ms | β | β | |
| | v1.1.0 | HyDE | 4018ms | 6452ms | 6455ms | |
| | v1.2.0 | HyDE+MQ | 1812ms | 6136ms | 6744ms | |
| | v1.3.0 | HyDE+MQ+CTX | 2610ms | 6720ms | 6959ms | |
|
|
| HyDE adds one Groq call per query β p50 2Γ. Contextual chunks are longer β LLM processes more tokens β higher p50 vs baseline. Wide p95/p99 gap = occasional Groq congestion spikes, not retrieval. |
|
|
| **Production benchmarks for context:** |
|
|
| | System | p50 target | Notes | |
| | --- | --- | --- | |
| | Search engine | <200ms | Pre-indexed, no LLM | |
| | RAG (GPU cloud) | 500β800ms | GPU reranker + fast LLM | |
| | Prism (Render free CPU) | ~2000ms | CPU reranker + network LLM | |
| | Prism + streaming | ~300ms to first token | Same total time, perceived faster | |
|
|
| --- |
|
|
| ### How the Metrics Interact β Reading the Dashboard |
|
|
| ``` |
| High correctness + high recall + high P@5 = system is working β
|
| |
| High P@5 + low recall: |
| β Retriever finds right chunks but misses others |
| β Fix: Multi-Query Retrieval, HyDE |
| |
| Low P@5 + high recall: |
| β Retriever finds many chunks but most are noise |
| β Fix: stricter reranking, smaller retrieve_k |
| |
| High recall + low correctness: |
| β Retriever gave good context, LLM failed to use it |
| β Fix: better system prompt, larger LLM model |
| |
| Low relevancy + high correctness: |
| β LLM is verbose/padded but factually right |
| β Fix: tighter prompt ("be concise, one paragraph maximum") |
| |
| High latency p99 >> p50: |
| β Occasional slow queries (complex retrieval or Groq congestion) |
| β Monitor, add timeout handling |
| ``` |
|
|
| **Metric evolution across Prism versions:** |
|
|
| ``` |
| v1.0.0 v1.1.0 v1.2.0 v1.3.0 |
| (base) (HyDE) (HyDE+MQ)(+CTX) |
| correctness: 0.82 0.75 0.77 0.78 |
| relevancy: 0.62 0.845 0.890 0.799 |
| recall: 0.51 0.721 0.645 0.768 β primary target |
| P@5: 0.89 0.911 0.904 0.984 |
| p50: 2029ms 4018ms 1812ms 2610ms |
| |
| Verdict (v1.0.0): recall is the primary bottleneck. |
| Fix: HyDE (+21pp recall), Multi-Query (wider pool), Contextual Retrieval (+18% recall). |
| Best production stack: v1.3.0 β HyDE + Multi-Query + Contextual Retrieval. |
| ``` |
|
|
| --- |
|
|
| ## 17. Multi-Query Retrieval β Wider Net for Higher Recall |
|
|
| ### The Problem: Single Phrasing Has Blind Spots |
|
|
| A query hits ChromaDB + BM25 with one set of tokens. Corpus chunks that express the same concept differently never surface. |
|
|
| ``` |
| Query: "What is the UPI merchant limit?" |
| |
| Corpus has: |
| Chunk A: "merchant UPI transaction cap is βΉ5 lakh" β found β (keyword match) |
| Chunk B: "ceiling for PSP collections was revised..." β MISSED (different vocab) |
| Chunk C: "payment service providers may not exceed..." β MISSED (no "merchant" token) |
| ``` |
|
|
| Chunk B and C are relevant. Single-phrasing retrieval misses them. This is why Prism v1.0.0 Violet had context_recall = 0.51 β only half the needed chunks were retrieved. |
| |
| ### The Fix: Generate 3 Phrasings, Pool Results |
| |
| ``` |
| Step 1: LLM generates 3 paraphrases of the original query |
| Original: "What is the UPI merchant limit?" |
| Phrasing 2: "What is the PSP payment ceiling for UPI collections?" |
| Phrasing 3: "How much can merchants accept via UPI transactions?" |
| |
| Step 2: Run retrieval for EACH phrasing |
| Original β [Chunk A, Chunk D, Chunk E, ...] (retrieve_k per list) |
| Phrasing 2 β [Chunk B, Chunk A, Chunk F, ...] β Chunk B appears now |
| Phrasing 3 β [Chunk C, Chunk B, Chunk G, ...] β Chunk C appears now |
|
|
| Step 3: Pool + deduplicate by content key (keep best rank per chunk) |
| Combined unique pool: [A, B, C, D, E, F, G, ...] |
|
|
| Step 4: RRF fuse the deduplicated pool β rerank top-5 |
| Reranker sees 30 candidates instead of 10 β picks best 5 from wider pool |
| ``` |
| |
| ### Why Deduplication Uses Best Rank |
| |
| If Chunk A appears at rank 1 in phrasing 1 and rank 3 in phrasing 2, keep it at rank 1 β its highest-confidence rank. The merged ranked list is then sorted by best rank before RRF fusion. |
| |
| ```python |
| # retriever.py β deduplication loop |
| for rank, doc in enumerate(d_results): |
| key = doc["content"][:120] |
| if key not in dense_seen or rank < dense_seen[key][0]: |
| dense_seen[key] = (rank, doc) # keep best (lowest) rank |
|
|
| dense_pool = [doc for _, doc in sorted(dense_seen.values(), key=lambda x: x[0])] |
| ``` |
| |
| ### In Prism |
| |
| ```python |
| # retriever.py |
| def _multi_query_expand(self, query: str) -> list[str]: |
| prompt = ( |
| "Generate 3 different phrasings of the following question for document retrieval. " |
| "Each phrasing must use different vocabulary but seek the same information. " |
| "Return ONLY the 3 questions, one per line, no numbering, no preamble.\n\n" |
| f"Question: {query}" |
| ) |
| # returns [original_query, phrasing_2, phrasing_3, phrasing_4] |
| |
| def _get_relevant_documents(self, query, ...): |
| queries = self._multi_query_expand(query) if self.use_multi_query else [query] |
| |
| dense_seen, sparse_seen = {}, {} |
| for q in queries: |
| # retrieve for each phrasing, keep best rank per unique chunk |
| ... |
| |
| dense_pool = sorted_by_best_rank(dense_seen) |
| sparse_pool = sorted_by_best_rank(sparse_seen) |
| |
| fused = self._rrf_fuse(dense_pool, sparse_pool) # wider pool |
| reranked = rerank(query, fused[:retrieve_k], top_k=rerank_k) # reranker uses original query |
| ``` |
| |
| **ON by default** (`config.yaml` `multi_query_enabled: true`) β adds one Groq call per query (~200ms). |
|
|
| **Measured result (v1.2.0, 25 samples, HyDE+MQ):** recall=0.645, P@5=0.904. Multi-Query alone provided smaller-than-expected recall lift. Root cause: query-side reformulation cannot fix decontextualized chunk embeddings β chunks with weak vectors rank low regardless of how the query is phrased. The larger recall gain came from Contextual Retrieval (Concept 21), which fixes chunk quality at ingest time. |
|
|
| **Cost:** 1 extra Groq call + 3Γ retrieval calls (fast, in-memory) + 3Γ BM25 (negligible). |
|
|
| **Reranker still uses original query** β not the phrasings. The phrasings widen the candidate pool; the reranker judges relevance against what the user actually asked. |
|
|
| --- |
|
|
| ## Summary β Key Concepts Cheatsheet |
|
|
| | Concept | What it solves | Where in Prism | |
| | --- | --- | --- | |
| | ParentDocumentRetriever | Retrieval precision + answer faithfulness | Ingestion + retrieval layer | |
| | InMemoryStore | Fast parent text lookup by ID | RAM store (lost on restart) | |
| | ChromaDB | Vector similarity search for child chunks | Persistent disk store | |
| | BM25Okapi | Exact keyword matching with TF saturation | Sparse retrieval (weight 0.3) | |
| | RRF | Merging dense + sparse rankings (scale-agnostic) | Post-retrieval fusion | |
| | Cross-Encoder | Accurate joint query-doc relevance scoring | Final reranking step | |
| | Chain condensation trap | Why LangChain strips web context silently | Web query bypass in chain.py | |
| | HyDE | Closes question-answer vector space gap; +21pp recall | Dense retrieval (hyde_enabled: true) | |
| | Faithfulness is circular | Why eval metrics designed alongside corpus lie | answer_correctness chosen as primary | |
| | P@5 + Recall diagnostic pair | Identifies whether retrieval pool is narrow or noisy | v1.0.0 Violet: P=0.89, R=0.51 | |
| | RecursiveCharacterTextSplitter | Splits at paragraph/line/word/char in priority order | ingest.py chunking | |
| | ConversationBufferWindowMemory | Sliding k-window of chat history + output_key trap | memory.py, used in chain | |
| | LLM at ingest time | Briefing pattern β run LLM once per doc, not per query | briefing.py on upload | |
| | Contextual Retrieval | Fix decontextualized chunks at ingest; +18% recall | `contextualize_chunks_async()` in ingest.py | |
| | Answer Correctness | LLM judge 1β5 vs ground truth, normalized β 0β1 | Primary metric, independent of retrieved chunks | |
| | Answer Relevancy | Reverse-question cosine similarity β penalizes verbose LLM | RAGAS, no ground truth needed | |
| | Context Recall | GT sentences attributed to retrieved chunks Γ· total | RAGAS, measures retrieval completeness | |
| | Precision@5 | Relevant chunks in top-5 Γ· 5, source + keyword match | Custom deterministic, no LLM call | |
| | Latency p50/p95/p99 | Percentile timing: median / worst-normal / absolute-worst | numpy.percentile over per-query ms measurements | |
| | Multi-Query Retrieval | Wider candidate pool; best rank dedup before RRF | retriever.py `_multi_query_expand()` (multi_query_enabled: true) | |
| | Semantic Chunking Tradeoff | Recall vs precision when eval is fixed-chunk-aligned | v1.4.0 ablation: +9.3pp recall, β27.3pp P@5, 5Γ latency | |
|
|
| --- |
|
|
|
|
| --- |
|
|
| ## 20. Semantic Chunking Tradeoff β Recall vs Precision |
|
|
| ### The Problem with Fixed-Size Splits |
|
|
| `RecursiveCharacterTextSplitter` at 500 chars cuts mid-sentence, mid-table, mid-list. Embedding a truncated sentence returns a weak vector β the chunk doesn't represent a complete idea. Semantic chunking fixes this by splitting at topic boundaries (where cosine similarity between adjacent sentences drops below a threshold), producing variable-size chunks that each represent one coherent thought. |
|
|
| ### What the Ablation Showed (Prism v1.4.0, 25 samples) |
|
|
| | Stack | recall | P@5 | latency p50 | |
| |-------|--------|-----|-------------| |
| | v1.3.0 HyDE+MQ+CTX (fixed 500-char) | 0.768 | **0.984** | 2610ms | |
| | v1.4.0 + Semantic chunking | **0.861** | 0.711 | 12952ms | |
| | Delta | +9.3pp | **β27.3pp** | 5Γ slower | |
|
|
| ### Why Recall Went Up |
|
|
| Semantic chunks contain complete sentences and complete ideas. The embedding captures the full meaning β better vector β higher chance of matching the relevant query β more reference content covered β higher context recall. |
|
|
| ### Why Precision Collapsed |
|
|
| P@5 measures overlap between retrieved chunks and the ground-truth `relevant_chunks` field in `eval_pairs.json`. Those reference chunks were defined against fixed 500-char splits. Semantic chunks have different, larger boundaries β they contain the relevant content but as part of a bigger unit, so exact overlap with the reference drops. The reranker also receives a wider, noisier candidate pool (variable-size chunks = less uniform scoring surface). |
|
|
| **Key insight:** P@5 is eval-alignment-sensitive. If ground truth was labelled against fixed chunks, semantic chunks will always score lower on P@5 even when they retrieve better content. For a production system with human-labelled relevance judgements (not self-aligned eval pairs), the gap would be smaller. |
|
|
| ### Why Latency Was 5Γ |
|
|
| Semantic chunks are longer on average than 500-char fixed chunks. Longer chunks = more tokens fed to the LLM per answer generation call. Retrieval time is unchanged (semantic chunking is an ingest-time decision), but inference time scales with context length. |
|
|
| ### Decision |
|
|
| Semantic chunking rejected for Prism production. v1.3.0 (HyDE + Multi-Query + Contextual) confirmed as best stack: P@5=0.984, recall=0.768, p50=2610ms. |
|
|
| Semantic chunking would make more sense when: |
| - Ground truth eval labels are created after chunking (aligned to the actual chunk boundaries) |
| - LLM context window is not a bottleneck |
| - Recall is the primary metric (e.g. legal/compliance: never miss a relevant clause) |
|
|
| --- |
|
|
| ## 21. Contextual Retrieval β Fixing Decontextualized Chunks at Ingest |
|
|
| ### The Problem: Fixed-Size Chunks Lose Context |
|
|
| `RecursiveCharacterTextSplitter` at 500 chars cuts documents into fragments. Many fragments are decontextualized β they lack the surrounding information that gives them meaning: |
|
|
| ``` |
| Chunk from NPCI merchant guidelines PDF: |
| "The limit was revised to βΉ2 lakh." |
| |
| Problems: |
| - "The limit" β which limit? Not in this chunk. |
| - "revised" β from what? Not in this chunk. |
| - "βΉ2 lakh" β for what transaction type? Not in this chunk. |
| |
| Embedding of this chunk β weak, generic vector. |
| Query "merchant UPI transaction cap" β this chunk may not surface. |
| ``` |
|
|
| No query-side technique (HyDE, Multi-Query) can fix this β the chunk embedding is weak regardless of how the query is phrased. |
|
|
| ### The Fix: Anthropic's Contextual Retrieval |
|
|
| At ingest time, before embedding, ask the LLM to prepend 2 sentences situating each chunk in its document: |
|
|
| ``` |
| Prompt: |
| "Given this document: [full doc or representative sample] |
| Write 2 sentences situating this chunk in context. |
| |
| Chunk: 'The limit was revised to βΉ2 lakh.'" |
| |
| LLM output: |
| "In the NPCI UPI merchant guidelines (2024), Section 4.3 covers |
| transaction ceiling revisions for PSPs. The limit was revised to βΉ2 lakh." |
| ``` |
|
|
| The contextual prefix is **prepended to the chunk** before embedding: |
|
|
| ```python |
| # ingest.py |
| chunk.page_content = f"{context_prefix}\n\n{chunk.page_content}" |
| # then embed this augmented text |
| ``` |
|
|
| The chunk stored for retrieval is now specific and rich. Same chunk now surfaces for "merchant UPI transaction cap" queries. |
|
|
| ### Why This Works |
|
|
| | | Original chunk | Contextual chunk | |
| |---|---|---| |
| | Text | "The limit was revised to βΉ2 lakh." | "NPCI UPI merchant guidelines 2024, PSP ceiling. The limit was revised to βΉ2 lakh." | |
| | Embedding | Generic "revision" vector | Specific "merchant UPI PSP ceiling" vector | |
| | Retrieval | Misses merchant-related queries | Surfaces correctly | |
|
|
| The embedding now represents a complete, specific idea instead of a floating fragment. |
|
|
| ### Design Pattern: Ingest-Time vs Query-Time |
|
|
| | | Ingest-time LLM (Contextual Retrieval) | Query-time LLM (HyDE, Multi-Query) | |
| |---|---|---| |
| | Runs | Once per chunk, at upload | Every query | |
| | Cost | Paid once; benefit on every future query | Paid per query | |
| | What it fixes | Bad chunk embeddings (quality problem) | Query-corpus vocabulary gap (coverage problem) | |
| | Latency impact | Upload slower (~40s for 30 chunks) | Query slower (+200ms per technique) | |
|
|
| **Key insight:** Query-side techniques improve how well a query matches existing chunk vectors. Contextual retrieval improves the chunk vectors themselves. Both are needed for maximum recall. |
|
|
| ### Measured Results β Prism v1.3.0 (25 samples) |
|
|
| | Metric | v1.0.0 baseline | v1.3.0 (HyDE+MQ+CTX) | Delta | |
| |--------|-----------------|----------------------|-------| |
| | context_recall | 0.51 | 0.768 | **+25.8pp** | |
| | precision_at_5 | 0.89 | 0.984 | **+9.4pp** | |
| | answer_relevancy | 0.62 | 0.799 | +17.9pp | |
| | latency p50 | 2029ms | 2610ms | +28% | |
|
|
| Contextual retrieval contributes ~+18% recall (v1.3.0 vs v1.2.0 without CTX: 0.768 vs 0.645). |
|
|
| ### Production Complication: 40s Upload |
|
|
| 30 chunks Γ 1 Groq call Γ ~1.3s/call (sequential) = ~40s blocking. Solution: parallel calls with `asyncio.Semaphore(3)` β 3 concurrent Groq calls at ~3000 TPM burst (under Groq's 6000 TPM limit). Reduces to ~15s. Two-phase upload: sync non-contextual embed first (user can query in <3s), contextual replacement in background. |
|
|
| ### Distinction from Briefing (Concept 14) |
|
|
| | | Briefing | Contextual Retrieval | |
| |---|---|---| |
| | What | Document-level 5-bullet summary + 3 questions | Chunk-level 2-sentence situating context | |
| | Shown to user | Yes (in upload response) | No (prepended to chunk text, invisible) | |
| | Purpose | User orientation | Embedding quality improvement | |
| | LLM calls | 1 per document | 1 per chunk (~30 per doc) | |
