devrajsinh2012 commited on
Commit
06f3edb
·
verified ·
1 Parent(s): 76b3d38

Delete MEXAR_Phase1_Architecture_Fixes.md

Browse files
Files changed (1) hide show
  1. MEXAR_Phase1_Architecture_Fixes.md +0 -309
MEXAR_Phase1_Architecture_Fixes.md DELETED
@@ -1,309 +0,0 @@
1
- # MEXAR Phase 1 — Architecture Fixes to Match the Paper
2
-
3
- **Purpose of this document:** hand this to an AI coding agent (Antigravity or similar) working directly in the `Mexar` repository (`github.com/devrajsinh2012/Mexar`). This is Phase 1 only: it closes the gap between what the paper *"Hybrid Retrieval and Claim Verification: An Empirical Study of Constrained RAG for High-Stakes Professional Domains"* claims about the system's architecture, and what the code currently does. No dataset work in this document, that's Phase 2, handled separately.
4
-
5
- **Presentation date:** July 30-31, 2026. Target for this phase: 3 days.
6
-
7
- **Ground rule for the agent:** every change below must keep the app deployable on the current stack (FastAPI backend on Hugging Face Spaces, Supabase Postgres + pgvector, React frontend on Vercel). Don't introduce services that need infra Dev doesn't already have.
8
-
9
- ---
10
-
11
- ## 1.0 Preconditions
12
-
13
- - Add to `backend/requirements.txt`:
14
- ```
15
- scikit-learn>=1.4.0
16
- spacy>=3.7.0
17
- ```
18
- - After install, download the small English model: `python -m spacy download en_core_web_sm`. Add this as a step in `Dockerfile` so it survives Hugging Face Spaces rebuilds.
19
- - Verify the pgvector extension version on the Supabase project supports HNSW (needs pgvector >= 0.5.0). Run `SELECT extversion FROM pg_extension WHERE extname = 'vector';`. If below 0.5.0, run `ALTER EXTENSION vector UPDATE;` before touching the index migration in 1.2.
20
-
21
- ---
22
-
23
- ## 1.1 Domain Guardrail: replace the heuristic with real TF-IDF + NER + Jaccard
24
-
25
- **Why:** Paper Section III-A defines the guardrail as TF-IDF term weighting (Eq. 1), NER-extracted domain entities, and Jaccard similarity between query and signature (Eq. 2). The current `_check_guardrail()` in `backend/modules/reasoning_engine.py` is a hand-tuned fuzzy-string-match heuristic with arbitrary bonus weights and no TF-IDF or NER anywhere. This needs to become the real thing.
26
-
27
- **New file: `backend/utils/domain_signature.py`**
28
-
29
- ```python
30
- """
31
- MEXAR - Domain Signature Construction
32
- Implements Section III-A: TF-IDF lexical signature + NER entity signature.
33
- """
34
- from typing import List, Dict, Set, Tuple
35
- from sklearn.feature_extraction.text import TfidfVectorizer
36
- import spacy
37
- import logging
38
-
39
- logger = logging.getLogger(__name__)
40
- _nlp = None
41
-
42
- def get_nlp():
43
- global _nlp
44
- if _nlp is None:
45
- _nlp = spacy.load("en_core_web_sm")
46
- return _nlp
47
-
48
- # Entity labels relevant to professional domains (medical/legal/financial)
49
- RELEVANT_ENTITY_LABELS = {"ORG", "LAW", "PRODUCT", "GPE", "PERSON", "NORP", "EVENT", "FAC"}
50
-
51
- def build_tfidf_signature(documents: List[str], top_n: int = 100, tau_tf: float = 0.0) -> List[Tuple[str, float]]:
52
- """
53
- Eq. 1: w(t, D) = tf(t, D) * log(|D| / |{d in D : t in d}|)
54
- Returns top_n (term, weight) pairs above tau_tf, sorted descending by weight.
55
- """
56
- if not documents or len(documents) < 2:
57
- # TF-IDF needs a corpus of multiple "documents" to compute IDF meaningfully.
58
- # If compiling from a single uploaded file, split it into paragraph-level
59
- # pseudo-documents so IDF is not degenerate.
60
- documents = _split_into_pseudo_documents(documents[0]) if documents else []
61
-
62
- vectorizer = TfidfVectorizer(
63
- max_features=2000,
64
- stop_words="english",
65
- ngram_range=(1, 2),
66
- min_df=1,
67
- )
68
- matrix = vectorizer.fit_transform(documents)
69
- scores = matrix.sum(axis=0).A1
70
- terms = vectorizer.get_feature_names_out()
71
- pairs = [(t, float(s)) for t, s in zip(terms, scores) if s > tau_tf]
72
- pairs.sort(key=lambda x: x[1], reverse=True)
73
- return pairs[:top_n]
74
-
75
- def _split_into_pseudo_documents(text: str, chunk_size: int = 500) -> List[str]:
76
- words = text.split()
77
- return [" ".join(words[i:i+chunk_size]) for i in range(0, len(words), chunk_size)] or [text]
78
-
79
- def extract_ner_entities(documents: List[str], tau_ent: int = 2, max_docs_to_scan: int = 200) -> List[str]:
80
- """
81
- Extracts named entities appearing in at least tau_ent documents.
82
- Capped at max_docs_to_scan for latency (spaCy is not free at compile time).
83
- """
84
- nlp = get_nlp()
85
- doc_freq: Dict[str, int] = {}
86
- for doc_text in documents[:max_docs_to_scan]:
87
- seen_in_this_doc: Set[str] = set()
88
- for ent in nlp(doc_text[:5000]).ents: # cap per-doc length for speed
89
- if ent.label_ in RELEVANT_ENTITY_LABELS:
90
- key = ent.text.lower().strip()
91
- if key and key not in seen_in_this_doc:
92
- seen_in_this_doc.add(key)
93
- for key in seen_in_this_doc:
94
- doc_freq[key] = doc_freq.get(key, 0) + 1
95
- return [k for k, v in doc_freq.items() if v >= tau_ent]
96
-
97
- def build_domain_signature(
98
- documents: List[str],
99
- tau_tf: float = 0.0,
100
- tau_ent: int = 2,
101
- top_n_lexical: int = 100,
102
- ) -> Dict[str, List]:
103
- """Returns Sigma = Sigma_lex ∪ Sigma_ent as described in Section III-A, kept separate for inspection."""
104
- lexical_pairs = build_tfidf_signature(documents, top_n=top_n_lexical, tau_tf=tau_tf)
105
- lexical_terms = [t for t, _ in lexical_pairs]
106
- entities = extract_ner_entities(documents, tau_ent=tau_ent)
107
- return {
108
- "lexical": lexical_terms,
109
- "lexical_weights": {t: w for t, w in lexical_pairs},
110
- "entities": entities,
111
- "combined": sorted(set(lexical_terms) | set(e for e in entities)),
112
- }
113
- ```
114
-
115
- **Modify `backend/modules/knowledge_compiler.py`:**
116
- Replace `_extract_domain_signature()` so it calls `build_domain_signature()` from the new module, passing in the raw per-source-document text list (not the flattened `text_context` string, we need per-document granularity for TF-IDF's document-frequency term to mean anything). Store the result's `combined` list in the existing `domain_signature` JSONB column, and add the `lexical_weights` dict to a **new** JSONB column (see migration below) so the guardrail can compute a weighted Jaccard rather than a flat one if useful later.
117
-
118
- **New migration** (Alembic or raw SQL depending on how `backend/migrations` is structured, check `backend/migrations/README.md` for the convention already in use):
119
- ```sql
120
- ALTER TABLE agents ADD COLUMN IF NOT EXISTS domain_signature_weights JSONB;
121
- ALTER TABLE agents ADD COLUMN IF NOT EXISTS domain_entities JSONB;
122
- ```
123
-
124
- **Rewrite `_check_guardrail()` in `backend/modules/reasoning_engine.py`:**
125
- Delete the fuzzy-match/bonus-score implementation entirely. Replace with real Eq. 2:
126
-
127
- ```python
128
- def _check_guardrail(self, query: str, domain_signature: List[str], prompt_analysis: Dict[str, Any]) -> Tuple[bool, float]:
129
- """Eq. 2: sim(q, Sigma) = |Phi(q) ∩ Sigma| / |Phi(q)|"""
130
- from utils.domain_signature import get_nlp
131
- nlp = get_nlp()
132
- query_doc = nlp(query.lower())
133
- # Phi(q): lemmatized content tokens, excluding stopwords/punctuation
134
- phi_q = {
135
- tok.lemma_ for tok in query_doc
136
- if not tok.is_stop and not tok.is_punct and len(tok.text) > 2
137
- }
138
- sigma = set(s.lower() for s in (domain_signature or []))
139
-
140
- if not phi_q:
141
- return False, 0.0
142
-
143
- intersection = phi_q & sigma
144
- score = len(intersection) / len(phi_q)
145
-
146
- is_in_domain = score >= self.DOMAIN_SIMILARITY_THRESHOLD
147
- logger.info(f"Guardrail: sim={score:.3f}, |Phi(q)|={len(phi_q)}, |intersection|={len(intersection)}, in_domain={is_in_domain}")
148
- return is_in_domain, score
149
- ```
150
-
151
- **Tune `DOMAIN_SIMILARITY_THRESHOLD`:** the current 0.05 was calibrated against the *old* heuristic scorer, it will not transfer to real Jaccard similarity, which tends to produce lower absolute values on short queries. Don't guess this: after wiring the new guardrail, run `backend/evaluation/guardrail_analysis.py` against a small hand-labeled set of ~20 in-domain and ~20 out-of-domain queries per existing test_data domain, sweep the threshold from 0.05 to 0.4 in steps of 0.05, and pick the value that maximizes F1. Record that sweep, it becomes supporting evidence for Table IV in the paper instead of a guessed constant.
152
-
153
- **Acceptance test:** for the medical test agent, a query like "what is the recommended treatment for stage 2 hypertension" should be accepted; a query like "how do I file a small claims lawsuit" should be rejected. Log both the old and new guardrail score during a transition period to sanity-check before removing the old code path.
154
-
155
- ---
156
-
157
- ## 1.2 Vector index: IVFFlat → HNSW
158
-
159
- **Why:** Paper Section III-B explicitly claims HNSW indexing. The current migration (`backend/migrations/hybrid_search_function.sql`, lines ~98-101) creates an IVFFlat index instead.
160
-
161
- **New migration file** `backend/migrations/switch_to_hnsw_index.sql`:
162
- ```sql
163
- -- Drop the old IVFFlat index
164
- DROP INDEX IF EXISTS idx_document_chunks_embedding;
165
-
166
- -- Create HNSW index to match paper Section III-B
167
- CREATE INDEX IF NOT EXISTS idx_document_chunks_embedding_hnsw
168
- ON document_chunks USING hnsw (embedding vector_cosine_ops)
169
- WITH (m = 16, ef_construction = 64);
170
-
171
- COMMENT ON INDEX idx_document_chunks_embedding_hnsw IS
172
- 'HNSW index for cosine similarity search, m=16, ef_construction=64 (Section III-B)';
173
- ```
174
- Apply via `Supabase:apply_migration` (or however existing migrations are run, check for a runner script referenced in `backend/migrations/README.md`) rather than editing the old file, so history is preserved.
175
-
176
- **Note for the agent:** HNSW build time scales worse than IVFFlat with corpus size, but at the 500-600 document / few-thousand-chunk scale planned for the presentation, this is a non-issue. No further tuning needed.
177
-
178
- **Acceptance test:** `\d document_chunks` in psql (or `SELECT indexdef FROM pg_indexes WHERE tablename = 'document_chunks';`) should show the HNSW index, not IVFFlat. Re-run a handful of existing queries and confirm retrieval still returns coherent top-5 chunks (HNSW is approximate, results may reorder slightly versus IVFFlat, that's expected and fine).
179
-
180
- ---
181
-
182
- ## 1.3 RRF weighting: reconcile code and paper
183
-
184
- **Why:** Paper Eq. 5 is unweighted RRF. `backend/migrations/hybrid_search_function.sql` applies a 0.6 semantic / 0.4 keyword weighting inside the SQL function.
185
-
186
- **Decision for this project (recommended, don't re-litigate):** keep the 0.6/0.4 weighting in code, it's a reasonable and defensible design choice, and update the paper instead. This is less risky than ripping out a tuned weighting under time pressure. The agent does not need to touch `hybrid_search_function.sql` for this item, just flag it in a "paper edits needed" note for later.
187
-
188
- If Dev instead decides to match the paper exactly, the one-line change is setting `semantic_weight := 0.5; keyword_weight := 0.5;` in that SQL function, but this is **not** the default action, don't do it unless explicitly told.
189
-
190
- ---
191
-
192
- ## 1.4 Faithfulness verification: make DeBERTa-v3 NLI the real, primary signal
193
-
194
- **Why:** this is the most important fix. Paper Section III-C and the entire Related Work argument against LLM-judging-LLM circularity claim the production system verifies claims with "an NLI model (DeBERTa-v3-large fine-tuned on MNLI)" using per-document max entailment (Eq. 6). In the actual code:
195
- - The primary confidence-driving call in `reasoning_engine.reason()` (Step 6) is `self.faithfulness_scorer.score(answer, context)`, which is `FaithfulnessScorer` in `backend/utils/faithfulness.py`, an LLM (Groq) extracting claims and then Groq again answering YES/NO. That's exactly the circularity the paper argues against.
196
- - The actual NLI model in the repo, `BartNLIScorer`, uses `facebook/bart-large-mnli`, not DeBERTa-v3-large, and it's computed only as a side "reviewer feedback" score (`bart_nli_result` in `reason()`), it never feeds the confidence calculation.
197
- - The NLI scorer concatenates all retrieved context into a single premise string, it does not implement the per-document max in Eq. 6.
198
-
199
- **Model choice:** use `cross-encoder/nli-deberta-v3-large` from Hugging Face. This is a DeBERTa-v3-large architecture fine-tuned specifically for NLI (SNLI + MultiNLI), it is the standard, citable checkpoint people mean when they say "DeBERTa-v3-large fine-tuned on MNLI" in RAG faithfulness papers, and it loads cleanly through `sentence-transformers`'s `CrossEncoder` class, which is already a dependency.
200
-
201
- **Modify `backend/utils/faithfulness.py`:** replace the `BartNLIScorer` class with:
202
-
203
- ```python
204
- class DebertaNLIScorer:
205
- """
206
- Primary faithfulness verifier per Section III-C.
207
- Per-claim, per-document max entailment probability (Eq. 6),
208
- aggregated as the fraction of claims exceeding tau_ent (Eq. 7).
209
- """
210
- LABEL_MAP = {0: "contradiction", 1: "entailment", 2: "neutral"} # verify against model card at load time
211
- TAU_ENT = 0.7
212
-
213
- def __init__(self):
214
- self._model = None
215
-
216
- @property
217
- def model(self):
218
- if self._model is None:
219
- from sentence_transformers import CrossEncoder
220
- logger.info("Loading cross-encoder/nli-deberta-v3-large...")
221
- self._model = CrossEncoder("cross-encoder/nli-deberta-v3-large")
222
- logger.info("DeBERTa-v3 NLI model loaded.")
223
- return self._model
224
-
225
- def score(self, answer: str, context_documents: List[str]) -> FaithfulnessResult:
226
- """
227
- context_documents: list of individual retrieved chunk texts (NOT concatenated),
228
- so per-document max in Eq. 6 is meaningful.
229
- """
230
- import re
231
- import numpy as np
232
-
233
- if not answer or not context_documents:
234
- return FaithfulnessResult(score=1.0, total_claims=0, supported_claims=0, unsupported_claims=[])
235
-
236
- sentences = [s.strip() for s in re.split(r'(?<=[.!?])\s+', answer) if len(s.strip()) > 15][:10]
237
- if not sentences:
238
- return FaithfulnessResult(score=1.0, total_claims=0, supported_claims=0, unsupported_claims=[])
239
-
240
- supported = 0
241
- unsupported = []
242
-
243
- for sentence in sentences:
244
- pairs = [(doc[:2000], sentence) for doc in context_documents]
245
- scores = self.model.predict(pairs) # shape: (n_docs, 3) logits/probs per label
246
- probs = self._softmax(scores)
247
- entailment_idx = 1 # confirm this matches the model's actual label order at integration time
248
- max_entailment = float(np.max(probs[:, entailment_idx]))
249
- if max_entailment > self.TAU_ENT:
250
- supported += 1
251
- else:
252
- unsupported.append(sentence)
253
-
254
- score = supported / len(sentences)
255
- return FaithfulnessResult(
256
- score=round(score, 3),
257
- total_claims=len(sentences),
258
- supported_claims=supported,
259
- unsupported_claims=unsupported[:5],
260
- )
261
-
262
- @staticmethod
263
- def _softmax(x):
264
- import numpy as np
265
- e_x = np.exp(x - np.max(x, axis=-1, keepdims=True))
266
- return e_x / e_x.sum(axis=-1, keepdims=True)
267
- ```
268
-
269
- **Important integration detail for the agent:** verify `cross-encoder/nli-deberta-v3-large`'s actual `id2label` mapping from its `config.json` on first load and print it to logs, cross-encoder NLI checkpoints don't always use the same label order. Don't hardcode `entailment_idx = 1` blindly, confirm it, this is exactly the kind of silent bug that would quietly invalidate the reported faithfulness numbers.
270
-
271
- **Modify `backend/modules/reasoning_engine.py`, `reason()` method:**
272
- - Rename the old `FaithfulnessScorer` usage. Keep the class in the codebase (don't delete it), but relabel its role: it becomes the **"LLM-as-judge baseline"** used for baseline comparison tables (this is genuinely useful, RAGAS-style LLM self-eval is a legitimate baseline to compare against in Table I/III, just not the production signal).
273
- - Swap Step 6 to call `self.deberta_nli_scorer.score(answer, [c.content for c in top_chunks])` (passing the list of individual chunks, not the concatenated `context` string) as the **primary** `faithfulness_result` used in `_calculate_confidence()` and in the returned `explainability` payload.
274
- - Keep computing the old LLM-based score too, but store it under a clearly-labeled `"llm_judge_baseline_score"` key in `explainability.confidence_breakdown`, not as the headline faithfulness number.
275
-
276
- **Acceptance test:** feed a known-hallucinated answer (write one manually, e.g. an answer that states a fabricated statistic not present in any retrieved chunk) through the pipeline and confirm the DeBERTa scorer flags it as unsupported. Feed a fully-grounded answer and confirm it scores above `TAU_ENT`. Compare against the LLM-judge baseline score on the same pair, they should usually agree, but if they diverge a lot on the same input, that's worth investigating before trusting either for the paper's numbers.
277
-
278
- ---
279
-
280
- ## 1.5 Wiring check
281
-
282
- After 1.1-1.4 are done, re-run the existing `backend/evaluation/baseline_runner.py` and `backend/evaluation/benchmark_runner.py` against the current small `test_data/` samples (not for final numbers, just as a smoke test) and confirm nothing throws. Fix any interface mismatches from the `context: str` → `context_documents: List[str]` signature change in the faithfulness scorer, other call sites (`evaluation/metrics.py`, `_run_baseline()` in `reasoning_engine.py`) pass a concatenated string today and will need the same list-of-chunks treatment.
283
-
284
- ---
285
-
286
- ## Summary of files touched in Phase 1
287
-
288
- | File | Change |
289
- |---|---|
290
- | `backend/requirements.txt` | add scikit-learn, spacy |
291
- | `backend/Dockerfile` | add spacy model download step |
292
- | `backend/utils/domain_signature.py` | **new** — TF-IDF + NER signature builder |
293
- | `backend/modules/knowledge_compiler.py` | `_extract_domain_signature()` rewritten to use new module |
294
- | `backend/modules/reasoning_engine.py` | `_check_guardrail()` rewritten to real Jaccard; Step 6 rewired to DeBERTa scorer |
295
- | `backend/utils/faithfulness.py` | `BartNLIScorer` replaced with `DebertaNLIScorer` (per-doc max, Eq. 6/7) |
296
- | `backend/migrations/switch_to_hnsw_index.sql` | **new** — HNSW index migration |
297
- | `backend/migrations/*` (new file) | add `domain_signature_weights`, `domain_entities` columns |
298
- | `backend/evaluation/*` | update call sites for the `context: str` → `context_documents: List[str]` signature change |
299
-
300
- ---
301
-
302
- ## What Phase 1 does NOT cover (handled in the separate Phase 2 document)
303
-
304
- - Fetching real medical/legal/financial documents
305
- - Building the query sets
306
- - Running the actual evaluation to generate paper numbers
307
- - Rewriting the paper's dataset section
308
-
309
- Phase 1 is architecture only. Once this is done and verified against the small existing `test_data/` samples, move to the Phase 2 document to bring in real data at the ~500-600 document scale.