Spaces:
Sleeping
Sleeping
Delete MEXAR_Phase1_Architecture_Fixes.md
Browse files
MEXAR_Phase1_Architecture_Fixes.md
DELETED
|
@@ -1,309 +0,0 @@
|
|
| 1 |
-
# MEXAR Phase 1 — Architecture Fixes to Match the Paper
|
| 2 |
-
|
| 3 |
-
**Purpose of this document:** hand this to an AI coding agent (Antigravity or similar) working directly in the `Mexar` repository (`github.com/devrajsinh2012/Mexar`). This is Phase 1 only: it closes the gap between what the paper *"Hybrid Retrieval and Claim Verification: An Empirical Study of Constrained RAG for High-Stakes Professional Domains"* claims about the system's architecture, and what the code currently does. No dataset work in this document, that's Phase 2, handled separately.
|
| 4 |
-
|
| 5 |
-
**Presentation date:** July 30-31, 2026. Target for this phase: 3 days.
|
| 6 |
-
|
| 7 |
-
**Ground rule for the agent:** every change below must keep the app deployable on the current stack (FastAPI backend on Hugging Face Spaces, Supabase Postgres + pgvector, React frontend on Vercel). Don't introduce services that need infra Dev doesn't already have.
|
| 8 |
-
|
| 9 |
-
---
|
| 10 |
-
|
| 11 |
-
## 1.0 Preconditions
|
| 12 |
-
|
| 13 |
-
- Add to `backend/requirements.txt`:
|
| 14 |
-
```
|
| 15 |
-
scikit-learn>=1.4.0
|
| 16 |
-
spacy>=3.7.0
|
| 17 |
-
```
|
| 18 |
-
- After install, download the small English model: `python -m spacy download en_core_web_sm`. Add this as a step in `Dockerfile` so it survives Hugging Face Spaces rebuilds.
|
| 19 |
-
- Verify the pgvector extension version on the Supabase project supports HNSW (needs pgvector >= 0.5.0). Run `SELECT extversion FROM pg_extension WHERE extname = 'vector';`. If below 0.5.0, run `ALTER EXTENSION vector UPDATE;` before touching the index migration in 1.2.
|
| 20 |
-
|
| 21 |
-
---
|
| 22 |
-
|
| 23 |
-
## 1.1 Domain Guardrail: replace the heuristic with real TF-IDF + NER + Jaccard
|
| 24 |
-
|
| 25 |
-
**Why:** Paper Section III-A defines the guardrail as TF-IDF term weighting (Eq. 1), NER-extracted domain entities, and Jaccard similarity between query and signature (Eq. 2). The current `_check_guardrail()` in `backend/modules/reasoning_engine.py` is a hand-tuned fuzzy-string-match heuristic with arbitrary bonus weights and no TF-IDF or NER anywhere. This needs to become the real thing.
|
| 26 |
-
|
| 27 |
-
**New file: `backend/utils/domain_signature.py`**
|
| 28 |
-
|
| 29 |
-
```python
|
| 30 |
-
"""
|
| 31 |
-
MEXAR - Domain Signature Construction
|
| 32 |
-
Implements Section III-A: TF-IDF lexical signature + NER entity signature.
|
| 33 |
-
"""
|
| 34 |
-
from typing import List, Dict, Set, Tuple
|
| 35 |
-
from sklearn.feature_extraction.text import TfidfVectorizer
|
| 36 |
-
import spacy
|
| 37 |
-
import logging
|
| 38 |
-
|
| 39 |
-
logger = logging.getLogger(__name__)
|
| 40 |
-
_nlp = None
|
| 41 |
-
|
| 42 |
-
def get_nlp():
|
| 43 |
-
global _nlp
|
| 44 |
-
if _nlp is None:
|
| 45 |
-
_nlp = spacy.load("en_core_web_sm")
|
| 46 |
-
return _nlp
|
| 47 |
-
|
| 48 |
-
# Entity labels relevant to professional domains (medical/legal/financial)
|
| 49 |
-
RELEVANT_ENTITY_LABELS = {"ORG", "LAW", "PRODUCT", "GPE", "PERSON", "NORP", "EVENT", "FAC"}
|
| 50 |
-
|
| 51 |
-
def build_tfidf_signature(documents: List[str], top_n: int = 100, tau_tf: float = 0.0) -> List[Tuple[str, float]]:
|
| 52 |
-
"""
|
| 53 |
-
Eq. 1: w(t, D) = tf(t, D) * log(|D| / |{d in D : t in d}|)
|
| 54 |
-
Returns top_n (term, weight) pairs above tau_tf, sorted descending by weight.
|
| 55 |
-
"""
|
| 56 |
-
if not documents or len(documents) < 2:
|
| 57 |
-
# TF-IDF needs a corpus of multiple "documents" to compute IDF meaningfully.
|
| 58 |
-
# If compiling from a single uploaded file, split it into paragraph-level
|
| 59 |
-
# pseudo-documents so IDF is not degenerate.
|
| 60 |
-
documents = _split_into_pseudo_documents(documents[0]) if documents else []
|
| 61 |
-
|
| 62 |
-
vectorizer = TfidfVectorizer(
|
| 63 |
-
max_features=2000,
|
| 64 |
-
stop_words="english",
|
| 65 |
-
ngram_range=(1, 2),
|
| 66 |
-
min_df=1,
|
| 67 |
-
)
|
| 68 |
-
matrix = vectorizer.fit_transform(documents)
|
| 69 |
-
scores = matrix.sum(axis=0).A1
|
| 70 |
-
terms = vectorizer.get_feature_names_out()
|
| 71 |
-
pairs = [(t, float(s)) for t, s in zip(terms, scores) if s > tau_tf]
|
| 72 |
-
pairs.sort(key=lambda x: x[1], reverse=True)
|
| 73 |
-
return pairs[:top_n]
|
| 74 |
-
|
| 75 |
-
def _split_into_pseudo_documents(text: str, chunk_size: int = 500) -> List[str]:
|
| 76 |
-
words = text.split()
|
| 77 |
-
return [" ".join(words[i:i+chunk_size]) for i in range(0, len(words), chunk_size)] or [text]
|
| 78 |
-
|
| 79 |
-
def extract_ner_entities(documents: List[str], tau_ent: int = 2, max_docs_to_scan: int = 200) -> List[str]:
|
| 80 |
-
"""
|
| 81 |
-
Extracts named entities appearing in at least tau_ent documents.
|
| 82 |
-
Capped at max_docs_to_scan for latency (spaCy is not free at compile time).
|
| 83 |
-
"""
|
| 84 |
-
nlp = get_nlp()
|
| 85 |
-
doc_freq: Dict[str, int] = {}
|
| 86 |
-
for doc_text in documents[:max_docs_to_scan]:
|
| 87 |
-
seen_in_this_doc: Set[str] = set()
|
| 88 |
-
for ent in nlp(doc_text[:5000]).ents: # cap per-doc length for speed
|
| 89 |
-
if ent.label_ in RELEVANT_ENTITY_LABELS:
|
| 90 |
-
key = ent.text.lower().strip()
|
| 91 |
-
if key and key not in seen_in_this_doc:
|
| 92 |
-
seen_in_this_doc.add(key)
|
| 93 |
-
for key in seen_in_this_doc:
|
| 94 |
-
doc_freq[key] = doc_freq.get(key, 0) + 1
|
| 95 |
-
return [k for k, v in doc_freq.items() if v >= tau_ent]
|
| 96 |
-
|
| 97 |
-
def build_domain_signature(
|
| 98 |
-
documents: List[str],
|
| 99 |
-
tau_tf: float = 0.0,
|
| 100 |
-
tau_ent: int = 2,
|
| 101 |
-
top_n_lexical: int = 100,
|
| 102 |
-
) -> Dict[str, List]:
|
| 103 |
-
"""Returns Sigma = Sigma_lex ∪ Sigma_ent as described in Section III-A, kept separate for inspection."""
|
| 104 |
-
lexical_pairs = build_tfidf_signature(documents, top_n=top_n_lexical, tau_tf=tau_tf)
|
| 105 |
-
lexical_terms = [t for t, _ in lexical_pairs]
|
| 106 |
-
entities = extract_ner_entities(documents, tau_ent=tau_ent)
|
| 107 |
-
return {
|
| 108 |
-
"lexical": lexical_terms,
|
| 109 |
-
"lexical_weights": {t: w for t, w in lexical_pairs},
|
| 110 |
-
"entities": entities,
|
| 111 |
-
"combined": sorted(set(lexical_terms) | set(e for e in entities)),
|
| 112 |
-
}
|
| 113 |
-
```
|
| 114 |
-
|
| 115 |
-
**Modify `backend/modules/knowledge_compiler.py`:**
|
| 116 |
-
Replace `_extract_domain_signature()` so it calls `build_domain_signature()` from the new module, passing in the raw per-source-document text list (not the flattened `text_context` string, we need per-document granularity for TF-IDF's document-frequency term to mean anything). Store the result's `combined` list in the existing `domain_signature` JSONB column, and add the `lexical_weights` dict to a **new** JSONB column (see migration below) so the guardrail can compute a weighted Jaccard rather than a flat one if useful later.
|
| 117 |
-
|
| 118 |
-
**New migration** (Alembic or raw SQL depending on how `backend/migrations` is structured, check `backend/migrations/README.md` for the convention already in use):
|
| 119 |
-
```sql
|
| 120 |
-
ALTER TABLE agents ADD COLUMN IF NOT EXISTS domain_signature_weights JSONB;
|
| 121 |
-
ALTER TABLE agents ADD COLUMN IF NOT EXISTS domain_entities JSONB;
|
| 122 |
-
```
|
| 123 |
-
|
| 124 |
-
**Rewrite `_check_guardrail()` in `backend/modules/reasoning_engine.py`:**
|
| 125 |
-
Delete the fuzzy-match/bonus-score implementation entirely. Replace with real Eq. 2:
|
| 126 |
-
|
| 127 |
-
```python
|
| 128 |
-
def _check_guardrail(self, query: str, domain_signature: List[str], prompt_analysis: Dict[str, Any]) -> Tuple[bool, float]:
|
| 129 |
-
"""Eq. 2: sim(q, Sigma) = |Phi(q) ∩ Sigma| / |Phi(q)|"""
|
| 130 |
-
from utils.domain_signature import get_nlp
|
| 131 |
-
nlp = get_nlp()
|
| 132 |
-
query_doc = nlp(query.lower())
|
| 133 |
-
# Phi(q): lemmatized content tokens, excluding stopwords/punctuation
|
| 134 |
-
phi_q = {
|
| 135 |
-
tok.lemma_ for tok in query_doc
|
| 136 |
-
if not tok.is_stop and not tok.is_punct and len(tok.text) > 2
|
| 137 |
-
}
|
| 138 |
-
sigma = set(s.lower() for s in (domain_signature or []))
|
| 139 |
-
|
| 140 |
-
if not phi_q:
|
| 141 |
-
return False, 0.0
|
| 142 |
-
|
| 143 |
-
intersection = phi_q & sigma
|
| 144 |
-
score = len(intersection) / len(phi_q)
|
| 145 |
-
|
| 146 |
-
is_in_domain = score >= self.DOMAIN_SIMILARITY_THRESHOLD
|
| 147 |
-
logger.info(f"Guardrail: sim={score:.3f}, |Phi(q)|={len(phi_q)}, |intersection|={len(intersection)}, in_domain={is_in_domain}")
|
| 148 |
-
return is_in_domain, score
|
| 149 |
-
```
|
| 150 |
-
|
| 151 |
-
**Tune `DOMAIN_SIMILARITY_THRESHOLD`:** the current 0.05 was calibrated against the *old* heuristic scorer, it will not transfer to real Jaccard similarity, which tends to produce lower absolute values on short queries. Don't guess this: after wiring the new guardrail, run `backend/evaluation/guardrail_analysis.py` against a small hand-labeled set of ~20 in-domain and ~20 out-of-domain queries per existing test_data domain, sweep the threshold from 0.05 to 0.4 in steps of 0.05, and pick the value that maximizes F1. Record that sweep, it becomes supporting evidence for Table IV in the paper instead of a guessed constant.
|
| 152 |
-
|
| 153 |
-
**Acceptance test:** for the medical test agent, a query like "what is the recommended treatment for stage 2 hypertension" should be accepted; a query like "how do I file a small claims lawsuit" should be rejected. Log both the old and new guardrail score during a transition period to sanity-check before removing the old code path.
|
| 154 |
-
|
| 155 |
-
---
|
| 156 |
-
|
| 157 |
-
## 1.2 Vector index: IVFFlat → HNSW
|
| 158 |
-
|
| 159 |
-
**Why:** Paper Section III-B explicitly claims HNSW indexing. The current migration (`backend/migrations/hybrid_search_function.sql`, lines ~98-101) creates an IVFFlat index instead.
|
| 160 |
-
|
| 161 |
-
**New migration file** `backend/migrations/switch_to_hnsw_index.sql`:
|
| 162 |
-
```sql
|
| 163 |
-
-- Drop the old IVFFlat index
|
| 164 |
-
DROP INDEX IF EXISTS idx_document_chunks_embedding;
|
| 165 |
-
|
| 166 |
-
-- Create HNSW index to match paper Section III-B
|
| 167 |
-
CREATE INDEX IF NOT EXISTS idx_document_chunks_embedding_hnsw
|
| 168 |
-
ON document_chunks USING hnsw (embedding vector_cosine_ops)
|
| 169 |
-
WITH (m = 16, ef_construction = 64);
|
| 170 |
-
|
| 171 |
-
COMMENT ON INDEX idx_document_chunks_embedding_hnsw IS
|
| 172 |
-
'HNSW index for cosine similarity search, m=16, ef_construction=64 (Section III-B)';
|
| 173 |
-
```
|
| 174 |
-
Apply via `Supabase:apply_migration` (or however existing migrations are run, check for a runner script referenced in `backend/migrations/README.md`) rather than editing the old file, so history is preserved.
|
| 175 |
-
|
| 176 |
-
**Note for the agent:** HNSW build time scales worse than IVFFlat with corpus size, but at the 500-600 document / few-thousand-chunk scale planned for the presentation, this is a non-issue. No further tuning needed.
|
| 177 |
-
|
| 178 |
-
**Acceptance test:** `\d document_chunks` in psql (or `SELECT indexdef FROM pg_indexes WHERE tablename = 'document_chunks';`) should show the HNSW index, not IVFFlat. Re-run a handful of existing queries and confirm retrieval still returns coherent top-5 chunks (HNSW is approximate, results may reorder slightly versus IVFFlat, that's expected and fine).
|
| 179 |
-
|
| 180 |
-
---
|
| 181 |
-
|
| 182 |
-
## 1.3 RRF weighting: reconcile code and paper
|
| 183 |
-
|
| 184 |
-
**Why:** Paper Eq. 5 is unweighted RRF. `backend/migrations/hybrid_search_function.sql` applies a 0.6 semantic / 0.4 keyword weighting inside the SQL function.
|
| 185 |
-
|
| 186 |
-
**Decision for this project (recommended, don't re-litigate):** keep the 0.6/0.4 weighting in code, it's a reasonable and defensible design choice, and update the paper instead. This is less risky than ripping out a tuned weighting under time pressure. The agent does not need to touch `hybrid_search_function.sql` for this item, just flag it in a "paper edits needed" note for later.
|
| 187 |
-
|
| 188 |
-
If Dev instead decides to match the paper exactly, the one-line change is setting `semantic_weight := 0.5; keyword_weight := 0.5;` in that SQL function, but this is **not** the default action, don't do it unless explicitly told.
|
| 189 |
-
|
| 190 |
-
---
|
| 191 |
-
|
| 192 |
-
## 1.4 Faithfulness verification: make DeBERTa-v3 NLI the real, primary signal
|
| 193 |
-
|
| 194 |
-
**Why:** this is the most important fix. Paper Section III-C and the entire Related Work argument against LLM-judging-LLM circularity claim the production system verifies claims with "an NLI model (DeBERTa-v3-large fine-tuned on MNLI)" using per-document max entailment (Eq. 6). In the actual code:
|
| 195 |
-
- The primary confidence-driving call in `reasoning_engine.reason()` (Step 6) is `self.faithfulness_scorer.score(answer, context)`, which is `FaithfulnessScorer` in `backend/utils/faithfulness.py`, an LLM (Groq) extracting claims and then Groq again answering YES/NO. That's exactly the circularity the paper argues against.
|
| 196 |
-
- The actual NLI model in the repo, `BartNLIScorer`, uses `facebook/bart-large-mnli`, not DeBERTa-v3-large, and it's computed only as a side "reviewer feedback" score (`bart_nli_result` in `reason()`), it never feeds the confidence calculation.
|
| 197 |
-
- The NLI scorer concatenates all retrieved context into a single premise string, it does not implement the per-document max in Eq. 6.
|
| 198 |
-
|
| 199 |
-
**Model choice:** use `cross-encoder/nli-deberta-v3-large` from Hugging Face. This is a DeBERTa-v3-large architecture fine-tuned specifically for NLI (SNLI + MultiNLI), it is the standard, citable checkpoint people mean when they say "DeBERTa-v3-large fine-tuned on MNLI" in RAG faithfulness papers, and it loads cleanly through `sentence-transformers`'s `CrossEncoder` class, which is already a dependency.
|
| 200 |
-
|
| 201 |
-
**Modify `backend/utils/faithfulness.py`:** replace the `BartNLIScorer` class with:
|
| 202 |
-
|
| 203 |
-
```python
|
| 204 |
-
class DebertaNLIScorer:
|
| 205 |
-
"""
|
| 206 |
-
Primary faithfulness verifier per Section III-C.
|
| 207 |
-
Per-claim, per-document max entailment probability (Eq. 6),
|
| 208 |
-
aggregated as the fraction of claims exceeding tau_ent (Eq. 7).
|
| 209 |
-
"""
|
| 210 |
-
LABEL_MAP = {0: "contradiction", 1: "entailment", 2: "neutral"} # verify against model card at load time
|
| 211 |
-
TAU_ENT = 0.7
|
| 212 |
-
|
| 213 |
-
def __init__(self):
|
| 214 |
-
self._model = None
|
| 215 |
-
|
| 216 |
-
@property
|
| 217 |
-
def model(self):
|
| 218 |
-
if self._model is None:
|
| 219 |
-
from sentence_transformers import CrossEncoder
|
| 220 |
-
logger.info("Loading cross-encoder/nli-deberta-v3-large...")
|
| 221 |
-
self._model = CrossEncoder("cross-encoder/nli-deberta-v3-large")
|
| 222 |
-
logger.info("DeBERTa-v3 NLI model loaded.")
|
| 223 |
-
return self._model
|
| 224 |
-
|
| 225 |
-
def score(self, answer: str, context_documents: List[str]) -> FaithfulnessResult:
|
| 226 |
-
"""
|
| 227 |
-
context_documents: list of individual retrieved chunk texts (NOT concatenated),
|
| 228 |
-
so per-document max in Eq. 6 is meaningful.
|
| 229 |
-
"""
|
| 230 |
-
import re
|
| 231 |
-
import numpy as np
|
| 232 |
-
|
| 233 |
-
if not answer or not context_documents:
|
| 234 |
-
return FaithfulnessResult(score=1.0, total_claims=0, supported_claims=0, unsupported_claims=[])
|
| 235 |
-
|
| 236 |
-
sentences = [s.strip() for s in re.split(r'(?<=[.!?])\s+', answer) if len(s.strip()) > 15][:10]
|
| 237 |
-
if not sentences:
|
| 238 |
-
return FaithfulnessResult(score=1.0, total_claims=0, supported_claims=0, unsupported_claims=[])
|
| 239 |
-
|
| 240 |
-
supported = 0
|
| 241 |
-
unsupported = []
|
| 242 |
-
|
| 243 |
-
for sentence in sentences:
|
| 244 |
-
pairs = [(doc[:2000], sentence) for doc in context_documents]
|
| 245 |
-
scores = self.model.predict(pairs) # shape: (n_docs, 3) logits/probs per label
|
| 246 |
-
probs = self._softmax(scores)
|
| 247 |
-
entailment_idx = 1 # confirm this matches the model's actual label order at integration time
|
| 248 |
-
max_entailment = float(np.max(probs[:, entailment_idx]))
|
| 249 |
-
if max_entailment > self.TAU_ENT:
|
| 250 |
-
supported += 1
|
| 251 |
-
else:
|
| 252 |
-
unsupported.append(sentence)
|
| 253 |
-
|
| 254 |
-
score = supported / len(sentences)
|
| 255 |
-
return FaithfulnessResult(
|
| 256 |
-
score=round(score, 3),
|
| 257 |
-
total_claims=len(sentences),
|
| 258 |
-
supported_claims=supported,
|
| 259 |
-
unsupported_claims=unsupported[:5],
|
| 260 |
-
)
|
| 261 |
-
|
| 262 |
-
@staticmethod
|
| 263 |
-
def _softmax(x):
|
| 264 |
-
import numpy as np
|
| 265 |
-
e_x = np.exp(x - np.max(x, axis=-1, keepdims=True))
|
| 266 |
-
return e_x / e_x.sum(axis=-1, keepdims=True)
|
| 267 |
-
```
|
| 268 |
-
|
| 269 |
-
**Important integration detail for the agent:** verify `cross-encoder/nli-deberta-v3-large`'s actual `id2label` mapping from its `config.json` on first load and print it to logs, cross-encoder NLI checkpoints don't always use the same label order. Don't hardcode `entailment_idx = 1` blindly, confirm it, this is exactly the kind of silent bug that would quietly invalidate the reported faithfulness numbers.
|
| 270 |
-
|
| 271 |
-
**Modify `backend/modules/reasoning_engine.py`, `reason()` method:**
|
| 272 |
-
- Rename the old `FaithfulnessScorer` usage. Keep the class in the codebase (don't delete it), but relabel its role: it becomes the **"LLM-as-judge baseline"** used for baseline comparison tables (this is genuinely useful, RAGAS-style LLM self-eval is a legitimate baseline to compare against in Table I/III, just not the production signal).
|
| 273 |
-
- Swap Step 6 to call `self.deberta_nli_scorer.score(answer, [c.content for c in top_chunks])` (passing the list of individual chunks, not the concatenated `context` string) as the **primary** `faithfulness_result` used in `_calculate_confidence()` and in the returned `explainability` payload.
|
| 274 |
-
- Keep computing the old LLM-based score too, but store it under a clearly-labeled `"llm_judge_baseline_score"` key in `explainability.confidence_breakdown`, not as the headline faithfulness number.
|
| 275 |
-
|
| 276 |
-
**Acceptance test:** feed a known-hallucinated answer (write one manually, e.g. an answer that states a fabricated statistic not present in any retrieved chunk) through the pipeline and confirm the DeBERTa scorer flags it as unsupported. Feed a fully-grounded answer and confirm it scores above `TAU_ENT`. Compare against the LLM-judge baseline score on the same pair, they should usually agree, but if they diverge a lot on the same input, that's worth investigating before trusting either for the paper's numbers.
|
| 277 |
-
|
| 278 |
-
---
|
| 279 |
-
|
| 280 |
-
## 1.5 Wiring check
|
| 281 |
-
|
| 282 |
-
After 1.1-1.4 are done, re-run the existing `backend/evaluation/baseline_runner.py` and `backend/evaluation/benchmark_runner.py` against the current small `test_data/` samples (not for final numbers, just as a smoke test) and confirm nothing throws. Fix any interface mismatches from the `context: str` → `context_documents: List[str]` signature change in the faithfulness scorer, other call sites (`evaluation/metrics.py`, `_run_baseline()` in `reasoning_engine.py`) pass a concatenated string today and will need the same list-of-chunks treatment.
|
| 283 |
-
|
| 284 |
-
---
|
| 285 |
-
|
| 286 |
-
## Summary of files touched in Phase 1
|
| 287 |
-
|
| 288 |
-
| File | Change |
|
| 289 |
-
|---|---|
|
| 290 |
-
| `backend/requirements.txt` | add scikit-learn, spacy |
|
| 291 |
-
| `backend/Dockerfile` | add spacy model download step |
|
| 292 |
-
| `backend/utils/domain_signature.py` | **new** — TF-IDF + NER signature builder |
|
| 293 |
-
| `backend/modules/knowledge_compiler.py` | `_extract_domain_signature()` rewritten to use new module |
|
| 294 |
-
| `backend/modules/reasoning_engine.py` | `_check_guardrail()` rewritten to real Jaccard; Step 6 rewired to DeBERTa scorer |
|
| 295 |
-
| `backend/utils/faithfulness.py` | `BartNLIScorer` replaced with `DebertaNLIScorer` (per-doc max, Eq. 6/7) |
|
| 296 |
-
| `backend/migrations/switch_to_hnsw_index.sql` | **new** — HNSW index migration |
|
| 297 |
-
| `backend/migrations/*` (new file) | add `domain_signature_weights`, `domain_entities` columns |
|
| 298 |
-
| `backend/evaluation/*` | update call sites for the `context: str` → `context_documents: List[str]` signature change |
|
| 299 |
-
|
| 300 |
-
---
|
| 301 |
-
|
| 302 |
-
## What Phase 1 does NOT cover (handled in the separate Phase 2 document)
|
| 303 |
-
|
| 304 |
-
- Fetching real medical/legal/financial documents
|
| 305 |
-
- Building the query sets
|
| 306 |
-
- Running the actual evaluation to generate paper numbers
|
| 307 |
-
- Rewriting the paper's dataset section
|
| 308 |
-
|
| 309 |
-
Phase 1 is architecture only. Once this is done and verified against the small existing `test_data/` samples, move to the Phase 2 document to bring in real data at the ~500-600 document scale.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|