Spaces:
Sleeping
Sleeping
Devdan Schretlen commited on
Commit ·
494a4bf
1
Parent(s): 46de6a9
Added RAGChef RAG engine and eval harness
Browse files- README.md +175 -0
- ask.py +94 -0
- eval/__init__.py +1 -0
- eval/judge.py +290 -0
- eval/make_testset.py +127 -0
- eval/reports/history.csv +4 -0
- eval/reports/report.md +136 -0
- eval/retrieval_metrics.py +138 -0
- eval/run_ablation.py +249 -0
- eval/run_eval.py +293 -0
- eval/testset.general.json +33 -0
- eval/testset.json +101 -0
- src/__init__.py +6 -0
- src/agent.py +96 -0
- src/bm25.py +115 -0
- src/generate.py +174 -0
- src/pipeline.py +42 -0
- src/rerank.py +68 -0
- src/retrieve.py +162 -0
- src/router.py +38 -0
- src/trace.py +73 -0
README.md
ADDED
|
@@ -0,0 +1,175 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# UBCHelper — RAG over the UBC Vancouver Academic Calendar
|
| 2 |
+
|
| 3 |
+
A retrieval-augmented **academic-calendar advisor**. Ask a question about UBC Vancouver
|
| 4 |
+
courses, programs, or policies → the system retrieves the relevant calendar excerpts from a
|
| 5 |
+
local index → an LLM writes a grounded answer that points you at the official calendar page.
|
| 6 |
+
Embeddings run on the **OpenAI API**, reranking on the **Cohere API**, and generation on
|
| 7 |
+
**Claude (Anthropic API)** by default — with an optional local generation backend via LM Studio.
|
| 8 |
+
|
| 9 |
+
A domain re-skin of [RAGChef](../RAGChef): the same from-scratch engine (no
|
| 10 |
+
LangChain/LlamaIndex) — hybrid dense + BM25 retrieval, RRF fusion, hosted rerank, an adaptive
|
| 11 |
+
router with a decompose→multi-hop→synthesize agent, and a two-tier eval harness — pointed at a
|
| 12 |
+
scraped calendar corpus instead of a recipe CSV.
|
| 13 |
+
|
| 14 |
+
```
|
| 15 |
+
┌────────────────────────── run once ──────────────────────────────┐
|
| 16 |
+
scrape.py ──► data/pages/ ──► chunk.py ──► ingest.py ──► index/ (numpy + json)
|
| 17 |
+
└──────────────────────────────────────────────────────────────────┘
|
| 18 |
+
|
| 19 |
+
"What are the prerequisites for CPSC 210?"
|
| 20 |
+
│
|
| 21 |
+
▼
|
| 22 |
+
retrieve.py ──┬─► dense — embed the query (OpenAI API), cosine search ─► ~20 ──┐
|
| 23 |
+
└─► sparse — BM25 keyword search (src/bm25.py) ─► ~20 ──┤
|
| 24 |
+
▼
|
| 25 |
+
RRF fusion ──► rerank (Cohere API, src/rerank.py) ──► top-k
|
| 26 |
+
│
|
| 27 |
+
▼
|
| 28 |
+
generate.py ──► grounded prompt ──► Claude (Anthropic API) ──► answer + source URL
|
| 29 |
+
```
|
| 30 |
+
|
| 31 |
+
Complex questions can take the adaptive path instead (`ask.py --adaptive`): a router
|
| 32 |
+
classifies the question, and **complex** ones go through the agent — decompose into
|
| 33 |
+
sub-questions → retrieve each → merge → synthesize one answer.
|
| 34 |
+
|
| 35 |
+
---
|
| 36 |
+
|
| 37 |
+
## The corpus (and why it's deliberately tricky)
|
| 38 |
+
|
| 39 |
+
`src/scrape.py` crawls the calendar into `data/pages/` — one JSON record per page, with
|
| 40 |
+
provenance (canonical URL, edition, fetch time, content hash). Two editions are collected:
|
| 41 |
+
|
| 42 |
+
- **live 2026/27** — `vancouver.calendar.ubc.ca` (sitemap-driven): the Faculty of Science
|
| 43 |
+
subtree, campus-wide policies, all cohort-split programs, and course-description pages.
|
| 44 |
+
- **archive 2025/26** — `archive.calendar.ubc.ca/vancouver/2526/...`: a narrow slice of the
|
| 45 |
+
same subtrees where the editions collide.
|
| 46 |
+
|
| 47 |
+
That gives the corpus two built-in traps that naive top-k RAG falls into: **edition
|
| 48 |
+
collisions** (near-identical pages differing only in calendar year) and **cohort splits**
|
| 49 |
+
(e.g. B.A. requirements for students entering 2023/24-or-earlier vs 2024/25-or-later).
|
| 50 |
+
`src/chunk.py` folds the disambiguators into every chunk's title and text — e.g.
|
| 51 |
+
`Bachelor of Arts — Degree Requirements ... (2024/25 or later) [2026/27]` — and carries them
|
| 52 |
+
as structured metadata, so retrieval, prompts, and the eval can all tell the twins apart.
|
| 53 |
+
|
| 54 |
+
**Scraping etiquette / Terms of Use:** the crawl honors robots.txt (10 s crawl delay), sends
|
| 55 |
+
a descriptive User-Agent, caches every page (re-runs use conditional GETs), and the scraped
|
| 56 |
+
content is **never committed or redistributed** — `data/pages/` is gitignored; only the
|
| 57 |
+
scraper and engine code live in the repo. Rebuild the corpus locally with:
|
| 58 |
+
|
| 59 |
+
```powershell
|
| 60 |
+
python -m src.scrape --dry-run # enumerate in-scope URLs, fetch nothing
|
| 61 |
+
python -m src.scrape --sample # just the 5 representative sample pages (quick start)
|
| 62 |
+
python -m src.scrape # full in-scope crawl (slow by design: 10 s/request)
|
| 63 |
+
```
|
| 64 |
+
|
| 65 |
+
---
|
| 66 |
+
|
| 67 |
+
## Setup
|
| 68 |
+
|
| 69 |
+
```powershell
|
| 70 |
+
py -m venv .venv
|
| 71 |
+
.\.venv\Scripts\Activate.ps1
|
| 72 |
+
pip install -r requirements.txt
|
| 73 |
+
cp .env.example .env # then edit .env
|
| 74 |
+
```
|
| 75 |
+
|
| 76 |
+
- `OPENAI_API_KEY` — **required**; embeddings run on OpenAI (`text-embedding-3-small`).
|
| 77 |
+
- `COHERE_API_KEY` — **required** for the default `hybrid_rerank` mode (`rerank-v3.5`).
|
| 78 |
+
Not needed with `UBCAL_RETRIEVAL_MODE=dense` or `hybrid`.
|
| 79 |
+
- `ANTHROPIC_API_KEY` — **required** for the default generation backend (Claude Haiku 4.5).
|
| 80 |
+
|
| 81 |
+
> The embedding model used to **build** the index and the one used to **query** it must be
|
| 82 |
+
> the same, or retrieval is meaningless. Re-run ingest if you change `UBCAL_EMBED_MODEL`.
|
| 83 |
+
|
| 84 |
+
Optional local generation (embeddings/rerank stay hosted): start any OpenAI-compatible
|
| 85 |
+
server and set `UBCAL_LLM_BACKEND=local`, `UBCAL_BASE_URL_LLM`, `UBCAL_CHAT_MODEL`.
|
| 86 |
+
|
| 87 |
+
## Run it
|
| 88 |
+
|
| 89 |
+
```powershell
|
| 90 |
+
python -m src.ingest # chunk + embed the cached pages -> index/
|
| 91 |
+
python ask.py "What are the prerequisites for CPSC 210?"
|
| 92 |
+
python ask.py --show-context "What is the academic standing policy?"
|
| 93 |
+
python ask.py --adaptive --trace "Compare the B.A. degree requirements for students who entered in 2023/24 vs 2024/25"
|
| 94 |
+
```
|
| 95 |
+
|
| 96 |
+
Example output:
|
| 97 |
+
|
| 98 |
+
```
|
| 99 |
+
Retrieved excerpts:
|
| 100 |
+
[0.818] (id=8) Computer Science, Faculty of Science [2026/27]
|
| 101 |
+
[0.808] (id=179) Computer Science, Faculty of Science [2025/26]
|
| 102 |
+
...
|
| 103 |
+
|
| 104 |
+
Answer:
|
| 105 |
+
The prerequisites for CPSC 210 (Software Construction) are one of: CPSC 107 or CPSC 110.
|
| 106 |
+
For the current 2026/27 calendar see:
|
| 107 |
+
https://vancouver.calendar.ubc.ca/course-descriptions/subject/cpscv
|
| 108 |
+
```
|
| 109 |
+
|
| 110 |
+
## The code
|
| 111 |
+
|
| 112 |
+
| File | Role |
|
| 113 |
+
|------|------|
|
| 114 |
+
| [src/config.py](src/config.py) | Every tunable + path (env prefix `UBCAL_`). |
|
| 115 |
+
| [src/scrape.py](src/scrape.py) | Polite sitemap-driven crawler → `data/pages/` + manifest. |
|
| 116 |
+
| [src/chunk.py](src/chunk.py) | Page HTML → markdown-ish chunk records (tables kept whole, cohort/edition metadata parsed). |
|
| 117 |
+
| [src/ingest.py](src/ingest.py) | Cached pages → chunks → embeddings → saved index. |
|
| 118 |
+
| [src/embed.py](src/embed.py) | Shared embedding function (same model for index & query). |
|
| 119 |
+
| [src/retrieve.py](src/retrieve.py) | The funnel: dense + BM25 → RRF fuse → rerank → top-k (with ids). |
|
| 120 |
+
| [src/bm25.py](src/bm25.py) | Hand-rolled BM25 Okapi keyword search (numpy). |
|
| 121 |
+
| [src/rerank.py](src/rerank.py) | Reranker via Cohere's hosted rerank API. |
|
| 122 |
+
| [src/generate.py](src/generate.py) | Excerpts + question → grounded prompt → answer with source URLs. |
|
| 123 |
+
| [src/router.py](src/router.py) | Classifies each question simple vs complex. |
|
| 124 |
+
| [src/agent.py](src/agent.py) | Complex path: decompose → multi-hop retrieve → merge → synthesize. |
|
| 125 |
+
| [src/pipeline.py](src/pipeline.py) | The one adaptive entry point (`answer()`). |
|
| 126 |
+
| [ask.py](ask.py) | The CLI that wires it all together. |
|
| 127 |
+
|
| 128 |
+
Design notes carried over from RAGChef: brute-force cosine search on a numpy matrix instead
|
| 129 |
+
of a vector DB (transparent, instant at this scale); BM25 built in memory from the indexed
|
| 130 |
+
metadata; hosted rerank so there's no local torch. New here: calendar pages are long and
|
| 131 |
+
heterogeneous, so unlike recipes they're **chunked** — split on headings, capped at
|
| 132 |
+
~`UBCAL_CHUNK_MAX_CHARS`, requirement tables never split. BM25 matters even more in this
|
| 133 |
+
domain: course codes ("CPSC 320", "BUCS") are exactly the rare, high-signal tokens dense
|
| 134 |
+
embeddings under-weight.
|
| 135 |
+
|
| 136 |
+
```powershell
|
| 137 |
+
$env:UBCAL_RETRIEVAL_MODE = "dense" # baseline (pure vector search)
|
| 138 |
+
$env:UBCAL_RETRIEVAL_MODE = "hybrid" # dense + BM25, RRF-fused
|
| 139 |
+
$env:UBCAL_RETRIEVAL_MODE = "hybrid_rerank" # + hosted rerank (the default)
|
| 140 |
+
```
|
| 141 |
+
|
| 142 |
+
## Evaluation
|
| 143 |
+
|
| 144 |
+
The two-tier harness scores the frozen test set through the same retrieve → generate
|
| 145 |
+
pipeline: **Tier 1** deterministic retrieval metrics (`hit@k`, `recall@k`, `MRR` —
|
| 146 |
+
[eval/retrieval_metrics.py](eval/retrieval_metrics.py)) and **Tier 2** RAGAS-style judged
|
| 147 |
+
metrics (faithfulness, answer relevancy, context precision/recall —
|
| 148 |
+
[eval/judge.py](eval/judge.py)); the headline number is hallucination rate = 1 − faithfulness.
|
| 149 |
+
[eval/run_ablation.py](eval/run_ablation.py) additionally answers every question closed-book
|
| 150 |
+
to measure what retrieval is actually worth.
|
| 151 |
+
|
| 152 |
+
```powershell
|
| 153 |
+
# Generate candidate questions from the indexed corpus, then CURATE BY HAND:
|
| 154 |
+
python -m eval.make_testset --num 20
|
| 155 |
+
|
| 156 |
+
# The before/after retrieval sweep (fast, deterministic):
|
| 157 |
+
$env:UBCAL_RETRIEVAL_MODE="dense"; python -m eval.run_eval --retrieval-only --name baseline-dense
|
| 158 |
+
$env:UBCAL_RETRIEVAL_MODE="hybrid"; python -m eval.run_eval --retrieval-only --name hybrid
|
| 159 |
+
$env:UBCAL_RETRIEVAL_MODE="hybrid_rerank"; python -m eval.run_eval --retrieval-only --name hybrid-rerank
|
| 160 |
+
|
| 161 |
+
# Full judged run + ablation (judge defaults to a local LM Studio model — see UBCAL_JUDGE_MODEL):
|
| 162 |
+
python -m eval.run_eval --name full
|
| 163 |
+
python -m eval.run_ablation --name rag-vs-norag
|
| 164 |
+
```
|
| 165 |
+
|
| 166 |
+
The test set targets three deliberately hard categories: **code-lookup** (hinges on a course
|
| 167 |
+
or program code — BM25's job), **multi-hop** (prerequisite/requirement chains spanning pages —
|
| 168 |
+
the reranker's and agent's job), and **edition/cohort-collision** (near-duplicate pages
|
| 169 |
+
differing only by calendar year or student cohort — the trap this corpus was built to test).
|
| 170 |
+
Chunk ids are positional, so test sets are only valid against the index they were generated
|
| 171 |
+
from — regenerate after any re-ingest that changes the corpus.
|
| 172 |
+
|
| 173 |
+
> **The calendar is authoritative and changes.** Answers always carry the official calendar
|
| 174 |
+
> URL; verify anything that matters there. This is an educational RAG project, not academic
|
| 175 |
+
> advising.
|
ask.py
ADDED
|
@@ -0,0 +1,94 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""
|
| 2 |
+
ask.py — the tiny CLI that runs the whole naive-RAG loop for one question.
|
| 3 |
+
|
| 4 |
+
retrieve(question) -> generate_answer(question, excerpts) -> print
|
| 5 |
+
|
| 6 |
+
Usage (from the project root, with the venv active):
|
| 7 |
+
python ask.py "What are the prerequisites for CPSC 210?"
|
| 8 |
+
python ask.py --top-k 6 "How many credits does the B.Sc. require?"
|
| 9 |
+
python ask.py --show-context "What is the academic standing policy?"
|
| 10 |
+
|
| 11 |
+
Prerequisites:
|
| 12 |
+
1. Keys in `.env` (copy `.env.example`): OPENAI_API_KEY for query embeddings and
|
| 13 |
+
ANTHROPIC_API_KEY for generation (the default backend).
|
| 14 |
+
2. The index built once: python -m src.ingest
|
| 15 |
+
"""
|
| 16 |
+
|
| 17 |
+
from __future__ import annotations
|
| 18 |
+
|
| 19 |
+
import argparse
|
| 20 |
+
|
| 21 |
+
from src import config, generate, pipeline, retrieve, trace
|
| 22 |
+
|
| 23 |
+
|
| 24 |
+
def main() -> None:
|
| 25 |
+
parser = argparse.ArgumentParser(description="Ask the UBC academic-calendar RAG advisor a question.")
|
| 26 |
+
parser.add_argument("question", help="your calendar question (wrap it in quotes)")
|
| 27 |
+
parser.add_argument(
|
| 28 |
+
"--top-k",
|
| 29 |
+
type=int,
|
| 30 |
+
default=config.TOP_K,
|
| 31 |
+
help=f"how many calendar excerpts to retrieve (default: {config.TOP_K})",
|
| 32 |
+
)
|
| 33 |
+
parser.add_argument(
|
| 34 |
+
"--show-context",
|
| 35 |
+
action="store_true",
|
| 36 |
+
help="also print the full text of the calendar excerpts sent to the model",
|
| 37 |
+
)
|
| 38 |
+
parser.add_argument(
|
| 39 |
+
"--mode",
|
| 40 |
+
choices=["dense", "hybrid", "hybrid_rerank"],
|
| 41 |
+
default=config.RETRIEVAL_MODE,
|
| 42 |
+
help=f"retrieval pipeline to use (default: {config.RETRIEVAL_MODE})",
|
| 43 |
+
)
|
| 44 |
+
parser.add_argument(
|
| 45 |
+
"--adaptive",
|
| 46 |
+
action="store_true",
|
| 47 |
+
help="route the question (Phase 4): simple -> fast retriever, complex -> agent",
|
| 48 |
+
)
|
| 49 |
+
parser.add_argument(
|
| 50 |
+
"--trace",
|
| 51 |
+
action="store_true",
|
| 52 |
+
help="print every internal step and the exact prompts sent to the APIs "
|
| 53 |
+
"(most informative with --adaptive)",
|
| 54 |
+
)
|
| 55 |
+
args = parser.parse_args()
|
| 56 |
+
|
| 57 |
+
if args.trace:
|
| 58 |
+
trace.enable()
|
| 59 |
+
|
| 60 |
+
# 1. RETRIEVE (+ optionally route). --adaptive sends the question through the Phase-4
|
| 61 |
+
# pipeline, which classifies it and picks the simple or agentic path; otherwise we run
|
| 62 |
+
# the plain retrieve -> generate loop with the chosen --mode.
|
| 63 |
+
if args.adaptive:
|
| 64 |
+
print("\nThinking ...\n")
|
| 65 |
+
result = pipeline.answer(args.question, top_k=args.top_k, mode=args.mode)
|
| 66 |
+
results, answer = result["results"], result["answer"]
|
| 67 |
+
print(f"Route: {result['route']}")
|
| 68 |
+
else:
|
| 69 |
+
results = retrieve.retrieve(args.question, top_k=args.top_k, mode=args.mode)
|
| 70 |
+
answer = None
|
| 71 |
+
|
| 72 |
+
# Show which excerpts grounded the answer (titles + similarity scores). This makes
|
| 73 |
+
# the retrieval step visible, which is the whole point of the learning exercise.
|
| 74 |
+
print("\nRetrieved excerpts:")
|
| 75 |
+
for r in results:
|
| 76 |
+
print(f" [{r.score:.3f}] (id={r.id}) {r.recipe['title']}")
|
| 77 |
+
|
| 78 |
+
if args.show_context:
|
| 79 |
+
print("\n--- context sent to the model ---")
|
| 80 |
+
print(generate.format_context(results))
|
| 81 |
+
print("--- end context ---")
|
| 82 |
+
|
| 83 |
+
# 2. GENERATE — have the LLM answer, grounded in those excerpts (the adaptive path
|
| 84 |
+
# already produced the answer above).
|
| 85 |
+
if answer is None:
|
| 86 |
+
print("\nThinking ...\n")
|
| 87 |
+
answer = generate.generate_answer(args.question, results)
|
| 88 |
+
|
| 89 |
+
print("\nAnswer:")
|
| 90 |
+
print(answer)
|
| 91 |
+
|
| 92 |
+
|
| 93 |
+
if __name__ == "__main__":
|
| 94 |
+
main()
|
eval/__init__.py
ADDED
|
@@ -0,0 +1 @@
|
|
|
|
|
|
|
| 1 |
+
"""Evaluation harness (Phase 3) — retrieval metrics now, RAGAS-style judge next."""
|
eval/judge.py
ADDED
|
@@ -0,0 +1,290 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""
|
| 2 |
+
Tier 2 — RAGAS-style generation metrics, judged by the local LLM.
|
| 3 |
+
|
| 4 |
+
These four metrics follow the RAGAS definitions, but instead of pulling in the heavy
|
| 5 |
+
RAGAS + LangChain dependency tree (which fights hard on new Python interpreters), each
|
| 6 |
+
metric is a small, self-contained prompt to the judge model via the same OpenAI client we
|
| 7 |
+
use everywhere. Fully transparent, works on Python 3.14, no extra dependencies.
|
| 8 |
+
|
| 9 |
+
faithfulness fraction of the ANSWER's claims that are supported by the
|
| 10 |
+
retrieved excerpts. Hallucination rate = 1 - faithfulness.
|
| 11 |
+
answer_relevancy how well the answer actually addresses the question (RAGAS trick:
|
| 12 |
+
generate questions the answer fits, embed them, compare to the real
|
| 13 |
+
question via cosine similarity).
|
| 14 |
+
context_precision of the retrieved excerpts, what fraction were relevant — rank-aware,
|
| 15 |
+
so relevant excerpts ranked higher score better.
|
| 16 |
+
context_recall does the retrieved context contain what's needed to reproduce the
|
| 17 |
+
reference answer (fraction of reference claims it supports).
|
| 18 |
+
|
| 19 |
+
Every score is in [0, 1], higher is better. The judge runs at temperature 0 for
|
| 20 |
+
determinism, and bad/unparseable JSON from the model degrades one metric to None for that
|
| 21 |
+
question rather than crashing the whole run.
|
| 22 |
+
|
| 23 |
+
NOTE: these are *directionally faithful* to canonical RAGAS, not numerically identical —
|
| 24 |
+
ideal for tracking relative change as you tune the system, which is the point.
|
| 25 |
+
"""
|
| 26 |
+
|
| 27 |
+
from __future__ import annotations
|
| 28 |
+
|
| 29 |
+
import json
|
| 30 |
+
import re
|
| 31 |
+
import time
|
| 32 |
+
|
| 33 |
+
import numpy as np
|
| 34 |
+
from openai import APIError, OpenAI
|
| 35 |
+
|
| 36 |
+
from src import config, embed
|
| 37 |
+
|
| 38 |
+
# Judge settings. Temperature 0 -> as deterministic as the model allows.
|
| 39 |
+
JUDGE_TEMPERATURE = 0.0
|
| 40 |
+
JUDGE_MAX_TOKENS = 1536 # headroom so long claim arrays don't truncate into invalid JSON
|
| 41 |
+
RELEVANCY_NUM_QUESTIONS = 3 # how many questions to reverse-generate for answer_relevancy
|
| 42 |
+
|
| 43 |
+
_JSON_SYSTEM = "You are a strict evaluator. Respond with ONLY valid JSON — no prose, no markdown, no code fences."
|
| 44 |
+
|
| 45 |
+
# LM Studio constrains generation to a JSON *schema* (OpenAI "structured outputs" style),
|
| 46 |
+
# which guarantees both valid JSON and the exact shape each metric expects — so the model
|
| 47 |
+
# can't emit prose, fences, trailing commas, or the wrong keys. One schema per output shape.
|
| 48 |
+
def _claims_schema(verdict_key: str) -> dict:
|
| 49 |
+
"""Schema for {"claims": [{"claim": str, <verdict_key>: bool}, ...]} (faithfulness/recall)."""
|
| 50 |
+
return {
|
| 51 |
+
"type": "object",
|
| 52 |
+
"additionalProperties": False,
|
| 53 |
+
"properties": {
|
| 54 |
+
"claims": {
|
| 55 |
+
"type": "array",
|
| 56 |
+
"items": {
|
| 57 |
+
"type": "object",
|
| 58 |
+
"additionalProperties": False,
|
| 59 |
+
"properties": {
|
| 60 |
+
"claim": {"type": "string"},
|
| 61 |
+
verdict_key: {"type": "boolean"},
|
| 62 |
+
},
|
| 63 |
+
"required": ["claim", verdict_key],
|
| 64 |
+
},
|
| 65 |
+
}
|
| 66 |
+
},
|
| 67 |
+
"required": ["claims"],
|
| 68 |
+
}
|
| 69 |
+
|
| 70 |
+
|
| 71 |
+
_QUESTIONS_SCHEMA = {
|
| 72 |
+
"type": "object",
|
| 73 |
+
"additionalProperties": False,
|
| 74 |
+
"properties": {"questions": {"type": "array", "items": {"type": "string"}}},
|
| 75 |
+
"required": ["questions"],
|
| 76 |
+
}
|
| 77 |
+
|
| 78 |
+
_VERDICTS_SCHEMA = {
|
| 79 |
+
"type": "object",
|
| 80 |
+
"additionalProperties": False,
|
| 81 |
+
"properties": {"verdicts": {"type": "array", "items": {"type": "boolean"}}},
|
| 82 |
+
"required": ["verdicts"],
|
| 83 |
+
}
|
| 84 |
+
|
| 85 |
+
_client: OpenAI | None = None
|
| 86 |
+
|
| 87 |
+
|
| 88 |
+
class JudgeParseError(Exception):
|
| 89 |
+
"""Raised when the judge's reply can't be parsed into the expected JSON."""
|
| 90 |
+
|
| 91 |
+
|
| 92 |
+
def _get_client() -> OpenAI:
|
| 93 |
+
global _client
|
| 94 |
+
if _client is None:
|
| 95 |
+
_client = OpenAI(base_url=config.OPENAI_BASE_URL, api_key=config.OPENAI_API_KEY)
|
| 96 |
+
return _client
|
| 97 |
+
|
| 98 |
+
|
| 99 |
+
def _extract_json(text: str):
|
| 100 |
+
"""Best-effort JSON extraction from a chat reply.
|
| 101 |
+
|
| 102 |
+
Local models often wrap JSON in ```code fences``` or add a sentence around it, so we
|
| 103 |
+
try a direct parse, then strip fences, then fall back to the outermost {...} or [...].
|
| 104 |
+
"""
|
| 105 |
+
s = (text or "").strip()
|
| 106 |
+
if s.startswith("```"):
|
| 107 |
+
s = re.sub(r"^```[a-zA-Z]*\s*", "", s)
|
| 108 |
+
s = re.sub(r"\s*```$", "", s).strip()
|
| 109 |
+
try:
|
| 110 |
+
return json.loads(s)
|
| 111 |
+
except json.JSONDecodeError:
|
| 112 |
+
pass
|
| 113 |
+
for open_ch, close_ch in (("{", "}"), ("[", "]")):
|
| 114 |
+
start, end = s.find(open_ch), s.rfind(close_ch)
|
| 115 |
+
if start != -1 and end > start:
|
| 116 |
+
try:
|
| 117 |
+
return json.loads(s[start : end + 1])
|
| 118 |
+
except json.JSONDecodeError:
|
| 119 |
+
continue
|
| 120 |
+
raise JudgeParseError(text)
|
| 121 |
+
|
| 122 |
+
|
| 123 |
+
def _with_retries(fn, attempts: int = 3, base_delay: float = 1.0):
|
| 124 |
+
"""Call fn(), retrying transient LM Studio API errors with linear backoff.
|
| 125 |
+
|
| 126 |
+
LM Studio can JIT-unload an idle model and return a transient 400 "Model unloaded"
|
| 127 |
+
(an APIError); the next call reloads it. Without this, one blip sinks a long run.
|
| 128 |
+
"""
|
| 129 |
+
for attempt in range(1, attempts + 1):
|
| 130 |
+
try:
|
| 131 |
+
return fn()
|
| 132 |
+
except APIError:
|
| 133 |
+
if attempt == attempts:
|
| 134 |
+
raise
|
| 135 |
+
time.sleep(base_delay * attempt)
|
| 136 |
+
|
| 137 |
+
|
| 138 |
+
def _chat_json(prompt: str, schema: dict, name: str):
|
| 139 |
+
"""Send one judging prompt and return the parsed JSON, shape-constrained by `schema`.
|
| 140 |
+
|
| 141 |
+
response_format=json_schema makes LM Studio grammar-constrain the output to that exact
|
| 142 |
+
schema, so the judge can't emit prose/fences/trailing commas or the wrong keys. We keep
|
| 143 |
+
_extract_json as a defensive fallback.
|
| 144 |
+
"""
|
| 145 |
+
client = _get_client()
|
| 146 |
+
response = _with_retries(
|
| 147 |
+
lambda: client.chat.completions.create(
|
| 148 |
+
model=config.JUDGE_MODEL,
|
| 149 |
+
messages=[
|
| 150 |
+
{"role": "system", "content": _JSON_SYSTEM},
|
| 151 |
+
{"role": "user", "content": prompt},
|
| 152 |
+
],
|
| 153 |
+
temperature=JUDGE_TEMPERATURE,
|
| 154 |
+
max_tokens=JUDGE_MAX_TOKENS,
|
| 155 |
+
response_format={
|
| 156 |
+
"type": "json_schema",
|
| 157 |
+
"json_schema": {"name": name, "strict": True, "schema": schema},
|
| 158 |
+
},
|
| 159 |
+
)
|
| 160 |
+
)
|
| 161 |
+
return _extract_json(response.choices[0].message.content or "")
|
| 162 |
+
|
| 163 |
+
|
| 164 |
+
def _join_contexts(contexts: list[str]) -> str:
|
| 165 |
+
"""Number the retrieved excerpts so the judge can refer to them by position."""
|
| 166 |
+
return "\n\n".join(f"[{i}] {c}" for i, c in enumerate(contexts, start=1))
|
| 167 |
+
|
| 168 |
+
|
| 169 |
+
# ----------------------------------------------------------------------------------------
|
| 170 |
+
# The four metrics. Each returns a float in [0, 1]; each may raise JudgeParseError (or a
|
| 171 |
+
# KeyError/TypeError on an unexpected JSON shape), which judge_item turns into a None.
|
| 172 |
+
# ----------------------------------------------------------------------------------------
|
| 173 |
+
|
| 174 |
+
|
| 175 |
+
def supported_claims(answer: str, contexts: list[str]) -> tuple[int, int]:
|
| 176 |
+
"""Decompose ANSWER into atomic claims and count how many CONTEXT supports.
|
| 177 |
+
|
| 178 |
+
Returns (supported, total). The ablation harness uses the raw counts (so an empty
|
| 179 |
+
answer -> 0/0 is visible) while faithfulness just takes the ratio.
|
| 180 |
+
"""
|
| 181 |
+
prompt = (
|
| 182 |
+
"Break the ANSWER into atomic factual claims. For each claim decide whether it is "
|
| 183 |
+
"supported by the CONTEXT (the retrieved calendar excerpts). A claim is supported only if "
|
| 184 |
+
"the CONTEXT contains information that backs it up.\n\n"
|
| 185 |
+
f"CONTEXT:\n{_join_contexts(contexts)}\n\n"
|
| 186 |
+
f"ANSWER:\n{answer}\n\n"
|
| 187 |
+
'Return JSON: {"claims": [{"claim": "<text>", "supported": true|false}, ...]}. '
|
| 188 |
+
'If the answer makes no factual claims, return {"claims": []}.'
|
| 189 |
+
)
|
| 190 |
+
claims = _chat_json(prompt, _claims_schema("supported"), "faithfulness")["claims"]
|
| 191 |
+
supported = sum(1 for c in claims if c.get("supported"))
|
| 192 |
+
return supported, len(claims)
|
| 193 |
+
|
| 194 |
+
|
| 195 |
+
def faithfulness(answer: str, contexts: list[str]) -> float:
|
| 196 |
+
"""supported_claims / total_claims. Hallucination rate = 1 - this."""
|
| 197 |
+
supported, total = supported_claims(answer, contexts)
|
| 198 |
+
if total == 0:
|
| 199 |
+
return 1.0 # nothing asserted -> nothing unfaithful
|
| 200 |
+
return supported / total
|
| 201 |
+
|
| 202 |
+
|
| 203 |
+
def answer_relevancy(question: str, answer: str) -> float:
|
| 204 |
+
"""Mean cosine similarity between the real question and questions the answer fits.
|
| 205 |
+
|
| 206 |
+
If the answer truly addresses the question, questions reverse-generated from it should
|
| 207 |
+
look a lot like the original question in embedding space.
|
| 208 |
+
"""
|
| 209 |
+
prompt = (
|
| 210 |
+
f"Given the ANSWER below, generate {RELEVANCY_NUM_QUESTIONS} distinct questions "
|
| 211 |
+
"that this answer would directly and completely answer.\n\n"
|
| 212 |
+
f"ANSWER:\n{answer}\n\n"
|
| 213 |
+
'Return JSON: {"questions": ["...", "..."]}.'
|
| 214 |
+
)
|
| 215 |
+
generated = _chat_json(prompt, _QUESTIONS_SCHEMA, "answer_relevancy")["questions"]
|
| 216 |
+
if not generated:
|
| 217 |
+
return 0.0
|
| 218 |
+
vecs = embed.embed_texts([question] + list(generated)) # unit vectors
|
| 219 |
+
sims = vecs[1:] @ vecs[0] # cosine == dot, since normalized
|
| 220 |
+
return float(np.clip(np.mean(sims), 0.0, 1.0))
|
| 221 |
+
|
| 222 |
+
|
| 223 |
+
def context_precision(question: str, reference_answer: str, contexts: list[str]) -> float:
|
| 224 |
+
"""Rank-aware precision: were relevant excerpts retrieved, and ranked near the top?"""
|
| 225 |
+
prompt = (
|
| 226 |
+
"For each retrieved CONTEXT, decide whether it is useful for answering the "
|
| 227 |
+
"QUESTION, given the REFERENCE ANSWER. Output true if the context contains "
|
| 228 |
+
"information useful to produce the reference answer, false otherwise.\n\n"
|
| 229 |
+
f"QUESTION: {question}\n"
|
| 230 |
+
f"REFERENCE ANSWER: {reference_answer}\n\n"
|
| 231 |
+
f"CONTEXTS:\n{_join_contexts(contexts)}\n\n"
|
| 232 |
+
'Return JSON: {"verdicts": [true|false, ...]} with one entry per context, in order.'
|
| 233 |
+
)
|
| 234 |
+
verdicts = [bool(v) for v in _chat_json(prompt, _VERDICTS_SCHEMA, "context_precision")["verdicts"]]
|
| 235 |
+
total_relevant = sum(verdicts)
|
| 236 |
+
if total_relevant == 0:
|
| 237 |
+
return 0.0
|
| 238 |
+
# Average precision: average of precision@i taken at each relevant rank i.
|
| 239 |
+
hits = 0
|
| 240 |
+
weighted = 0.0
|
| 241 |
+
for i, relevant in enumerate(verdicts, start=1):
|
| 242 |
+
if relevant:
|
| 243 |
+
hits += 1
|
| 244 |
+
weighted += hits / i
|
| 245 |
+
return weighted / total_relevant
|
| 246 |
+
|
| 247 |
+
|
| 248 |
+
def context_recall(reference_answer: str, contexts: list[str]) -> float:
|
| 249 |
+
"""Fraction of the reference answer's claims that the retrieved context supports."""
|
| 250 |
+
prompt = (
|
| 251 |
+
"Break the REFERENCE ANSWER into atomic claims. For each claim decide whether it "
|
| 252 |
+
"can be attributed to (is supported by) the CONTEXT.\n\n"
|
| 253 |
+
f"CONTEXT:\n{_join_contexts(contexts)}\n\n"
|
| 254 |
+
f"REFERENCE ANSWER:\n{reference_answer}\n\n"
|
| 255 |
+
'Return JSON: {"claims": [{"claim": "<text>", "attributed": true|false}, ...]}.'
|
| 256 |
+
)
|
| 257 |
+
claims = _chat_json(prompt, _claims_schema("attributed"), "context_recall")["claims"]
|
| 258 |
+
if not claims:
|
| 259 |
+
return 1.0
|
| 260 |
+
attributed = sum(1 for c in claims if c.get("attributed"))
|
| 261 |
+
return attributed / len(claims)
|
| 262 |
+
|
| 263 |
+
|
| 264 |
+
# Maps metric name -> a zero-arg callable, so judge_item can run each in a uniform
|
| 265 |
+
# try/except and record which ones failed without aborting the whole evaluation.
|
| 266 |
+
def judge_item(
|
| 267 |
+
question: str, answer: str, contexts: list[str], reference_answer: str
|
| 268 |
+
) -> dict:
|
| 269 |
+
"""Run all four metrics for one (question, answer, contexts) triple.
|
| 270 |
+
|
| 271 |
+
Returns a dict of {metric: float|None, ..., "errors": [names that failed to parse]}.
|
| 272 |
+
A None means the judge's reply for that metric couldn't be parsed — recorded, not
|
| 273 |
+
fatal, so a single bad response never sinks a long run.
|
| 274 |
+
"""
|
| 275 |
+
metrics = {
|
| 276 |
+
"faithfulness": lambda: faithfulness(answer, contexts),
|
| 277 |
+
"answer_relevancy": lambda: answer_relevancy(question, answer),
|
| 278 |
+
"context_precision": lambda: context_precision(question, reference_answer, contexts),
|
| 279 |
+
"context_recall": lambda: context_recall(reference_answer, contexts),
|
| 280 |
+
}
|
| 281 |
+
scores: dict = {}
|
| 282 |
+
errors: list[str] = []
|
| 283 |
+
for name, fn in metrics.items():
|
| 284 |
+
try:
|
| 285 |
+
scores[name] = round(float(fn()), 4)
|
| 286 |
+
except (JudgeParseError, KeyError, TypeError, ValueError, IndexError):
|
| 287 |
+
scores[name] = None
|
| 288 |
+
errors.append(name)
|
| 289 |
+
scores["errors"] = errors
|
| 290 |
+
return scores
|
eval/make_testset.py
ADDED
|
@@ -0,0 +1,127 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""
|
| 2 |
+
make_testset — generate a corpus-grounded test set from the indexed calendar chunks.
|
| 3 |
+
|
| 4 |
+
A test set is only meaningful if its gold ids point at chunks that are actually in the
|
| 5 |
+
index. So instead of hand-writing questions, we sample real indexed chunks and ask the
|
| 6 |
+
local LLM to write, for each one, a specific question that chunk answers plus a concise
|
| 7 |
+
reference answer grounded in it. The chunk's own id becomes the gold id — which is what
|
| 8 |
+
makes the deterministic retrieval metrics (hit@k etc.) work.
|
| 9 |
+
|
| 10 |
+
Landing pages and tiny fragments make useless questions, so sampling skips chunks with
|
| 11 |
+
page_type == "landing" or very short text.
|
| 12 |
+
|
| 13 |
+
This produces *candidates*. The intended workflow is to then CURATE by hand — delete vague
|
| 14 |
+
or wrong questions, fix wording, and (for near-duplicate chunks, e.g. the same requirement
|
| 15 |
+
in the 2025/26 vs 2026/27 edition or across cohort pages) add sibling ids to gold_ids —
|
| 16 |
+
and commit the frozen result. Start small and grow.
|
| 17 |
+
|
| 18 |
+
Run (needs a built index; uses the local LM Studio backend — see config.CHAT_MODEL):
|
| 19 |
+
python -m eval.make_testset --num 15 --out eval/testset.json
|
| 20 |
+
"""
|
| 21 |
+
|
| 22 |
+
from __future__ import annotations
|
| 23 |
+
|
| 24 |
+
import argparse
|
| 25 |
+
import json
|
| 26 |
+
import random
|
| 27 |
+
|
| 28 |
+
from openai import OpenAI
|
| 29 |
+
|
| 30 |
+
from src import config, retrieve
|
| 31 |
+
from eval.judge import _extract_json, JudgeParseError # reuse robust JSON parsing
|
| 32 |
+
|
| 33 |
+
_MIN_CHUNK_CHARS = 300 # too little text to ground a specific question + answer
|
| 34 |
+
|
| 35 |
+
_SYSTEM = (
|
| 36 |
+
"You write evaluation data for a UBC academic-calendar question-answering system. "
|
| 37 |
+
"Respond with ONLY valid JSON — no prose, no markdown, no code fences."
|
| 38 |
+
)
|
| 39 |
+
|
| 40 |
+
|
| 41 |
+
def _prompt(chunk: dict) -> str:
|
| 42 |
+
return (
|
| 43 |
+
"Given the CALENDAR EXCERPT below, write ONE specific question that a UBC student "
|
| 44 |
+
"might ask and that THIS excerpt answers well, plus a concise reference answer "
|
| 45 |
+
"grounded only in the excerpt. The question should be specific (name the course, "
|
| 46 |
+
"program, or policy it is about — and the student cohort or calendar year when the "
|
| 47 |
+
"excerpt is cohort- or edition-specific) and answerable from the excerpt.\n\n"
|
| 48 |
+
f"EXCERPT TITLE: {chunk['title']}\n"
|
| 49 |
+
f"SOURCE URL: {chunk['url']}\n"
|
| 50 |
+
f"EXCERPT TEXT: {chunk['text']}\n\n"
|
| 51 |
+
'Return JSON: {"question": "<question>", "reference_answer": "<answer>"}'
|
| 52 |
+
)
|
| 53 |
+
|
| 54 |
+
|
| 55 |
+
def _eligible(chunk: dict) -> bool:
|
| 56 |
+
return chunk.get("page_type") != "landing" and len(chunk.get("text", "")) >= _MIN_CHUNK_CHARS
|
| 57 |
+
|
| 58 |
+
|
| 59 |
+
def generate(num: int, seed: int) -> list[dict]:
|
| 60 |
+
"""Sample `num` eligible indexed chunks and turn each into a test item."""
|
| 61 |
+
_, chunks = retrieve._load_index() # the indexed corpus (id-aligned metadata)
|
| 62 |
+
pool = [c for c in chunks if _eligible(c)]
|
| 63 |
+
print(f"Sampling from {len(pool)} eligible chunks (of {len(chunks)} indexed).")
|
| 64 |
+
if num > len(pool):
|
| 65 |
+
num = len(pool)
|
| 66 |
+
sampled = random.Random(seed).sample(pool, num)
|
| 67 |
+
|
| 68 |
+
client = OpenAI(base_url=config.OPENAI_BASE_URL, api_key=config.OPENAI_API_KEY)
|
| 69 |
+
items: list[dict] = []
|
| 70 |
+
for chunk in sampled:
|
| 71 |
+
response = client.chat.completions.create(
|
| 72 |
+
model=config.CHAT_MODEL,
|
| 73 |
+
messages=[
|
| 74 |
+
{"role": "system", "content": _SYSTEM},
|
| 75 |
+
{"role": "user", "content": _prompt(chunk)},
|
| 76 |
+
],
|
| 77 |
+
temperature=0.3,
|
| 78 |
+
max_tokens=400,
|
| 79 |
+
)
|
| 80 |
+
try:
|
| 81 |
+
parsed = _extract_json(response.choices[0].message.content or "")
|
| 82 |
+
question = parsed["question"].strip()
|
| 83 |
+
reference_answer = parsed["reference_answer"].strip()
|
| 84 |
+
except (JudgeParseError, KeyError, AttributeError, TypeError):
|
| 85 |
+
print(f" skipped chunk {chunk['id']} ({chunk['title']!r}): bad JSON")
|
| 86 |
+
continue
|
| 87 |
+
|
| 88 |
+
items.append(
|
| 89 |
+
{
|
| 90 |
+
"id": f"q{len(items) + 1:03d}",
|
| 91 |
+
"question": question,
|
| 92 |
+
"reference_answer": reference_answer,
|
| 93 |
+
"gold_ids": [chunk["id"]],
|
| 94 |
+
"source_chunk_id": chunk["id"],
|
| 95 |
+
}
|
| 96 |
+
)
|
| 97 |
+
print(f" [{len(items)}] id={chunk['id']:<4} {chunk['title']}")
|
| 98 |
+
return items
|
| 99 |
+
|
| 100 |
+
|
| 101 |
+
def main() -> None:
|
| 102 |
+
parser = argparse.ArgumentParser(description="Generate a corpus-grounded eval test set.")
|
| 103 |
+
parser.add_argument("--num", type=int, default=15, help="how many questions to generate")
|
| 104 |
+
parser.add_argument("--seed", type=int, default=42, help="sampling seed (reproducible)")
|
| 105 |
+
parser.add_argument("--out", default=str(config.TESTSET_PATH), help="output JSON path")
|
| 106 |
+
args = parser.parse_args()
|
| 107 |
+
|
| 108 |
+
print(f"Generating {args.num} questions from the indexed corpus "
|
| 109 |
+
f"(model={config.CHAT_MODEL}) ...")
|
| 110 |
+
items = generate(args.num, args.seed)
|
| 111 |
+
|
| 112 |
+
payload = {
|
| 113 |
+
"description": (
|
| 114 |
+
"Auto-generated, corpus-grounded test set (eval/make_testset.py). Each gold id "
|
| 115 |
+
"points at an indexed calendar chunk. REVIEW AND CURATE before trusting the "
|
| 116 |
+
"numbers: delete vague/wrong items, fix wording, add sibling ids to gold_ids "
|
| 117 |
+
"for near-duplicate chunks (edition/cohort twins)."
|
| 118 |
+
),
|
| 119 |
+
"items": items,
|
| 120 |
+
}
|
| 121 |
+
with open(args.out, "w", encoding="utf-8") as f:
|
| 122 |
+
json.dump(payload, f, ensure_ascii=False, indent=2)
|
| 123 |
+
print(f"\nWrote {len(items)} questions to {args.out}")
|
| 124 |
+
|
| 125 |
+
|
| 126 |
+
if __name__ == "__main__":
|
| 127 |
+
main()
|
eval/reports/history.csv
ADDED
|
@@ -0,0 +1,4 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
timestamp,name,k,num_questions,judged,chat_model,judge_model,hit@k,recall@k,mrr,faithfulness,answer_relevancy,context_precision,context_recall,hallucination_rate
|
| 2 |
+
2026-08-31T00:30:01,sweep-dense,4,12,False,google/gemma-4-e4b,google/gemma-4-e4b,0.5,0.4583,0.4028,,,,,
|
| 3 |
+
2026-08-31T00:30:06,sweep-hybrid,4,12,False,google/gemma-4-e4b,google/gemma-4-e4b,0.6667,0.625,0.5417,,,,,
|
| 4 |
+
2026-08-31T00:32:59,sweep-hybrid-rerank,4,12,False,google/gemma-4-e4b,google/gemma-4-e4b,0.6667,0.625,0.5417,,,,,
|
eval/reports/report.md
ADDED
|
@@ -0,0 +1,136 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# UBCHelper eval — sweep-hybrid-rerank
|
| 2 |
+
|
| 3 |
+
2026-08-31T00:32:59 · 12 questions · k=4 · judge off
|
| 4 |
+
|
| 5 |
+
Models: chat=`google/gemma-4-e4b`, judge=`google/gemma-4-e4b`, embed=`text-embedding-3-small` · corpus=3770 · retrieval=hybrid_rerank
|
| 6 |
+
|
| 7 |
+
## Summary
|
| 8 |
+
|
| 9 |
+
| metric | score |
|
| 10 |
+
|--------|-------|
|
| 11 |
+
| hit@4 | 0.667 |
|
| 12 |
+
| recall@4 | 0.625 |
|
| 13 |
+
| MRR | 0.542 |
|
| 14 |
+
|
| 15 |
+
## Questions
|
| 16 |
+
|
| 17 |
+
### q001 — hit — gold [10, 182]
|
| 18 |
+
|
| 19 |
+
**Q:** What are the prerequisites for CPSC 221?
|
| 20 |
+
|
| 21 |
+
**Reference:** One of CPSC 210 or CPEN 221, and either one of CPSC 121, MATH 220 (or MATH_O 220), or a score of 68% or higher in MATH 226.
|
| 22 |
+
|
| 23 |
+
**Retrieved:** [182] Computer Science, Faculty of Science [2025/26] · [10] Computer Science, Faculty of Science [2026/27] · [232] Computer Science, Faculty of Science [2025/26] · [339] Bachelor of Science — Computer Science [2025/26]
|
| 24 |
+
|
| 25 |
+
**Scores:** rr 1.00
|
| 26 |
+
|
| 27 |
+
### q002 — miss — gold [24, 197]
|
| 28 |
+
|
| 29 |
+
**Q:** What are the prerequisites for CPSC 320?
|
| 30 |
+
|
| 31 |
+
**Reference:** CPSC 221 (or DSCI 221) plus at least 3 credits of MATH or STAT at the 200 level or above (the 2025/26 edition also accepted COMM 291 or BIOL 300 for the second part).
|
| 32 |
+
|
| 33 |
+
**Retrieved:** [42] Computer Science, Faculty of Science [2026/27] · [214] Computer Science, Faculty of Science [2025/26] · [339] Bachelor of Science — Computer Science [2025/26] · [170] Bachelor of Science — Computer Science [2026/27]
|
| 34 |
+
|
| 35 |
+
**Scores:** rr 0.00
|
| 36 |
+
|
| 37 |
+
### q003 — hit — gold [20, 193]
|
| 38 |
+
|
| 39 |
+
**Q:** What are the prerequisites for CPSC 313?
|
| 40 |
+
|
| 41 |
+
**Reference:** CPSC 213, and either CPSC 221 or DSCI 221.
|
| 42 |
+
|
| 43 |
+
**Retrieved:** [20] Computer Science, Faculty of Science [2026/27] · [193] Computer Science, Faculty of Science [2025/26] · [213] Computer Science, Faculty of Science [2025/26] · [212] Computer Science, Faculty of Science [2025/26]
|
| 44 |
+
|
| 45 |
+
**Scores:** rr 1.00
|
| 46 |
+
|
| 47 |
+
### q004 — hit — gold [580, 2319]
|
| 48 |
+
|
| 49 |
+
**Q:** Which courses satisfy the prerequisite for MATH 200, Calculus III?
|
| 50 |
+
|
| 51 |
+
**Reference:** One of MATH 101, MATH 103, MATH 105, MATH 121, SCIE 001, MATH_O 101, or MATH_O 103.
|
| 52 |
+
|
| 53 |
+
**Retrieved:** [580] Mathematics, Faculty of Science [2026/27] · [2319] Mathematics, Faculty of Science [2025/26] · [2348] Mathematics, Faculty of Science [2025/26] · [609] Mathematics, Faculty of Science [2026/27]
|
| 54 |
+
|
| 55 |
+
**Scores:** rr 1.00
|
| 56 |
+
|
| 57 |
+
### q005 — miss — gold [5, 177]
|
| 58 |
+
|
| 59 |
+
**Q:** How many credits is CPSC 110 worth and what does it cover?
|
| 60 |
+
|
| 61 |
+
**Reference:** CPSC 110 (Computation, Programs, and Programming) is worth 4 credits and covers fundamental program and computation structures.
|
| 62 |
+
|
| 63 |
+
**Retrieved:** [146] Bachelor of Science — Computer Science [2026/27] · [2082] Bachelor of Science — Mathematics [2026/27] · [160] Bachelor of Science — Computer Science [2026/27] · [3568] Bachelor of Science — Mathematics [2025/26]
|
| 64 |
+
|
| 65 |
+
**Scores:** rr 0.00
|
| 66 |
+
|
| 67 |
+
### q006 — miss — gold [114, 128]
|
| 68 |
+
|
| 69 |
+
**Q:** Compare the B.A. degree requirements for students who entered the program in 2023/24 with those who entered in 2024/25 or later.
|
| 70 |
+
|
| 71 |
+
**Reference:** Students entering 2023/24 or earlier must complete seven requirements: Writing and Research, Language, Science, Literature, Outside, Upper-level, and Arts Credit Minimum. Students entering 2024/25 or later complete five: Writing and Research, Ways of Knowing Breadth, Outside, Upper-level, and Arts Credit Minimum — the separate Language, Science, and Literature requirements are folded into the Ways of Knowing Breadth requirement.
|
| 72 |
+
|
| 73 |
+
**Retrieved:** [3039] Bachelor of Arts — Second Degree Studies [2025/26] · [1295] Bachelor of Arts — Second Degree Studies [2026/27] · [2712] Dual Degree Program Option: Bachelor of Arts, UBC and Sciences Po [2025/26] · [963] Dual Degree Program Option: Bachelor of Arts, UBC and Sciences Po [2026/27]
|
| 74 |
+
|
| 75 |
+
**Scores:** rr 0.00
|
| 76 |
+
|
| 77 |
+
### q007 — hit — gold [1769, 1791]
|
| 78 |
+
|
| 79 |
+
**Q:** How does admission to the Bachelor of Computer Science program differ from B.Sc. admission from secondary school?
|
| 80 |
+
|
| 81 |
+
**Reference:** Bachelor of Computer Science admission is a competitive selection process evaluating applicants on academic and other criteria (not every qualified applicant is admitted); B.Sc. admission from secondary school requires starting first year in September of the Winter session of admission (January starts are not permitted).
|
| 82 |
+
|
| 83 |
+
**Retrieved:** [1768] Bachelor of Computer Science [2026/27] · [1769] Bachelor of Computer Science [2026/27] · [1767] Bachelor of Computer Science [2026/27] · [3275] Bachelor of Science — Admission and Transfer [2025/26]
|
| 84 |
+
|
| 85 |
+
**Scores:** rr 0.50
|
| 86 |
+
|
| 87 |
+
### q008 — miss — gold [20, 10]
|
| 88 |
+
|
| 89 |
+
**Q:** I have finished CPSC 210. What else do I need to complete before I can take CPSC 313?
|
| 90 |
+
|
| 91 |
+
**Reference:** CPSC 313 requires CPSC 213 and either CPSC 221 or DSCI 221. CPSC 221 in turn requires (beyond CPSC 210) one of CPSC 121, MATH 220, MATH_O 220, or a 68%+ score in MATH 226. So after CPSC 210 you still need CPSC 213 and CPSC 221 (with its discrete-math prerequisite).
|
| 92 |
+
|
| 93 |
+
**Retrieved:** [917] Bachelor of Arts — Computer Science [2026/27] · [2663] Bachelor of Arts — Computer Science [2025/26] · [154] Bachelor of Science — Computer Science [2026/27] · [160] Bachelor of Science — Computer Science [2026/27]
|
| 94 |
+
|
| 95 |
+
**Scores:** rr 0.00
|
| 96 |
+
|
| 97 |
+
### q009 — hit — gold [3508]
|
| 98 |
+
|
| 99 |
+
**Q:** In the 2025/26 calendar, what is the minimum number of credits required for a B.Sc. degree?
|
| 100 |
+
|
| 101 |
+
**Reference:** A minimum of 120 credits (a major, double major, or General Science option requires at least 120 credits but may require more).
|
| 102 |
+
|
| 103 |
+
**Retrieved:** [3509] Bachelor of Science — General Degree Requirements [2025/26] · [3508] Bachelor of Science — General Degree Requirements [2025/26] · [3438] Bachelor of Science — Credit at UBC and Elsewhere [2025/26] · [2023] Bachelor of Science — General Degree Requirements [2026/27]
|
| 104 |
+
|
| 105 |
+
**Scores:** rr 0.50
|
| 106 |
+
|
| 107 |
+
### q010 — hit — gold [2023]
|
| 108 |
+
|
| 109 |
+
**Q:** According to the current 2026/27 calendar, what is the minimum number of credits required for a B.Sc. degree?
|
| 110 |
+
|
| 111 |
+
**Reference:** A minimum of 120 credits (a major, double major, or General Science option requires at least 120 credits but may require more).
|
| 112 |
+
|
| 113 |
+
**Retrieved:** [2023] Bachelor of Science — General Degree Requirements [2026/27] · [2024] Bachelor of Science — General Degree Requirements [2026/27] · [1954] Bachelor of Science — Credit at UBC and Elsewhere [2026/27] · [2174] Bachelor of Science — Recognition of Academic Achievement [2026/27]
|
| 114 |
+
|
| 115 |
+
**Scores:** rr 1.00
|
| 116 |
+
|
| 117 |
+
### q011 — hit — gold [128]
|
| 118 |
+
|
| 119 |
+
**Q:** Does a B.A. student who entered the program in 2024/25 or later have to complete a language requirement?
|
| 120 |
+
|
| 121 |
+
**Reference:** No. Their five degree requirements are Writing and Research, Ways of Knowing Breadth, Outside, Upper-level, and Arts Credit Minimum — there is no separate language requirement (that applies to students who entered in 2023/24 or earlier).
|
| 122 |
+
|
| 123 |
+
**Retrieved:** [2890] Bachelor of Arts — Linguistics [2025/26] · [128] Bachelor of Arts — Degree Requirements for students who enter the program in 2024/25 or later [2026/27] · [2903] Bachelor of Arts — Linguistics [2025/26] · [2285] Bachelor of International Economics — Degree Requirements for students who enter the program in 2024/25 or later [2026/27]
|
| 124 |
+
|
| 125 |
+
**Scores:** rr 0.50
|
| 126 |
+
|
| 127 |
+
### q012 — hit — gold [0]
|
| 128 |
+
|
| 129 |
+
**Q:** What are the three levels of academic standing at UBC?
|
| 130 |
+
|
| 131 |
+
**Reference:** In Good Standing; On Academic Probation; and Failed, Required to Withdraw. All students are In Good Standing on initial entry to the University.
|
| 132 |
+
|
| 133 |
+
**Retrieved:** [0] Academic Standing [2026/27] · [431] Course Standings [2026/27] · [430] Course Standings [2026/27] · [3658] Bachelor of Science — Recognition of Academic Achievement [2025/26]
|
| 134 |
+
|
| 135 |
+
**Scores:** rr 1.00
|
| 136 |
+
|
eval/retrieval_metrics.py
ADDED
|
@@ -0,0 +1,138 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""
|
| 2 |
+
Tier 1 — retrieval metrics. Deterministic, no LLM, instant, free.
|
| 3 |
+
|
| 4 |
+
This is the fast inner loop for tuning retrieval: it asks "did the retriever surface the
|
| 5 |
+
right excerpt?" purely by comparing the ids it returned against the gold ids in the test
|
| 6 |
+
set. No judge model, no embeddings beyond the query embedding the retriever already does.
|
| 7 |
+
|
| 8 |
+
Three standard metrics, each computed per question over the ranked list of returned ids,
|
| 9 |
+
then averaged across all questions:
|
| 10 |
+
|
| 11 |
+
hit@k did ANY gold id land in the top-k? (1 or 0) -> mean
|
| 12 |
+
recall@k what FRACTION of gold ids landed in the top-k? -> mean
|
| 13 |
+
MRR 1 / rank of the FIRST gold hit (0 if none) -> mean
|
| 14 |
+
|
| 15 |
+
Run it (needs LM Studio for the query embeddings + a built index):
|
| 16 |
+
python -m eval.retrieval_metrics
|
| 17 |
+
python -m eval.retrieval_metrics --k 3 --limit 5
|
| 18 |
+
"""
|
| 19 |
+
|
| 20 |
+
from __future__ import annotations
|
| 21 |
+
|
| 22 |
+
import argparse
|
| 23 |
+
import json
|
| 24 |
+
|
| 25 |
+
from src import config, retrieve
|
| 26 |
+
|
| 27 |
+
# ----------------------------------------------------------------------------------------
|
| 28 |
+
# Pure metric functions. These take plain lists of ids, so they are fully deterministic
|
| 29 |
+
# and unit-testable with no LM Studio, no index, no network.
|
| 30 |
+
# ----------------------------------------------------------------------------------------
|
| 31 |
+
|
| 32 |
+
|
| 33 |
+
def hit_at_k(retrieved_ids: list[int], gold_ids: list[int], k: int) -> float:
|
| 34 |
+
"""1.0 if at least one gold id appears in the top-k retrieved ids, else 0.0."""
|
| 35 |
+
top_k = set(retrieved_ids[:k])
|
| 36 |
+
return 1.0 if top_k & set(gold_ids) else 0.0
|
| 37 |
+
|
| 38 |
+
|
| 39 |
+
def recall_at_k(retrieved_ids: list[int], gold_ids: list[int], k: int) -> float:
|
| 40 |
+
"""Fraction of the gold ids that appear in the top-k retrieved ids."""
|
| 41 |
+
if not gold_ids:
|
| 42 |
+
return 0.0
|
| 43 |
+
top_k = set(retrieved_ids[:k])
|
| 44 |
+
found = top_k & set(gold_ids)
|
| 45 |
+
return len(found) / len(gold_ids)
|
| 46 |
+
|
| 47 |
+
|
| 48 |
+
def reciprocal_rank(retrieved_ids: list[int], gold_ids: list[int]) -> float:
|
| 49 |
+
"""1 / (rank of the first gold hit). 0.0 if no gold id is in the list.
|
| 50 |
+
|
| 51 |
+
Averaging this across questions gives MRR (Mean Reciprocal Rank).
|
| 52 |
+
"""
|
| 53 |
+
gold = set(gold_ids)
|
| 54 |
+
for rank, doc_id in enumerate(retrieved_ids, start=1):
|
| 55 |
+
if doc_id in gold:
|
| 56 |
+
return 1.0 / rank
|
| 57 |
+
return 0.0
|
| 58 |
+
|
| 59 |
+
|
| 60 |
+
# ----------------------------------------------------------------------------------------
|
| 61 |
+
# Runner: drive the retriever over the test set and aggregate the metrics above.
|
| 62 |
+
# (This part DOES need LM Studio — to embed each query — and a built index.)
|
| 63 |
+
# ----------------------------------------------------------------------------------------
|
| 64 |
+
|
| 65 |
+
|
| 66 |
+
def load_testset(path=config.TESTSET_PATH) -> list[dict]:
|
| 67 |
+
with open(path, "r", encoding="utf-8") as f:
|
| 68 |
+
return json.load(f)["items"]
|
| 69 |
+
|
| 70 |
+
|
| 71 |
+
def evaluate_retrieval(testset: list[dict], k: int) -> dict:
|
| 72 |
+
"""Run retrieval for every question and return aggregate + per-item metrics.
|
| 73 |
+
|
| 74 |
+
`k` controls both how many excerpts we retrieve and the @k in the metrics, so the eval
|
| 75 |
+
measures retrieval at the depth it reports.
|
| 76 |
+
"""
|
| 77 |
+
per_item = []
|
| 78 |
+
for item in testset:
|
| 79 |
+
results = retrieve.retrieve(item["question"], top_k=k)
|
| 80 |
+
retrieved_ids = [r.id for r in results]
|
| 81 |
+
gold_ids = item["gold_ids"]
|
| 82 |
+
per_item.append(
|
| 83 |
+
{
|
| 84 |
+
"id": item["id"],
|
| 85 |
+
"gold_ids": gold_ids,
|
| 86 |
+
"retrieved_ids": retrieved_ids,
|
| 87 |
+
"hit": hit_at_k(retrieved_ids, gold_ids, k),
|
| 88 |
+
"recall": recall_at_k(retrieved_ids, gold_ids, k),
|
| 89 |
+
"rr": reciprocal_rank(retrieved_ids, gold_ids),
|
| 90 |
+
}
|
| 91 |
+
)
|
| 92 |
+
|
| 93 |
+
n = len(per_item) or 1 # avoid divide-by-zero on an empty test set
|
| 94 |
+
aggregate = {
|
| 95 |
+
"k": k,
|
| 96 |
+
"num_questions": len(per_item),
|
| 97 |
+
f"hit@{k}": sum(x["hit"] for x in per_item) / n,
|
| 98 |
+
f"recall@{k}": sum(x["recall"] for x in per_item) / n,
|
| 99 |
+
"mrr": sum(x["rr"] for x in per_item) / n,
|
| 100 |
+
}
|
| 101 |
+
return {"aggregate": aggregate, "per_item": per_item}
|
| 102 |
+
|
| 103 |
+
|
| 104 |
+
def _print_report(report: dict) -> None:
|
| 105 |
+
agg = report["aggregate"]
|
| 106 |
+
k = agg["k"]
|
| 107 |
+
hit = agg[f"hit@{k}"]
|
| 108 |
+
recall = agg[f"recall@{k}"]
|
| 109 |
+
print(f"\nRetrieval metrics over {agg['num_questions']} questions (k={k}):")
|
| 110 |
+
print(f" hit@{k} : {hit:.3f}")
|
| 111 |
+
print(f" recall@{k} : {recall:.3f}")
|
| 112 |
+
print(f" MRR : {agg['mrr']:.3f}")
|
| 113 |
+
|
| 114 |
+
print("\nPer-question ([+] = a gold id was retrieved):")
|
| 115 |
+
for x in report["per_item"]:
|
| 116 |
+
mark = "[+]" if x["hit"] else "[ ]"
|
| 117 |
+
print(
|
| 118 |
+
f" {mark} {x['id']} gold={x['gold_ids']} "
|
| 119 |
+
f"retrieved={x['retrieved_ids']} rr={x['rr']:.3f}"
|
| 120 |
+
)
|
| 121 |
+
|
| 122 |
+
|
| 123 |
+
def main() -> None:
|
| 124 |
+
parser = argparse.ArgumentParser(description="Run Tier-1 retrieval metrics.")
|
| 125 |
+
parser.add_argument("--k", type=int, default=config.EVAL_K, help="retrieval depth / @k")
|
| 126 |
+
parser.add_argument("--limit", type=int, default=None, help="only the first N questions")
|
| 127 |
+
args = parser.parse_args()
|
| 128 |
+
|
| 129 |
+
testset = load_testset()
|
| 130 |
+
if args.limit:
|
| 131 |
+
testset = testset[: args.limit]
|
| 132 |
+
|
| 133 |
+
report = evaluate_retrieval(testset, k=args.k)
|
| 134 |
+
_print_report(report)
|
| 135 |
+
|
| 136 |
+
|
| 137 |
+
if __name__ == "__main__":
|
| 138 |
+
main()
|
eval/run_ablation.py
ADDED
|
@@ -0,0 +1,249 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""
|
| 2 |
+
Ablation — how much is the RAG actually helping?
|
| 3 |
+
|
| 4 |
+
Answers every test question two ways and scores both with the SAME metric:
|
| 5 |
+
|
| 6 |
+
RAG : retrieve relevant excerpts -> grounded generation (the real pipeline)
|
| 7 |
+
no-RAG : closed-book, no context -> the model answers from memory (the baseline)
|
| 8 |
+
|
| 9 |
+
Both answers are judged for GROUNDEDNESS against the question's *gold excerpt* — the fraction
|
| 10 |
+
of the answer's atomic claims that the true excerpt supports (reusing judge.supported_claims).
|
| 11 |
+
Hallucination rate = 1 - groundedness. Because the judge compares against the gold excerpt (not
|
| 12 |
+
the retrieved context), the metric is well-defined for the no-RAG arm too, so the two are
|
| 13 |
+
directly comparable. The gap between the arms is the value retrieval adds.
|
| 14 |
+
|
| 15 |
+
The test set is the frozen specific-source questions (eval/testset.json) plus a few broad
|
| 16 |
+
general calendar questions (eval/testset.general.json); results are broken down by type, since
|
| 17 |
+
RAG helps most on specific-source questions and least on general knowledge.
|
| 18 |
+
|
| 19 |
+
NOTE: "hallucination" here means "unsupported by the gold excerpt", so a closed-book answer
|
| 20 |
+
that gives plausible but *different* requirements still counts as unsupported. That is intended
|
| 21 |
+
for specific-source questions; it is why general questions are reported separately.
|
| 22 |
+
|
| 23 |
+
Run (needs an index built; the judge defaults to a local LM Studio server — see
|
| 24 |
+
config.JUDGE_MODEL):
|
| 25 |
+
python -m eval.run_ablation
|
| 26 |
+
python -m eval.run_ablation --limit 2 # quick smoke test
|
| 27 |
+
"""
|
| 28 |
+
|
| 29 |
+
from __future__ import annotations
|
| 30 |
+
|
| 31 |
+
import argparse
|
| 32 |
+
import json
|
| 33 |
+
from datetime import datetime
|
| 34 |
+
from statistics import mean
|
| 35 |
+
|
| 36 |
+
from src import config, generate, retrieve
|
| 37 |
+
from eval import judge
|
| 38 |
+
|
| 39 |
+
|
| 40 |
+
def _load_items(path, kind: str) -> list[dict]:
|
| 41 |
+
"""Load a test set file and tag every item with its `type` (specific/general)."""
|
| 42 |
+
if not path.exists():
|
| 43 |
+
return []
|
| 44 |
+
data = json.loads(path.read_text(encoding="utf-8"))
|
| 45 |
+
items = data["items"]
|
| 46 |
+
for it in items:
|
| 47 |
+
it["type"] = kind
|
| 48 |
+
return items
|
| 49 |
+
|
| 50 |
+
|
| 51 |
+
def _groundedness(answer: str, gold_texts: list[str]) -> dict:
|
| 52 |
+
"""Score one answer against the gold excerpt(s). Returns score + raw claim counts.
|
| 53 |
+
|
| 54 |
+
For near-dup questions with several valid golds, a claim supported by ANY gold counts —
|
| 55 |
+
so the answer is judged against the union of all gold excerpts.
|
| 56 |
+
"""
|
| 57 |
+
supported, total = judge.supported_claims(answer, gold_texts)
|
| 58 |
+
score = 1.0 if total == 0 else supported / total
|
| 59 |
+
return {"score": round(score, 4), "supported": supported, "total": total}
|
| 60 |
+
|
| 61 |
+
|
| 62 |
+
def run(items: list[dict], k: int) -> dict:
|
| 63 |
+
"""Run both arms for every item and return the full ablation report dict."""
|
| 64 |
+
_, metadata = retrieve._load_index()
|
| 65 |
+
chunk_by_id = {r["id"]: r for r in metadata}
|
| 66 |
+
|
| 67 |
+
per_item = []
|
| 68 |
+
for n, item in enumerate(items, start=1):
|
| 69 |
+
gold_ids = item["gold_ids"]
|
| 70 |
+
golds = [chunk_by_id[g] for g in gold_ids if g in chunk_by_id]
|
| 71 |
+
if not golds:
|
| 72 |
+
print(f" [{n}/{len(items)}] {item['id']} SKIPPED — no gold ids {gold_ids} in index")
|
| 73 |
+
continue
|
| 74 |
+
gold_texts = [g["text"] for g in golds]
|
| 75 |
+
gold_title = golds[0]["title"] + (f" (+{len(golds) - 1} more)" if len(golds) > 1 else "")
|
| 76 |
+
question = item["question"]
|
| 77 |
+
|
| 78 |
+
# Arm A: RAG (retrieve -> grounded generation). Arm B: closed-book (no context).
|
| 79 |
+
rag_answer = generate.generate_answer(question, retrieve.retrieve(question, top_k=k))
|
| 80 |
+
cb_answer = generate.generate_answer_closed_book(question)
|
| 81 |
+
|
| 82 |
+
rag = _groundedness(rag_answer, gold_texts)
|
| 83 |
+
cb = _groundedness(cb_answer, gold_texts)
|
| 84 |
+
|
| 85 |
+
per_item.append({
|
| 86 |
+
"id": item["id"],
|
| 87 |
+
"type": item["type"],
|
| 88 |
+
"question": question,
|
| 89 |
+
"reference_answer": item.get("reference_answer", ""),
|
| 90 |
+
"gold_id": gold_ids,
|
| 91 |
+
"gold_title": gold_title,
|
| 92 |
+
"rag": {"answer": rag_answer, **rag},
|
| 93 |
+
"no_rag": {"answer": cb_answer, **cb},
|
| 94 |
+
})
|
| 95 |
+
print(
|
| 96 |
+
f" [{n}/{len(items)}] {item['id']} ({item['type']}) "
|
| 97 |
+
f"RAG ground={rag['score']:.2f} ({rag['supported']}/{rag['total']}) "
|
| 98 |
+
f"no-RAG ground={cb['score']:.2f} ({cb['supported']}/{cb['total']})"
|
| 99 |
+
)
|
| 100 |
+
|
| 101 |
+
return _aggregate(per_item, k)
|
| 102 |
+
|
| 103 |
+
|
| 104 |
+
def _aggregate(per_item: list[dict], k: int) -> dict:
|
| 105 |
+
def summarize(rows: list[dict]) -> dict | None:
|
| 106 |
+
if not rows:
|
| 107 |
+
return None
|
| 108 |
+
rag_g = mean(r["rag"]["score"] for r in rows)
|
| 109 |
+
cb_g = mean(r["no_rag"]["score"] for r in rows)
|
| 110 |
+
return {
|
| 111 |
+
"n": len(rows),
|
| 112 |
+
"rag_groundedness": round(rag_g, 4),
|
| 113 |
+
"no_rag_groundedness": round(cb_g, 4),
|
| 114 |
+
"rag_hallucination": round(1.0 - rag_g, 4),
|
| 115 |
+
"no_rag_hallucination": round(1.0 - cb_g, 4),
|
| 116 |
+
"hallucination_delta": round((1.0 - rag_g) - (1.0 - cb_g), 4), # neg = RAG helps
|
| 117 |
+
}
|
| 118 |
+
|
| 119 |
+
by_type = {
|
| 120 |
+
"overall": summarize(per_item),
|
| 121 |
+
"specific": summarize([r for r in per_item if r["type"] == "specific"]),
|
| 122 |
+
"general": summarize([r for r in per_item if r["type"] == "general"]),
|
| 123 |
+
}
|
| 124 |
+
return {
|
| 125 |
+
"run": {
|
| 126 |
+
"timestamp": datetime.now().isoformat(timespec="seconds"),
|
| 127 |
+
"k": k,
|
| 128 |
+
"num_questions": len(per_item),
|
| 129 |
+
"chat_model": config.CHAT_MODEL,
|
| 130 |
+
"judge_model": config.JUDGE_MODEL,
|
| 131 |
+
"embedding_model": config.EMBEDDING_MODEL,
|
| 132 |
+
"corpus_size": len(retrieve._load_index()[1]), # indexed chunk count
|
| 133 |
+
},
|
| 134 |
+
"summary": by_type,
|
| 135 |
+
"per_item": per_item,
|
| 136 |
+
}
|
| 137 |
+
|
| 138 |
+
|
| 139 |
+
# ----------------------------------------------------------------------------------------
|
| 140 |
+
# Report: a single self-contained markdown doc (summary table + every answer side by side).
|
| 141 |
+
# ----------------------------------------------------------------------------------------
|
| 142 |
+
|
| 143 |
+
def _fmt(v) -> str:
|
| 144 |
+
return "-" if v is None else f"{v:.3f}"
|
| 145 |
+
|
| 146 |
+
|
| 147 |
+
def write_report(report: dict, name: str):
|
| 148 |
+
config.REPORTS_DIR.mkdir(parents=True, exist_ok=True)
|
| 149 |
+
config.ABLATION_REPORT_PATH.write_text(_render_markdown(report, name), encoding="utf-8")
|
| 150 |
+
return config.ABLATION_REPORT_PATH
|
| 151 |
+
|
| 152 |
+
|
| 153 |
+
def _render_markdown(report: dict, name: str) -> str:
|
| 154 |
+
run = report["run"]
|
| 155 |
+
s = report["summary"]
|
| 156 |
+
n_specific = sum(1 for r in report["per_item"] if r["type"] == "specific")
|
| 157 |
+
n_general = sum(1 for r in report["per_item"] if r["type"] == "general")
|
| 158 |
+
|
| 159 |
+
lines = [
|
| 160 |
+
f"# UBCHelper ablation — {name}",
|
| 161 |
+
"",
|
| 162 |
+
f"{run['timestamp']} · {run['num_questions']} questions "
|
| 163 |
+
f"({n_specific} specific + {n_general} general) · k={run['k']}",
|
| 164 |
+
"",
|
| 165 |
+
f"Models: chat=`{run['chat_model']}`, judge=`{run['judge_model']}`, "
|
| 166 |
+
f"embed=`{run['embedding_model']}` · corpus={run['corpus_size']}",
|
| 167 |
+
"",
|
| 168 |
+
"Groundedness = fraction of the answer's claims supported by the gold excerpt "
|
| 169 |
+
"(higher is better). Hallucination = 1 - groundedness. Both arms are judged against "
|
| 170 |
+
"the same gold excerpt, so a closed-book answer that gives plausible but *different* "
|
| 171 |
+
"requirements still counts as unsupported — which is why general questions are listed "
|
| 172 |
+
"separately.",
|
| 173 |
+
"",
|
| 174 |
+
"## Summary — RAG vs no-RAG",
|
| 175 |
+
"",
|
| 176 |
+
"| set | n | RAG groundedness | no-RAG groundedness | RAG halluc. | no-RAG halluc. | halluc. delta (RAG-noRAG) |",
|
| 177 |
+
"|-----|---|------------------|---------------------|-------------|----------------|---------------------------|",
|
| 178 |
+
]
|
| 179 |
+
for label in ("overall", "specific", "general"):
|
| 180 |
+
row = s.get(label)
|
| 181 |
+
if row is None:
|
| 182 |
+
continue
|
| 183 |
+
lines.append(
|
| 184 |
+
f"| {label} | {row['n']} | {_fmt(row['rag_groundedness'])} | "
|
| 185 |
+
f"{_fmt(row['no_rag_groundedness'])} | {_fmt(row['rag_hallucination'])} | "
|
| 186 |
+
f"{_fmt(row['no_rag_hallucination'])} | {_fmt(row['hallucination_delta'])} |"
|
| 187 |
+
)
|
| 188 |
+
|
| 189 |
+
lines += ["", "## Questions", ""]
|
| 190 |
+
for x in report["per_item"]:
|
| 191 |
+
rag, cb = x["rag"], x["no_rag"]
|
| 192 |
+
lines += [
|
| 193 |
+
f"### {x['id']} — {x['type']} — gold {x['gold_id']} {x['gold_title']}",
|
| 194 |
+
"",
|
| 195 |
+
f"**Q:** {x['question']}",
|
| 196 |
+
"",
|
| 197 |
+
f"**Reference:** {x['reference_answer']}",
|
| 198 |
+
"",
|
| 199 |
+
f"**RAG answer** — groundedness {rag['score']:.3f} "
|
| 200 |
+
f"({rag['supported']}/{rag['total']} claims):",
|
| 201 |
+
"",
|
| 202 |
+
rag["answer"],
|
| 203 |
+
"",
|
| 204 |
+
f"**No-RAG answer** — groundedness {cb['score']:.3f} "
|
| 205 |
+
f"({cb['supported']}/{cb['total']} claims):",
|
| 206 |
+
"",
|
| 207 |
+
cb["answer"],
|
| 208 |
+
"",
|
| 209 |
+
]
|
| 210 |
+
return "\n".join(lines) + "\n"
|
| 211 |
+
|
| 212 |
+
|
| 213 |
+
def _print_summary(report: dict) -> None:
|
| 214 |
+
print("\n=== RAG vs no-RAG (groundedness, higher is better) ===")
|
| 215 |
+
for label in ("overall", "specific", "general"):
|
| 216 |
+
row = report["summary"].get(label)
|
| 217 |
+
if row is None:
|
| 218 |
+
continue
|
| 219 |
+
print(
|
| 220 |
+
f"{label:<9} (n={row['n']:>2}): "
|
| 221 |
+
f"RAG={_fmt(row['rag_groundedness'])} no-RAG={_fmt(row['no_rag_groundedness'])} "
|
| 222 |
+
f"| halluc RAG={_fmt(row['rag_hallucination'])} vs no-RAG={_fmt(row['no_rag_hallucination'])}"
|
| 223 |
+
)
|
| 224 |
+
|
| 225 |
+
|
| 226 |
+
def main() -> None:
|
| 227 |
+
parser = argparse.ArgumentParser(description="Run the RAG vs no-RAG ablation.")
|
| 228 |
+
parser.add_argument("--name", default="rag-vs-norag", help="label for this run")
|
| 229 |
+
parser.add_argument("--k", type=int, default=config.EVAL_K, help="retrieval depth for the RAG arm")
|
| 230 |
+
parser.add_argument("--limit", type=int, default=None, help="only the first N questions")
|
| 231 |
+
args = parser.parse_args()
|
| 232 |
+
|
| 233 |
+
items = _load_items(config.TESTSET_PATH, "specific") + _load_items(
|
| 234 |
+
config.GENERAL_TESTSET_PATH, "general"
|
| 235 |
+
)
|
| 236 |
+
if args.limit:
|
| 237 |
+
items = items[: args.limit]
|
| 238 |
+
|
| 239 |
+
print(f"Running ablation '{args.name}' over {len(items)} questions "
|
| 240 |
+
f"(RAG k={args.k} + closed-book) ...")
|
| 241 |
+
report = run(items, k=args.k)
|
| 242 |
+
|
| 243 |
+
_print_summary(report)
|
| 244 |
+
report_path = write_report(report, args.name)
|
| 245 |
+
print(f"\nWrote:\n {report_path}")
|
| 246 |
+
|
| 247 |
+
|
| 248 |
+
if __name__ == "__main__":
|
| 249 |
+
main()
|
eval/run_eval.py
ADDED
|
@@ -0,0 +1,293 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""
|
| 2 |
+
Orchestrator — run the whole evaluation and record the result.
|
| 3 |
+
|
| 4 |
+
For every question in the frozen test set it does ONE retrieval pass, then (unless
|
| 5 |
+
--retrieval-only) generates an answer and judges it, computing both metric tiers:
|
| 6 |
+
|
| 7 |
+
Tier 1 (retrieval) : hit@k, recall@k, MRR -- deterministic, from eval/retrieval_metrics
|
| 8 |
+
Tier 2 (generation): faithfulness, answer_relevancy, context_precision/recall -- eval/judge
|
| 9 |
+
|
| 10 |
+
It writes two things to eval/reports/:
|
| 11 |
+
report.md ONE self-contained doc: summary metrics + every question's full test
|
| 12 |
+
data (question, reference answer, retrieved excerpts, answer, scores).
|
| 13 |
+
Overwritten each run.
|
| 14 |
+
history.csv ONE row per run -- the cross-run performance ledger (kept so you can
|
| 15 |
+
compare runs, e.g. phase1 vs phase2, since report.md only holds the last).
|
| 16 |
+
|
| 17 |
+
Run (needs an index built; the judge defaults to a local LM Studio server — see
|
| 18 |
+
config.JUDGE_MODEL):
|
| 19 |
+
python -m eval.run_eval --name phase1-dense
|
| 20 |
+
python -m eval.run_eval --retrieval-only # fast: skip generation + judge
|
| 21 |
+
python -m eval.run_eval --limit 3 # quick smoke test
|
| 22 |
+
"""
|
| 23 |
+
|
| 24 |
+
from __future__ import annotations
|
| 25 |
+
|
| 26 |
+
import argparse
|
| 27 |
+
import csv
|
| 28 |
+
from datetime import datetime
|
| 29 |
+
from statistics import mean
|
| 30 |
+
|
| 31 |
+
from src import config, generate, retrieve
|
| 32 |
+
from eval import judge
|
| 33 |
+
from eval import retrieval_metrics as rm
|
| 34 |
+
|
| 35 |
+
GEN_METRICS = ["faithfulness", "answer_relevancy", "context_precision", "context_recall"]
|
| 36 |
+
|
| 37 |
+
|
| 38 |
+
def _mean_or_none(values: list[float]):
|
| 39 |
+
vals = [v for v in values if v is not None]
|
| 40 |
+
return round(mean(vals), 4) if vals else None
|
| 41 |
+
|
| 42 |
+
|
| 43 |
+
def run(testset: list[dict], k: int, do_judge: bool) -> dict:
|
| 44 |
+
"""Evaluate every test item and return the full report dict."""
|
| 45 |
+
per_item = []
|
| 46 |
+
for n, item in enumerate(testset, start=1):
|
| 47 |
+
# --- Tier 1: retrieve once, score deterministically -----------------------------
|
| 48 |
+
results = retrieve.retrieve(item["question"], top_k=k)
|
| 49 |
+
retrieved_ids = [r.id for r in results]
|
| 50 |
+
gold_ids = item["gold_ids"]
|
| 51 |
+
record = {
|
| 52 |
+
"id": item["id"],
|
| 53 |
+
"question": item["question"],
|
| 54 |
+
"reference_answer": item.get("reference_answer", ""),
|
| 55 |
+
"gold_ids": gold_ids,
|
| 56 |
+
"retrieved_ids": retrieved_ids,
|
| 57 |
+
"retrieved_titles": [r.recipe["title"] for r in results],
|
| 58 |
+
"retrieval": {
|
| 59 |
+
"hit": rm.hit_at_k(retrieved_ids, gold_ids, k),
|
| 60 |
+
"recall": rm.recall_at_k(retrieved_ids, gold_ids, k),
|
| 61 |
+
"rr": rm.reciprocal_rank(retrieved_ids, gold_ids),
|
| 62 |
+
},
|
| 63 |
+
}
|
| 64 |
+
|
| 65 |
+
# --- Tier 2: generate an answer, then judge it ----------------------------------
|
| 66 |
+
if do_judge:
|
| 67 |
+
contexts = [r.recipe["text"] for r in results]
|
| 68 |
+
answer = generate.generate_answer(item["question"], results)
|
| 69 |
+
record["answer"] = answer
|
| 70 |
+
record["judge"] = judge.judge_item(
|
| 71 |
+
item["question"], answer, contexts, item["reference_answer"]
|
| 72 |
+
)
|
| 73 |
+
faith = record["judge"]["faithfulness"]
|
| 74 |
+
print(
|
| 75 |
+
f" [{n}/{len(testset)}] {item['id']} hit={record['retrieval']['hit']:.0f}"
|
| 76 |
+
f" faithfulness={'-' if faith is None else f'{faith:.2f}'}"
|
| 77 |
+
)
|
| 78 |
+
else:
|
| 79 |
+
print(f" [{n}/{len(testset)}] {item['id']} hit={record['retrieval']['hit']:.0f}")
|
| 80 |
+
|
| 81 |
+
per_item.append(record)
|
| 82 |
+
|
| 83 |
+
return _aggregate(per_item, k, do_judge)
|
| 84 |
+
|
| 85 |
+
|
| 86 |
+
def _aggregate(per_item: list[dict], k: int, do_judge: bool) -> dict:
|
| 87 |
+
n = len(per_item) or 1
|
| 88 |
+
retrieval = {
|
| 89 |
+
"hit@k": round(sum(x["retrieval"]["hit"] for x in per_item) / n, 4),
|
| 90 |
+
"recall@k": round(sum(x["retrieval"]["recall"] for x in per_item) / n, 4),
|
| 91 |
+
"mrr": round(sum(x["retrieval"]["rr"] for x in per_item) / n, 4),
|
| 92 |
+
}
|
| 93 |
+
|
| 94 |
+
generation = None
|
| 95 |
+
hallucination_rate = None
|
| 96 |
+
if do_judge:
|
| 97 |
+
generation = {
|
| 98 |
+
m: _mean_or_none([x["judge"][m] for x in per_item]) for m in GEN_METRICS
|
| 99 |
+
}
|
| 100 |
+
if generation["faithfulness"] is not None:
|
| 101 |
+
hallucination_rate = round(1.0 - generation["faithfulness"], 4)
|
| 102 |
+
|
| 103 |
+
return {
|
| 104 |
+
"run": {
|
| 105 |
+
"timestamp": datetime.now().isoformat(timespec="seconds"),
|
| 106 |
+
"k": k,
|
| 107 |
+
"num_questions": len(per_item),
|
| 108 |
+
"judged": do_judge,
|
| 109 |
+
"retrieval_mode": config.RETRIEVAL_MODE,
|
| 110 |
+
"chat_model": config.CHAT_MODEL,
|
| 111 |
+
"judge_model": config.JUDGE_MODEL,
|
| 112 |
+
"embedding_model": config.EMBEDDING_MODEL,
|
| 113 |
+
"corpus_size": len(retrieve._load_index()[1]), # indexed chunk count
|
| 114 |
+
},
|
| 115 |
+
"aggregate": {
|
| 116 |
+
"retrieval": retrieval,
|
| 117 |
+
"generation": generation,
|
| 118 |
+
"hallucination_rate": hallucination_rate,
|
| 119 |
+
},
|
| 120 |
+
"per_item": per_item,
|
| 121 |
+
}
|
| 122 |
+
|
| 123 |
+
|
| 124 |
+
# ----------------------------------------------------------------------------------------
|
| 125 |
+
# Output: a single, self-contained markdown report, plus the history.csv ledger.
|
| 126 |
+
# ----------------------------------------------------------------------------------------
|
| 127 |
+
|
| 128 |
+
HISTORY_COLUMNS = [
|
| 129 |
+
"timestamp", "name", "k", "num_questions", "judged",
|
| 130 |
+
"chat_model", "judge_model",
|
| 131 |
+
"hit@k", "recall@k", "mrr",
|
| 132 |
+
"faithfulness", "answer_relevancy", "context_precision", "context_recall",
|
| 133 |
+
"hallucination_rate",
|
| 134 |
+
]
|
| 135 |
+
|
| 136 |
+
|
| 137 |
+
def _fmt(value) -> str:
|
| 138 |
+
return "-" if value is None else f"{value:.3f}"
|
| 139 |
+
|
| 140 |
+
|
| 141 |
+
def write_report(report: dict, name: str):
|
| 142 |
+
"""Write the single markdown report and append one row to the history ledger."""
|
| 143 |
+
config.REPORTS_DIR.mkdir(parents=True, exist_ok=True)
|
| 144 |
+
config.REPORT_PATH.write_text(_render_markdown(report, name), encoding="utf-8")
|
| 145 |
+
_append_history(report, name)
|
| 146 |
+
return config.REPORT_PATH
|
| 147 |
+
|
| 148 |
+
|
| 149 |
+
def _render_markdown(report: dict, name: str) -> str:
|
| 150 |
+
"""One concise markdown doc: a summary table, then every question's full test data."""
|
| 151 |
+
run = report["run"]
|
| 152 |
+
agg = report["aggregate"]
|
| 153 |
+
ret = agg["retrieval"]
|
| 154 |
+
gen = agg["generation"]
|
| 155 |
+
k = run["k"]
|
| 156 |
+
|
| 157 |
+
# --- header + summary metric table ---
|
| 158 |
+
lines = [
|
| 159 |
+
f"# UBCHelper eval — {name}",
|
| 160 |
+
"",
|
| 161 |
+
f"{run['timestamp']} · {run['num_questions']} questions · k={k} · "
|
| 162 |
+
f"judge {'on' if run['judged'] else 'off'}",
|
| 163 |
+
"",
|
| 164 |
+
f"Models: chat=`{run['chat_model']}`, judge=`{run['judge_model']}`, "
|
| 165 |
+
f"embed=`{run['embedding_model']}` · corpus={run['corpus_size']} · "
|
| 166 |
+
f"retrieval={run['retrieval_mode']}",
|
| 167 |
+
"",
|
| 168 |
+
"## Summary",
|
| 169 |
+
"",
|
| 170 |
+
"| metric | score |",
|
| 171 |
+
"|--------|-------|",
|
| 172 |
+
f"| hit@{k} | {_fmt(ret['hit@k'])} |",
|
| 173 |
+
f"| recall@{k} | {_fmt(ret['recall@k'])} |",
|
| 174 |
+
f"| MRR | {_fmt(ret['mrr'])} |",
|
| 175 |
+
]
|
| 176 |
+
if gen is not None:
|
| 177 |
+
lines += [
|
| 178 |
+
f"| faithfulness | {_fmt(gen['faithfulness'])} |",
|
| 179 |
+
f"| answer_relevancy | {_fmt(gen['answer_relevancy'])} |",
|
| 180 |
+
f"| context_precision | {_fmt(gen['context_precision'])} |",
|
| 181 |
+
f"| context_recall | {_fmt(gen['context_recall'])} |",
|
| 182 |
+
f"| **hallucination_rate** | **{_fmt(agg['hallucination_rate'])}** |",
|
| 183 |
+
]
|
| 184 |
+
|
| 185 |
+
# --- per-question detail: all test data, one block each ---
|
| 186 |
+
lines += ["", "## Questions", ""]
|
| 187 |
+
for x in report["per_item"]:
|
| 188 |
+
r = x["retrieval"]
|
| 189 |
+
marker = "hit" if r["hit"] else "miss"
|
| 190 |
+
retrieved = " · ".join(
|
| 191 |
+
f"[{i}] {t}" for i, t in zip(x["retrieved_ids"], x["retrieved_titles"])
|
| 192 |
+
)
|
| 193 |
+
lines += [
|
| 194 |
+
f"### {x['id']} — {marker} — gold {x['gold_ids']}",
|
| 195 |
+
"",
|
| 196 |
+
f"**Q:** {x['question']}",
|
| 197 |
+
"",
|
| 198 |
+
f"**Reference:** {x['reference_answer']}",
|
| 199 |
+
"",
|
| 200 |
+
f"**Retrieved:** {retrieved}",
|
| 201 |
+
]
|
| 202 |
+
if "judge" in x:
|
| 203 |
+
j = x["judge"]
|
| 204 |
+
lines += [
|
| 205 |
+
"",
|
| 206 |
+
f"**Answer:** {x['answer']}",
|
| 207 |
+
"",
|
| 208 |
+
f"**Scores:** rr {r['rr']:.2f} · faithfulness {_fmt(j['faithfulness'])} "
|
| 209 |
+
f"· answer_relevancy {_fmt(j['answer_relevancy'])} "
|
| 210 |
+
f"· context_precision {_fmt(j['context_precision'])} "
|
| 211 |
+
f"· context_recall {_fmt(j['context_recall'])}"
|
| 212 |
+
+ (f" · parse-failed: {', '.join(j['errors'])}" if j["errors"] else ""),
|
| 213 |
+
]
|
| 214 |
+
else:
|
| 215 |
+
lines += ["", f"**Scores:** rr {r['rr']:.2f}"]
|
| 216 |
+
lines.append("")
|
| 217 |
+
|
| 218 |
+
return "\n".join(lines) + "\n"
|
| 219 |
+
|
| 220 |
+
|
| 221 |
+
def _append_history(report: dict, name: str) -> None:
|
| 222 |
+
run = report["run"]
|
| 223 |
+
ret = report["aggregate"]["retrieval"]
|
| 224 |
+
gen = report["aggregate"]["generation"] or {}
|
| 225 |
+
row = {
|
| 226 |
+
"timestamp": run["timestamp"],
|
| 227 |
+
"name": name,
|
| 228 |
+
"k": run["k"],
|
| 229 |
+
"num_questions": run["num_questions"],
|
| 230 |
+
"judged": run["judged"],
|
| 231 |
+
"chat_model": run["chat_model"],
|
| 232 |
+
"judge_model": run["judge_model"],
|
| 233 |
+
"hit@k": ret["hit@k"],
|
| 234 |
+
"recall@k": ret["recall@k"],
|
| 235 |
+
"mrr": ret["mrr"],
|
| 236 |
+
"faithfulness": gen.get("faithfulness"),
|
| 237 |
+
"answer_relevancy": gen.get("answer_relevancy"),
|
| 238 |
+
"context_precision": gen.get("context_precision"),
|
| 239 |
+
"context_recall": gen.get("context_recall"),
|
| 240 |
+
"hallucination_rate": report["aggregate"]["hallucination_rate"],
|
| 241 |
+
}
|
| 242 |
+
is_new = not config.HISTORY_PATH.exists()
|
| 243 |
+
with open(config.HISTORY_PATH, "a", encoding="utf-8", newline="") as f:
|
| 244 |
+
writer = csv.DictWriter(f, fieldnames=HISTORY_COLUMNS)
|
| 245 |
+
if is_new:
|
| 246 |
+
writer.writeheader()
|
| 247 |
+
writer.writerow(row)
|
| 248 |
+
|
| 249 |
+
|
| 250 |
+
def _print_summary(report: dict) -> None:
|
| 251 |
+
agg = report["aggregate"]
|
| 252 |
+
ret = agg["retrieval"]
|
| 253 |
+
k = report["run"]["k"]
|
| 254 |
+
print("\n=== Summary ===")
|
| 255 |
+
print(f"Retrieval : hit@{k}={_fmt(ret['hit@k'])} "
|
| 256 |
+
f"recall@{k}={_fmt(ret['recall@k'])} mrr={_fmt(ret['mrr'])}")
|
| 257 |
+
gen = agg["generation"]
|
| 258 |
+
if gen is not None:
|
| 259 |
+
print(f"Generation: faithfulness={_fmt(gen['faithfulness'])} "
|
| 260 |
+
f"answer_relevancy={_fmt(gen['answer_relevancy'])}")
|
| 261 |
+
print(f" context_precision={_fmt(gen['context_precision'])} "
|
| 262 |
+
f"context_recall={_fmt(gen['context_recall'])}")
|
| 263 |
+
print(f"Hallucination rate (1 - faithfulness): "
|
| 264 |
+
f"{_fmt(agg['hallucination_rate'])}")
|
| 265 |
+
|
| 266 |
+
|
| 267 |
+
def main() -> None:
|
| 268 |
+
parser = argparse.ArgumentParser(description="Run the UBCHelper evaluation harness.")
|
| 269 |
+
parser.add_argument("--name", default="run", help="label for this run (e.g. phase1-dense)")
|
| 270 |
+
parser.add_argument("--k", type=int, default=config.EVAL_K, help="retrieval depth / @k")
|
| 271 |
+
parser.add_argument("--limit", type=int, default=None, help="only the first N questions")
|
| 272 |
+
parser.add_argument(
|
| 273 |
+
"--retrieval-only",
|
| 274 |
+
action="store_true",
|
| 275 |
+
help="skip generation + the LLM judge (fast, deterministic, no chat model needed)",
|
| 276 |
+
)
|
| 277 |
+
args = parser.parse_args()
|
| 278 |
+
|
| 279 |
+
testset = rm.load_testset()
|
| 280 |
+
if args.limit:
|
| 281 |
+
testset = testset[: args.limit]
|
| 282 |
+
|
| 283 |
+
print(f"Running '{args.name}' over {len(testset)} questions (k={args.k}, "
|
| 284 |
+
f"mode={config.RETRIEVAL_MODE}, judge={'off' if args.retrieval_only else 'on'}) ...")
|
| 285 |
+
report = run(testset, k=args.k, do_judge=not args.retrieval_only)
|
| 286 |
+
|
| 287 |
+
_print_summary(report)
|
| 288 |
+
report_path = write_report(report, args.name)
|
| 289 |
+
print(f"\nWrote:\n {report_path}\n {config.HISTORY_PATH}")
|
| 290 |
+
|
| 291 |
+
|
| 292 |
+
if __name__ == "__main__":
|
| 293 |
+
main()
|
eval/testset.general.json
ADDED
|
@@ -0,0 +1,33 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"description": "Broad questions a closed-book model can reasonably attempt, each still bound to a gold calendar chunk — used by eval/run_ablation.py as the 'general' arm. gold_ids are valid only for the 3,770-chunk index built 2026-08-31; regenerate after any re-ingest.",
|
| 3 |
+
"items": [
|
| 4 |
+
{
|
| 5 |
+
"id": "g001",
|
| 6 |
+
"question": "What happens to a UBC student's academic standing, and what standings exist?",
|
| 7 |
+
"reference_answer": "There are three levels of academic standing: In Good Standing, On Academic Probation, and Failed, Required to Withdraw. Students enter the University In Good Standing.",
|
| 8 |
+
"gold_ids": [0],
|
| 9 |
+
"source_chunk_id": 0
|
| 10 |
+
},
|
| 11 |
+
{
|
| 12 |
+
"id": "g002",
|
| 13 |
+
"question": "Roughly how many credits does a UBC B.Sc. degree require?",
|
| 14 |
+
"reference_answer": "A minimum of 120 credits; some options require more.",
|
| 15 |
+
"gold_ids": [2023],
|
| 16 |
+
"source_chunk_id": 2023
|
| 17 |
+
},
|
| 18 |
+
{
|
| 19 |
+
"id": "g003",
|
| 20 |
+
"question": "What is CPSC 110 at UBC about?",
|
| 21 |
+
"reference_answer": "CPSC 110, Computation, Programs, and Programming (4 credits), covers fundamental program and computation structures.",
|
| 22 |
+
"gold_ids": [5],
|
| 23 |
+
"source_chunk_id": 5
|
| 24 |
+
},
|
| 25 |
+
{
|
| 26 |
+
"id": "g004",
|
| 27 |
+
"question": "What kinds of requirements does a UBC Bachelor of Arts student have to complete?",
|
| 28 |
+
"reference_answer": "B.A. students complete a set of faculty-wide degree requirements — for current entrants (2024/25 or later): Writing and Research, Ways of Knowing Breadth, Outside, Upper-level, and Arts Credit Minimum; earlier cohorts also had separate Language, Science, and Literature requirements.",
|
| 29 |
+
"gold_ids": [128, 114],
|
| 30 |
+
"source_chunk_id": 128
|
| 31 |
+
}
|
| 32 |
+
]
|
| 33 |
+
}
|
eval/testset.json
ADDED
|
@@ -0,0 +1,101 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"description": "Hand-curated, corpus-grounded test set (authored directly from indexed chunks; regenerate candidates with eval/make_testset.py once a local LLM backend is available). Categories: code-lookup (hinges on a course code — BM25's job), multi-hop (spans chunks — the agent/reranker's job), collision (edition- or cohort-qualified — the corpus's trap), policy. gold_ids are valid ONLY for the 3,770-chunk index built 2026-08-31 from the full live-2627 + archive-2526 crawl; regenerate after any re-ingest.",
|
| 3 |
+
"items": [
|
| 4 |
+
{
|
| 5 |
+
"id": "q001",
|
| 6 |
+
"category": "code-lookup",
|
| 7 |
+
"question": "What are the prerequisites for CPSC 221?",
|
| 8 |
+
"reference_answer": "One of CPSC 210 or CPEN 221, and either one of CPSC 121, MATH 220 (or MATH_O 220), or a score of 68% or higher in MATH 226.",
|
| 9 |
+
"gold_ids": [10, 182],
|
| 10 |
+
"source_chunk_id": 10
|
| 11 |
+
},
|
| 12 |
+
{
|
| 13 |
+
"id": "q002",
|
| 14 |
+
"category": "code-lookup",
|
| 15 |
+
"question": "What are the prerequisites for CPSC 320?",
|
| 16 |
+
"reference_answer": "CPSC 221 (or DSCI 221) plus at least 3 credits of MATH or STAT at the 200 level or above (the 2025/26 edition also accepted COMM 291 or BIOL 300 for the second part).",
|
| 17 |
+
"gold_ids": [24, 197],
|
| 18 |
+
"source_chunk_id": 24
|
| 19 |
+
},
|
| 20 |
+
{
|
| 21 |
+
"id": "q003",
|
| 22 |
+
"category": "code-lookup",
|
| 23 |
+
"question": "What are the prerequisites for CPSC 313?",
|
| 24 |
+
"reference_answer": "CPSC 213, and either CPSC 221 or DSCI 221.",
|
| 25 |
+
"gold_ids": [20, 193],
|
| 26 |
+
"source_chunk_id": 20
|
| 27 |
+
},
|
| 28 |
+
{
|
| 29 |
+
"id": "q004",
|
| 30 |
+
"category": "code-lookup",
|
| 31 |
+
"question": "Which courses satisfy the prerequisite for MATH 200, Calculus III?",
|
| 32 |
+
"reference_answer": "One of MATH 101, MATH 103, MATH 105, MATH 121, SCIE 001, MATH_O 101, or MATH_O 103.",
|
| 33 |
+
"gold_ids": [580, 2319],
|
| 34 |
+
"source_chunk_id": 580
|
| 35 |
+
},
|
| 36 |
+
{
|
| 37 |
+
"id": "q005",
|
| 38 |
+
"category": "code-lookup",
|
| 39 |
+
"question": "How many credits is CPSC 110 worth and what does it cover?",
|
| 40 |
+
"reference_answer": "CPSC 110 (Computation, Programs, and Programming) is worth 4 credits and covers fundamental program and computation structures.",
|
| 41 |
+
"gold_ids": [5, 177],
|
| 42 |
+
"source_chunk_id": 5
|
| 43 |
+
},
|
| 44 |
+
{
|
| 45 |
+
"id": "q006",
|
| 46 |
+
"category": "multi-hop",
|
| 47 |
+
"question": "Compare the B.A. degree requirements for students who entered the program in 2023/24 with those who entered in 2024/25 or later.",
|
| 48 |
+
"reference_answer": "Students entering 2023/24 or earlier must complete seven requirements: Writing and Research, Language, Science, Literature, Outside, Upper-level, and Arts Credit Minimum. Students entering 2024/25 or later complete five: Writing and Research, Ways of Knowing Breadth, Outside, Upper-level, and Arts Credit Minimum — the separate Language, Science, and Literature requirements are folded into the Ways of Knowing Breadth requirement.",
|
| 49 |
+
"gold_ids": [114, 128],
|
| 50 |
+
"source_chunk_id": 114
|
| 51 |
+
},
|
| 52 |
+
{
|
| 53 |
+
"id": "q007",
|
| 54 |
+
"category": "multi-hop",
|
| 55 |
+
"question": "How does admission to the Bachelor of Computer Science program differ from B.Sc. admission from secondary school?",
|
| 56 |
+
"reference_answer": "Bachelor of Computer Science admission is a competitive selection process evaluating applicants on academic and other criteria (not every qualified applicant is admitted); B.Sc. admission from secondary school requires starting first year in September of the Winter session of admission (January starts are not permitted).",
|
| 57 |
+
"gold_ids": [1769, 1791],
|
| 58 |
+
"source_chunk_id": 1769
|
| 59 |
+
},
|
| 60 |
+
{
|
| 61 |
+
"id": "q008",
|
| 62 |
+
"category": "multi-hop",
|
| 63 |
+
"question": "I have finished CPSC 210. What else do I need to complete before I can take CPSC 313?",
|
| 64 |
+
"reference_answer": "CPSC 313 requires CPSC 213 and either CPSC 221 or DSCI 221. CPSC 221 in turn requires (beyond CPSC 210) one of CPSC 121, MATH 220, MATH_O 220, or a 68%+ score in MATH 226. So after CPSC 210 you still need CPSC 213 and CPSC 221 (with its discrete-math prerequisite).",
|
| 65 |
+
"gold_ids": [20, 10],
|
| 66 |
+
"source_chunk_id": 20
|
| 67 |
+
},
|
| 68 |
+
{
|
| 69 |
+
"id": "q009",
|
| 70 |
+
"category": "collision",
|
| 71 |
+
"question": "In the 2025/26 calendar, what is the minimum number of credits required for a B.Sc. degree?",
|
| 72 |
+
"reference_answer": "A minimum of 120 credits (a major, double major, or General Science option requires at least 120 credits but may require more).",
|
| 73 |
+
"gold_ids": [3508],
|
| 74 |
+
"source_chunk_id": 3508
|
| 75 |
+
},
|
| 76 |
+
{
|
| 77 |
+
"id": "q010",
|
| 78 |
+
"category": "collision",
|
| 79 |
+
"question": "According to the current 2026/27 calendar, what is the minimum number of credits required for a B.Sc. degree?",
|
| 80 |
+
"reference_answer": "A minimum of 120 credits (a major, double major, or General Science option requires at least 120 credits but may require more).",
|
| 81 |
+
"gold_ids": [2023],
|
| 82 |
+
"source_chunk_id": 2023
|
| 83 |
+
},
|
| 84 |
+
{
|
| 85 |
+
"id": "q011",
|
| 86 |
+
"category": "collision",
|
| 87 |
+
"question": "Does a B.A. student who entered the program in 2024/25 or later have to complete a language requirement?",
|
| 88 |
+
"reference_answer": "No. Their five degree requirements are Writing and Research, Ways of Knowing Breadth, Outside, Upper-level, and Arts Credit Minimum — there is no separate language requirement (that applies to students who entered in 2023/24 or earlier).",
|
| 89 |
+
"gold_ids": [128],
|
| 90 |
+
"source_chunk_id": 128
|
| 91 |
+
},
|
| 92 |
+
{
|
| 93 |
+
"id": "q012",
|
| 94 |
+
"category": "policy",
|
| 95 |
+
"question": "What are the three levels of academic standing at UBC?",
|
| 96 |
+
"reference_answer": "In Good Standing; On Academic Probation; and Failed, Required to Withdraw. All students are In Good Standing on initial entry to the University.",
|
| 97 |
+
"gold_ids": [0],
|
| 98 |
+
"source_chunk_id": 0
|
| 99 |
+
}
|
| 100 |
+
]
|
| 101 |
+
}
|
src/__init__.py
ADDED
|
@@ -0,0 +1,6 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""UBCHelper — a retrieval-augmented UBC Vancouver Academic Calendar advisor.
|
| 2 |
+
|
| 3 |
+
A domain re-skin of RAGChef: the same generic retrieval/generation engine over a scraped
|
| 4 |
+
Academic Calendar corpus. This package currently holds the data layer — scrape -> chunk ->
|
| 5 |
+
ingest — which fills the engine's document index.
|
| 6 |
+
"""
|
src/agent.py
ADDED
|
@@ -0,0 +1,96 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""
|
| 2 |
+
Agent — the complex path of the adaptive pipeline (Phase 4), framework-free.
|
| 3 |
+
|
| 4 |
+
A plain Python loop, no LangChain/LangGraph, so the whole thing stays readable:
|
| 5 |
+
|
| 6 |
+
decompose(question) -> sub-questions one LLM call
|
| 7 |
+
for each sub-question: retrieve.retrieve(...) multi-hop recall
|
| 8 |
+
round-robin merge + dedup the Result lists each hop's best hit rises to the top
|
| 9 |
+
synthesize(question, merged) one grounded answer over the union
|
| 10 |
+
|
| 11 |
+
The merge is what earns the complex path its keep: a single retrieval tends to satisfy one
|
| 12 |
+
part of a multi-hop question and starve the other, whereas retrieving each sub-question
|
| 13 |
+
separately and interleaving the rankings surfaces an excerpt for *every* hop — the recall
|
| 14 |
+
win the eval measures.
|
| 15 |
+
|
| 16 |
+
Synthesis deliberately reuses `generate.generate_answer`: the grounded SYSTEM_PROMPT and
|
| 17 |
+
`format_context` already turn a list of excerpts + the original question into one clean
|
| 18 |
+
answer, so there is nothing to reinvent here.
|
| 19 |
+
"""
|
| 20 |
+
|
| 21 |
+
from __future__ import annotations
|
| 22 |
+
|
| 23 |
+
from . import config, generate, retrieve, trace
|
| 24 |
+
from .retrieve import Result
|
| 25 |
+
|
| 26 |
+
DECOMPOSE_SYSTEM = (
|
| 27 |
+
"You break a complex question about the UBC Academic Calendar into a few focused "
|
| 28 |
+
"sub-questions, each of which can be answered from ONE calendar page (one course, "
|
| 29 |
+
"one program's requirements, one policy). Cover every distinct course, program, "
|
| 30 |
+
"student cohort, or side of a comparison the question asks about. Write one "
|
| 31 |
+
"sub-question per line, no numbering, and produce at most {max_subq}. If the "
|
| 32 |
+
"question is already single-focus, just restate it as the only line."
|
| 33 |
+
)
|
| 34 |
+
|
| 35 |
+
|
| 36 |
+
def decompose(question: str) -> list[str]:
|
| 37 |
+
"""Split `question` into up to AGENT_MAX_SUBQ sub-questions; fall back to the original."""
|
| 38 |
+
trace.step("AGENT - decompose into sub-questions")
|
| 39 |
+
system = DECOMPOSE_SYSTEM.format(max_subq=config.AGENT_MAX_SUBQ)
|
| 40 |
+
reply = generate.complete(system, question)
|
| 41 |
+
subqs = [_strip_bullet(line) for line in reply.splitlines()]
|
| 42 |
+
subqs = [s for s in subqs if s]
|
| 43 |
+
subqs = subqs[: config.AGENT_MAX_SUBQ] if subqs else [question]
|
| 44 |
+
for i, sq in enumerate(subqs, start=1):
|
| 45 |
+
trace.detail(f"sub-question {i}", sq)
|
| 46 |
+
return subqs
|
| 47 |
+
|
| 48 |
+
|
| 49 |
+
def _strip_bullet(line: str) -> str:
|
| 50 |
+
"""Drop any leading list marker ("1.", "-", "*") the model may have added."""
|
| 51 |
+
return line.strip().lstrip("0123456789.)-*• \t").strip()
|
| 52 |
+
|
| 53 |
+
|
| 54 |
+
def _merge(result_lists: list[list[Result]]) -> list[Result]:
|
| 55 |
+
"""Round-robin interleave per-sub-question results, deduped by chunk id (first wins).
|
| 56 |
+
|
| 57 |
+
Taking rank-1 from every sub-question before any rank-2 keeps each hop's strongest hit
|
| 58 |
+
near the top, which is exactly what recall@k on a multi-hop question rewards.
|
| 59 |
+
"""
|
| 60 |
+
merged: list[Result] = []
|
| 61 |
+
seen: set[int] = set()
|
| 62 |
+
for rank in range(max((len(rl) for rl in result_lists), default=0)):
|
| 63 |
+
for rl in result_lists:
|
| 64 |
+
if rank < len(rl) and rl[rank].id not in seen:
|
| 65 |
+
seen.add(rl[rank].id)
|
| 66 |
+
merged.append(rl[rank])
|
| 67 |
+
return merged
|
| 68 |
+
|
| 69 |
+
|
| 70 |
+
def synthesize(question: str, results: list[Result]) -> str:
|
| 71 |
+
"""Answer the original question grounded in the merged excerpts (capped for prompt size)."""
|
| 72 |
+
return generate.generate_answer(question, results[: config.SYNTH_CONTEXT_K])
|
| 73 |
+
|
| 74 |
+
|
| 75 |
+
def run(question: str, top_k: int | None = None) -> tuple[list[Result], str]:
|
| 76 |
+
"""Decompose -> multi-hop retrieve -> merge -> synthesize. Returns (merged_results, answer).
|
| 77 |
+
|
| 78 |
+
`top_k` sets how many excerpts to pull per sub-question (defaults to AGENT_SUBQ_TOP_K);
|
| 79 |
+
the merged list can be longer, since it unions every hop.
|
| 80 |
+
"""
|
| 81 |
+
per_hop = top_k or config.AGENT_SUBQ_TOP_K
|
| 82 |
+
subqs = decompose(question)
|
| 83 |
+
|
| 84 |
+
result_lists = []
|
| 85 |
+
for i, sq in enumerate(subqs, start=1):
|
| 86 |
+
trace.step(f"AGENT - multi-hop retrieve, sub-question {i}/{len(subqs)}: {sq!r}")
|
| 87 |
+
result_lists.append(retrieve.retrieve(sq, top_k=per_hop))
|
| 88 |
+
|
| 89 |
+
merged = _merge(result_lists)
|
| 90 |
+
trace.step("AGENT - merge sub-question results (round-robin, dedup by chunk id)")
|
| 91 |
+
trace.detail("merged excerpts", len(merged))
|
| 92 |
+
trace.results(merged)
|
| 93 |
+
|
| 94 |
+
trace.step("AGENT - synthesize final answer over the merged excerpts")
|
| 95 |
+
answer = synthesize(question, merged)
|
| 96 |
+
return merged, answer
|
src/bm25.py
ADDED
|
@@ -0,0 +1,115 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""
|
| 2 |
+
BM25 — hand-rolled sparse (keyword) retrieval, the other half of hybrid search.
|
| 3 |
+
|
| 4 |
+
Dense embeddings capture *meaning* but under-weight rare, high-signal tokens: a query
|
| 5 |
+
about "CPSC 320" lands near generic computer-science text because the code barely moves
|
| 6 |
+
the vector. BM25 is the opposite — it scores documents by exact term overlap, weighting
|
| 7 |
+
each term by how rare it is in the corpus (IDF), saturating repeated occurrences (k1), and
|
| 8 |
+
penalizing long documents (b). Rare course-code/program tokens are exactly where it
|
| 9 |
+
shines, which is why fusing it with dense retrieval helps.
|
| 10 |
+
|
| 11 |
+
This is the classic Okapi BM25 formula, built with numpy so the mechanism stays visible
|
| 12 |
+
(same spirit as the hand-rolled vector store):
|
| 13 |
+
|
| 14 |
+
score(q, d) = sum over query terms t of
|
| 15 |
+
idf(t) * tf(t,d) * (k1 + 1) / (tf(t,d) + k1 * (1 - b + b * |d| / avgdl))
|
| 16 |
+
idf(t) = ln(1 + (N - df(t) + 0.5) / (df(t) + 0.5))
|
| 17 |
+
|
| 18 |
+
Everything inside the sum depends only on the corpus, so each term's per-document
|
| 19 |
+
contribution is precomputed at build time; scoring a query is then just summing a few
|
| 20 |
+
postings arrays. The index is built in memory from the text the dense index already
|
| 21 |
+
stores (a few seconds even for tens of thousands of chunks, once per process) — no new artifact on disk and
|
| 22 |
+
no re-ingest needed.
|
| 23 |
+
"""
|
| 24 |
+
|
| 25 |
+
from __future__ import annotations
|
| 26 |
+
|
| 27 |
+
import math
|
| 28 |
+
import re
|
| 29 |
+
from collections import Counter
|
| 30 |
+
from dataclasses import dataclass
|
| 31 |
+
|
| 32 |
+
import numpy as np
|
| 33 |
+
|
| 34 |
+
# Standard Okapi parameters.
|
| 35 |
+
K1 = 1.5 # term-frequency saturation: repeats help, with diminishing returns
|
| 36 |
+
B = 0.75 # document-length normalization strength
|
| 37 |
+
|
| 38 |
+
_TOKEN_RE = re.compile(r"[a-z0-9]+")
|
| 39 |
+
|
| 40 |
+
|
| 41 |
+
def tokenize(text: str) -> list[str]:
|
| 42 |
+
"""Lowercase word/number tokens. Used for BOTH documents and queries — the sparse
|
| 43 |
+
analogue of "the same embedding model must embed the index and the query"."""
|
| 44 |
+
return _TOKEN_RE.findall(text.lower())
|
| 45 |
+
|
| 46 |
+
|
| 47 |
+
@dataclass
|
| 48 |
+
class _Index:
|
| 49 |
+
# term -> (doc indices, that term's precomputed idf*tf-saturation score per doc)
|
| 50 |
+
postings: dict[str, tuple[np.ndarray, np.ndarray]]
|
| 51 |
+
n_docs: int
|
| 52 |
+
|
| 53 |
+
|
| 54 |
+
def _build(texts: list[str]) -> _Index:
|
| 55 |
+
docs = [Counter(tokenize(t)) for t in texts]
|
| 56 |
+
doc_lens = np.array([sum(c.values()) for c in docs], dtype=np.float64)
|
| 57 |
+
avgdl = doc_lens.mean() or 1.0
|
| 58 |
+
n = len(texts)
|
| 59 |
+
|
| 60 |
+
# Document frequency: in how many docs does each term appear?
|
| 61 |
+
df: Counter[str] = Counter()
|
| 62 |
+
for counts in docs:
|
| 63 |
+
df.update(counts.keys())
|
| 64 |
+
|
| 65 |
+
# Per-document length penalty (the k1 * (1 - b + b*|d|/avgdl) denominator part).
|
| 66 |
+
norm = K1 * (1 - B + B * doc_lens / avgdl)
|
| 67 |
+
|
| 68 |
+
by_term: dict[str, tuple[list[int], list[float]]] = {t: ([], []) for t in df}
|
| 69 |
+
for i, counts in enumerate(docs):
|
| 70 |
+
for term, tf in counts.items():
|
| 71 |
+
indices, weights = by_term[term]
|
| 72 |
+
indices.append(i)
|
| 73 |
+
weights.append(tf * (K1 + 1) / (tf + norm[i]))
|
| 74 |
+
|
| 75 |
+
postings = {}
|
| 76 |
+
for term, (indices, weights) in by_term.items():
|
| 77 |
+
idf = math.log(1 + (n - df[term] + 0.5) / (df[term] + 0.5))
|
| 78 |
+
postings[term] = (
|
| 79 |
+
np.array(indices, dtype=np.int32),
|
| 80 |
+
idf * np.array(weights, dtype=np.float32),
|
| 81 |
+
)
|
| 82 |
+
return _Index(postings=postings, n_docs=n)
|
| 83 |
+
|
| 84 |
+
|
| 85 |
+
# Built once per process on first search, cached for the corpus it was built from.
|
| 86 |
+
_index: _Index | None = None
|
| 87 |
+
_source: list[str] | None = None
|
| 88 |
+
|
| 89 |
+
|
| 90 |
+
def _get_index(texts: list[str]) -> _Index:
|
| 91 |
+
global _index, _source
|
| 92 |
+
if _index is None or _source is not texts:
|
| 93 |
+
_index = _build(texts)
|
| 94 |
+
_source = texts
|
| 95 |
+
return _index
|
| 96 |
+
|
| 97 |
+
|
| 98 |
+
def search(query: str, k: int, texts: list[str]) -> list[tuple[int, float]]:
|
| 99 |
+
"""Return the top-k (doc index, BM25 score) pairs for `query`, best first.
|
| 100 |
+
|
| 101 |
+
Only documents sharing at least one term with the query are returned, so the list
|
| 102 |
+
can be shorter than k.
|
| 103 |
+
"""
|
| 104 |
+
index = _get_index(texts)
|
| 105 |
+
scores = np.zeros(index.n_docs, dtype=np.float32)
|
| 106 |
+
for term in tokenize(query):
|
| 107 |
+
entry = index.postings.get(term)
|
| 108 |
+
if entry is not None:
|
| 109 |
+
indices, weights = entry
|
| 110 |
+
scores[indices] += weights
|
| 111 |
+
|
| 112 |
+
k = min(k, index.n_docs)
|
| 113 |
+
top_idx = np.argpartition(scores, -k)[-k:]
|
| 114 |
+
top_idx = top_idx[np.argsort(scores[top_idx])[::-1]]
|
| 115 |
+
return [(int(i), float(scores[i])) for i in top_idx if scores[i] > 0]
|
src/generate.py
ADDED
|
@@ -0,0 +1,174 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""
|
| 2 |
+
Generate — the "G" in RAG. Turn retrieved calendar excerpts + a question into a grounded answer.
|
| 3 |
+
|
| 4 |
+
The whole idea of RAG: instead of asking the LLM to answer from memory (where it may
|
| 5 |
+
hallucinate), we hand it real excerpts from the UBC Academic Calendar as context and
|
| 6 |
+
instruct it to answer *from that context*. The model becomes a reader/synthesizer over
|
| 7 |
+
retrieved facts.
|
| 8 |
+
|
| 9 |
+
`generate_answer` returns a string and never prints. That keeps it usable both from the
|
| 10 |
+
CLI (which prints the return value) and, later, from the Phase 3 eval loop (which runs
|
| 11 |
+
100 questions and must not spew to stdout).
|
| 12 |
+
|
| 13 |
+
Two backends produce the answer, selected by `config.LLM_BACKEND`:
|
| 14 |
+
- "anthropic": Claude via the Anthropic API (the default, so the app runs without a
|
| 15 |
+
local model server).
|
| 16 |
+
- "local": any OpenAI-compatible local server (LM Studio, llama.cpp, vLLM, Ollama's
|
| 17 |
+
OpenAI shim) via `/v1/chat/completions`.
|
| 18 |
+
|
| 19 |
+
Both paths take the same (system prompt, user turn) and return a plain string, so the
|
| 20 |
+
callers and the eval loop don't care which backend is active.
|
| 21 |
+
"""
|
| 22 |
+
|
| 23 |
+
from __future__ import annotations
|
| 24 |
+
|
| 25 |
+
from . import config, trace
|
| 26 |
+
from .retrieve import Result
|
| 27 |
+
|
| 28 |
+
# System prompt: defines the assistant's role and the grounding rule.
|
| 29 |
+
# The user only sees their question and the final answer — never the retrieved context —
|
| 30 |
+
# so the answer must read as standalone advice and must not point back at the numbered
|
| 31 |
+
# excerpt blocks. Two calendar-specific twists: (1) the calendar is authoritative and
|
| 32 |
+
# changes yearly, so the answer should hand the user the official page URL(s) carried on
|
| 33 |
+
# each excerpt; (2) the corpus deliberately contains colliding editions (2026/27 live vs
|
| 34 |
+
# 2025/26 archive) and cohort-split requirement pages, so the model must disambiguate.
|
| 35 |
+
SYSTEM_PROMPT = (
|
| 36 |
+
"You are a UBC academic advisor. Answer the user's question using ONLY the excerpts "
|
| 37 |
+
"from the UBC Vancouver Academic Calendar provided as context, but write the answer "
|
| 38 |
+
"as if you simply know it. Do NOT mention or number the excerpts: the user cannot "
|
| 39 |
+
'see them, so phrases like "Excerpt 3" are meaningless to them. You SHOULD, however, '
|
| 40 |
+
"point the user to the official calendar page URL(s) shown with the excerpts for the "
|
| 41 |
+
"authoritative and most current wording. Excerpts may come from different calendar "
|
| 42 |
+
"editions (e.g. 2026/27 vs 2025/26) or different student cohorts — answer for the "
|
| 43 |
+
"edition and cohort the question asks about, defaulting to the current 2026/27 "
|
| 44 |
+
"calendar, and note when requirements differ between them. Keep the answer concise "
|
| 45 |
+
"and focused directly on what was asked. If the excerpts do not contain enough "
|
| 46 |
+
"information to answer, say so plainly rather than inventing requirements."
|
| 47 |
+
)
|
| 48 |
+
|
| 49 |
+
# Closed-book prompt: the "no-RAG" baseline for the ablation. The model answers from its
|
| 50 |
+
# own parametric memory with no retrieved context. We push it to commit to concrete
|
| 51 |
+
# specifics (rather than refuse) so its answer makes measurable claims to score for
|
| 52 |
+
# hallucination against the gold calendar excerpt.
|
| 53 |
+
CLOSED_BOOK_SYSTEM = (
|
| 54 |
+
"You are a UBC academic advisor. Answer the user's question about UBC Vancouver "
|
| 55 |
+
"courses, programs, and policies from your own knowledge, with concrete specifics "
|
| 56 |
+
"(course codes, credit counts, requirements), kept concise. Give your single best "
|
| 57 |
+
"answer; do not refuse or say you lack access to the official calendar."
|
| 58 |
+
)
|
| 59 |
+
|
| 60 |
+
# Both clients are created lazily and cached, so importing this module is cheap and only
|
| 61 |
+
# the backend actually in use gets constructed.
|
| 62 |
+
_openai_client = None
|
| 63 |
+
_anthropic_client = None
|
| 64 |
+
|
| 65 |
+
|
| 66 |
+
def _get_openai_client():
|
| 67 |
+
"""The OpenAI client pointed at the local server (LLM_BACKEND == 'local')."""
|
| 68 |
+
global _openai_client
|
| 69 |
+
if _openai_client is None:
|
| 70 |
+
from openai import OpenAI
|
| 71 |
+
|
| 72 |
+
_openai_client = OpenAI(
|
| 73 |
+
base_url=config.OPENAI_BASE_URL,
|
| 74 |
+
api_key=config.OPENAI_API_KEY,
|
| 75 |
+
)
|
| 76 |
+
return _openai_client
|
| 77 |
+
|
| 78 |
+
|
| 79 |
+
def _get_anthropic_client():
|
| 80 |
+
"""The Anthropic client (LLM_BACKEND == 'anthropic'); reads ANTHROPIC_API_KEY."""
|
| 81 |
+
global _anthropic_client
|
| 82 |
+
if _anthropic_client is None:
|
| 83 |
+
import anthropic
|
| 84 |
+
|
| 85 |
+
_anthropic_client = anthropic.Anthropic()
|
| 86 |
+
return _anthropic_client
|
| 87 |
+
|
| 88 |
+
|
| 89 |
+
def format_context(results: list[Result]) -> str:
|
| 90 |
+
"""Render retrieved calendar excerpts into a readable, numbered context block for the
|
| 91 |
+
prompt. Each chunk's `text` already leads with its page title + section breadcrumb;
|
| 92 |
+
the header line and Source URL make the excerpt scannable and citable."""
|
| 93 |
+
blocks = []
|
| 94 |
+
for n, r in enumerate(results, start=1):
|
| 95 |
+
chunk = r.recipe # engine-wide name for "the document record"
|
| 96 |
+
blocks.append(
|
| 97 |
+
f"[Excerpt {n}] {chunk['title']}\n"
|
| 98 |
+
f"Source: {chunk['url']}\n"
|
| 99 |
+
f"{chunk['text']}"
|
| 100 |
+
)
|
| 101 |
+
return "\n\n".join(blocks)
|
| 102 |
+
|
| 103 |
+
|
| 104 |
+
def build_user_turn(query: str, results: list[Result]) -> str:
|
| 105 |
+
"""Render the grounded user turn: the retrieved excerpts followed by the question."""
|
| 106 |
+
context = format_context(results)
|
| 107 |
+
return (
|
| 108 |
+
f"Here are some excerpts from the UBC Vancouver Academic Calendar that may be "
|
| 109 |
+
f"relevant:\n\n{context}\n\n"
|
| 110 |
+
f"Question: {query}"
|
| 111 |
+
)
|
| 112 |
+
|
| 113 |
+
|
| 114 |
+
def complete(system: str, user: str) -> str:
|
| 115 |
+
"""One-shot completion via the configured backend, for callers that build their own
|
| 116 |
+
prompt (the Phase-4 router and agent) rather than the grounded calendar turn."""
|
| 117 |
+
return _complete(system, user)
|
| 118 |
+
|
| 119 |
+
|
| 120 |
+
def _complete(system: str, user: str) -> str:
|
| 121 |
+
"""Dispatch one (system, user) turn to the configured backend and return the text.
|
| 122 |
+
|
| 123 |
+
Every LLM call in the project funnels through here, so this is the one place that traces
|
| 124 |
+
the exact prompt sent to the API (when tracing is on; a no-op otherwise).
|
| 125 |
+
"""
|
| 126 |
+
model = config.ANTHROPIC_MODEL if config.LLM_BACKEND == "anthropic" else config.CHAT_MODEL
|
| 127 |
+
trace.prompt(system, user, model)
|
| 128 |
+
if config.LLM_BACKEND == "anthropic":
|
| 129 |
+
text = _complete_anthropic(system, user)
|
| 130 |
+
else:
|
| 131 |
+
text = _complete_local(system, user)
|
| 132 |
+
trace.response(text)
|
| 133 |
+
return text
|
| 134 |
+
|
| 135 |
+
|
| 136 |
+
def _complete_local(system: str, user: str) -> str:
|
| 137 |
+
"""OpenAI-compatible local server via /v1/chat/completions."""
|
| 138 |
+
client = _get_openai_client()
|
| 139 |
+
response = client.chat.completions.create(
|
| 140 |
+
model=config.CHAT_MODEL,
|
| 141 |
+
messages=[
|
| 142 |
+
{"role": "system", "content": system},
|
| 143 |
+
{"role": "user", "content": user},
|
| 144 |
+
],
|
| 145 |
+
temperature=config.TEMPERATURE,
|
| 146 |
+
max_tokens=config.MAX_TOKENS,
|
| 147 |
+
)
|
| 148 |
+
return (response.choices[0].message.content or "").strip()
|
| 149 |
+
|
| 150 |
+
|
| 151 |
+
def _complete_anthropic(system: str, user: str) -> str:
|
| 152 |
+
"""Claude via the Anthropic API. `system` is a top-level arg, not a message.
|
| 153 |
+
|
| 154 |
+
Note: current Claude models don't take sampling params (no `temperature`/`top_p`),
|
| 155 |
+
so grounding is steered by SYSTEM_PROMPT rather than by config.TEMPERATURE.
|
| 156 |
+
"""
|
| 157 |
+
client = _get_anthropic_client()
|
| 158 |
+
response = client.messages.create(
|
| 159 |
+
model=config.ANTHROPIC_MODEL,
|
| 160 |
+
max_tokens=config.MAX_TOKENS,
|
| 161 |
+
system=system,
|
| 162 |
+
messages=[{"role": "user", "content": user}],
|
| 163 |
+
)
|
| 164 |
+
return "".join(b.text for b in response.content if b.type == "text").strip()
|
| 165 |
+
|
| 166 |
+
|
| 167 |
+
def generate_answer(query: str, results: list[Result]) -> str:
|
| 168 |
+
"""Answer the question grounded in the retrieved calendar excerpts."""
|
| 169 |
+
return _complete(SYSTEM_PROMPT, build_user_turn(query, results))
|
| 170 |
+
|
| 171 |
+
|
| 172 |
+
def generate_answer_closed_book(query: str) -> str:
|
| 173 |
+
"""Answer with NO retrieved context — the no-RAG baseline for the ablation harness."""
|
| 174 |
+
return _complete(CLOSED_BOOK_SYSTEM, query)
|
src/pipeline.py
ADDED
|
@@ -0,0 +1,42 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""
|
| 2 |
+
Pipeline — the single adaptive entry point (Phase 4).
|
| 3 |
+
|
| 4 |
+
`answer()` is the one function the CLI, the API, and the eval all call. It routes each
|
| 5 |
+
question, then runs the matching path and returns a uniform result:
|
| 6 |
+
|
| 7 |
+
router.classify(q) -> "simple" retrieve + generate a grounded answer
|
| 8 |
+
"complex" hand off to the agent (decompose -> multi-hop
|
| 9 |
+
retrieve -> merge -> synthesize)
|
| 10 |
+
|
| 11 |
+
Returning {answer, results, route} keeps callers backend-agnostic: the CLI prints it, the
|
| 12 |
+
API serializes it, and the eval scores `results` (retrieval) and `answer` (generation).
|
| 13 |
+
"""
|
| 14 |
+
|
| 15 |
+
from __future__ import annotations
|
| 16 |
+
|
| 17 |
+
from . import agent, generate, retrieve, router, trace
|
| 18 |
+
from .retrieve import Result
|
| 19 |
+
|
| 20 |
+
|
| 21 |
+
def answer(
|
| 22 |
+
question: str,
|
| 23 |
+
top_k: int | None = None,
|
| 24 |
+
mode: str | None = None,
|
| 25 |
+
route: str | None = None,
|
| 26 |
+
) -> dict:
|
| 27 |
+
"""Route `question` and answer it. Pass `route` to force a path (skips classification)."""
|
| 28 |
+
trace.step(f"PIPELINE - answer: {question!r}")
|
| 29 |
+
route = route or router.classify(question)
|
| 30 |
+
if route == "complex":
|
| 31 |
+
results, ans = agent.run(question, top_k=top_k)
|
| 32 |
+
else:
|
| 33 |
+
trace.step("RETRIEVE - simple route (single query)")
|
| 34 |
+
results = retrieve.retrieve(question, top_k=top_k, mode=mode)
|
| 35 |
+
trace.step("GENERATE - grounded answer over the retrieved excerpts")
|
| 36 |
+
ans = generate.generate_answer(question, results)
|
| 37 |
+
trace.step(f"PIPELINE - done (route={route}, {len(results)} excerpts)")
|
| 38 |
+
return {"answer": ans, "results": results, "route": route}
|
| 39 |
+
|
| 40 |
+
|
| 41 |
+
# Re-exported so callers can annotate/inspect results without importing retrieve directly.
|
| 42 |
+
__all__ = ["answer", "Result"]
|
src/rerank.py
ADDED
|
@@ -0,0 +1,68 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""
|
| 2 |
+
Rerank — the precision stage of the retrieval funnel.
|
| 3 |
+
|
| 4 |
+
The bi-encoder (embeddings) scores query and document *separately*, so it can only
|
| 5 |
+
compare their summaries. A cross-encoder reads the query and a document *together* and
|
| 6 |
+
scores the pair jointly — far sharper, but far too slow to run against the whole corpus.
|
| 7 |
+
So it only ever sees the short candidate list that the cheap recall stage (dense + BM25)
|
| 8 |
+
already produced.
|
| 9 |
+
|
| 10 |
+
Reranking runs on Cohere's hosted rerank API (config.RERANKER_MODEL). Keeping it hosted
|
| 11 |
+
means no local torch / sentence-transformers dependency, so the service deploys on an
|
| 12 |
+
ordinary CPU host. Excerpts longer than the model's window are truncated server-side,
|
| 13 |
+
which is fine: the title/section header + the start of the body carry the ranking signal.
|
| 14 |
+
"""
|
| 15 |
+
|
| 16 |
+
from __future__ import annotations
|
| 17 |
+
|
| 18 |
+
import time
|
| 19 |
+
|
| 20 |
+
from . import config
|
| 21 |
+
from .retrieve import Result
|
| 22 |
+
|
| 23 |
+
# The Cohere client is created lazily and cached: only the hybrid_rerank mode needs it,
|
| 24 |
+
# so dense/hybrid modes never construct it.
|
| 25 |
+
_client = None
|
| 26 |
+
|
| 27 |
+
|
| 28 |
+
def _get_client():
|
| 29 |
+
global _client
|
| 30 |
+
if _client is None:
|
| 31 |
+
import cohere
|
| 32 |
+
|
| 33 |
+
_client = cohere.Client(config.COHERE_API_KEY)
|
| 34 |
+
return _client
|
| 35 |
+
|
| 36 |
+
|
| 37 |
+
def rerank(query: str, candidates: list[Result], top_k: int) -> list[Result]:
|
| 38 |
+
"""Re-score `candidates` jointly against `query`; return the top_k, best first.
|
| 39 |
+
|
| 40 |
+
The returned Results carry the Cohere relevance score (0-1, higher is better),
|
| 41 |
+
replacing whatever score the recall stage assigned.
|
| 42 |
+
"""
|
| 43 |
+
import cohere
|
| 44 |
+
|
| 45 |
+
client = _get_client()
|
| 46 |
+
# Trial keys are capped at 10 calls/minute; back off and retry on 429 so a batch
|
| 47 |
+
# (e.g. the eval loop) doesn't die mid-run.
|
| 48 |
+
for attempt in range(4):
|
| 49 |
+
try:
|
| 50 |
+
response = client.rerank(
|
| 51 |
+
model=config.RERANKER_MODEL,
|
| 52 |
+
query=query,
|
| 53 |
+
documents=[r.recipe["text"] for r in candidates],
|
| 54 |
+
top_n=top_k,
|
| 55 |
+
)
|
| 56 |
+
break
|
| 57 |
+
except cohere.errors.TooManyRequestsError:
|
| 58 |
+
if attempt == 3:
|
| 59 |
+
raise
|
| 60 |
+
time.sleep(15 * (attempt + 1))
|
| 61 |
+
return [
|
| 62 |
+
Result(
|
| 63 |
+
id=candidates[result.index].id,
|
| 64 |
+
score=result.relevance_score,
|
| 65 |
+
recipe=candidates[result.index].recipe,
|
| 66 |
+
)
|
| 67 |
+
for result in response.results
|
| 68 |
+
]
|
src/retrieve.py
ADDED
|
@@ -0,0 +1,162 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""
|
| 2 |
+
Retrieve — given a question, return the most relevant calendar excerpts.
|
| 3 |
+
|
| 4 |
+
This is the "R" in RAG. Phase 2 turns it into a two-stage funnel:
|
| 5 |
+
|
| 6 |
+
stage 1 — RECALL dense (vector) and sparse (BM25) search each nominate ~20
|
| 7 |
+
candidates; Reciprocal Rank Fusion merges the two rankings.
|
| 8 |
+
stage 2 — PRECISION a cross-encoder re-scores the fused shortlist jointly with
|
| 9 |
+
the query; the final top-k come out the top.
|
| 10 |
+
|
| 11 |
+
Which stages run is a config knob (UBCAL_RETRIEVAL_MODE), so the eval can compare:
|
| 12 |
+
|
| 13 |
+
dense Phase-1 pure vector search (the baseline)
|
| 14 |
+
hybrid stage 1 only (dense + BM25, RRF-fused)
|
| 15 |
+
hybrid_rerank both stages (the Phase-2 default)
|
| 16 |
+
|
| 17 |
+
Why RRF for fusion? Cosine similarities and BM25 scores live on totally different
|
| 18 |
+
scales, so averaging them is meaningless. RRF ignores the scores entirely and combines
|
| 19 |
+
*ranks* — a document's fused score is sum(1 / (RRF_K + rank)) across the lists it
|
| 20 |
+
appears in — sidestepping the calibration problem.
|
| 21 |
+
|
| 22 |
+
Exposing the doc `id` on every result is deliberate: the Phase 3 evaluation harness
|
| 23 |
+
matches retrieved ids against "gold" chunk ids to compute hit@k / recall@k.
|
| 24 |
+
"""
|
| 25 |
+
|
| 26 |
+
from __future__ import annotations
|
| 27 |
+
|
| 28 |
+
import json
|
| 29 |
+
import sys
|
| 30 |
+
from dataclasses import dataclass
|
| 31 |
+
|
| 32 |
+
import numpy as np
|
| 33 |
+
|
| 34 |
+
from . import bm25, config, embed, trace
|
| 35 |
+
|
| 36 |
+
|
| 37 |
+
@dataclass
|
| 38 |
+
class Result:
|
| 39 |
+
"""One retrieved excerpt (chunk record) plus its retrieval score.
|
| 40 |
+
|
| 41 |
+
What `score` means depends on the mode that produced it: cosine similarity (dense),
|
| 42 |
+
RRF fused score (hybrid), or a cross-encoder logit (hybrid_rerank). Higher is always
|
| 43 |
+
better within one result list, but values are not comparable across modes.
|
| 44 |
+
"""
|
| 45 |
+
|
| 46 |
+
id: int
|
| 47 |
+
score: float
|
| 48 |
+
recipe: dict # the full metadata record from the index
|
| 49 |
+
|
| 50 |
+
|
| 51 |
+
# The index is loaded once and cached for the process lifetime.
|
| 52 |
+
_embeddings: np.ndarray | None = None
|
| 53 |
+
_metadata: list[dict] | None = None
|
| 54 |
+
_texts: list[str] | None = None
|
| 55 |
+
|
| 56 |
+
|
| 57 |
+
def _load_index() -> tuple[np.ndarray, list[dict]]:
|
| 58 |
+
global _embeddings, _metadata
|
| 59 |
+
if _embeddings is None or _metadata is None:
|
| 60 |
+
if not config.EMBEDDINGS_PATH.exists() or not config.METADATA_PATH.exists():
|
| 61 |
+
sys.exit(
|
| 62 |
+
"No index found. Build it first with: python -m src.ingest"
|
| 63 |
+
)
|
| 64 |
+
_embeddings = np.load(config.EMBEDDINGS_PATH)
|
| 65 |
+
with open(config.METADATA_PATH, "r", encoding="utf-8") as f:
|
| 66 |
+
_metadata = json.load(f)
|
| 67 |
+
return _embeddings, _metadata
|
| 68 |
+
|
| 69 |
+
|
| 70 |
+
def _corpus_texts() -> list[str]:
|
| 71 |
+
"""The raw document texts, id-aligned with the index (what BM25 searches over)."""
|
| 72 |
+
global _texts
|
| 73 |
+
if _texts is None:
|
| 74 |
+
_, metadata = _load_index()
|
| 75 |
+
_texts = [r["text"] for r in metadata]
|
| 76 |
+
return _texts
|
| 77 |
+
|
| 78 |
+
|
| 79 |
+
def _dense_search(query: str, k: int) -> list[tuple[int, float]]:
|
| 80 |
+
"""Embed the query and rank every chunk by cosine similarity; top-k, best first."""
|
| 81 |
+
embeddings, metadata = _load_index()
|
| 82 |
+
|
| 83 |
+
# Embed the query (shape (1, dim)) and reduce to a 1-D vector.
|
| 84 |
+
query_vec = embed.embed_texts([query])[0]
|
| 85 |
+
|
| 86 |
+
# Cosine similarity == dot product, since every vector is unit-length.
|
| 87 |
+
# scores[i] is the similarity between the query and chunk i.
|
| 88 |
+
scores = embeddings @ query_vec
|
| 89 |
+
|
| 90 |
+
# Indices of the top_k highest scores. argpartition finds them without fully
|
| 91 |
+
# sorting the whole array; we then sort just those k by score, descending.
|
| 92 |
+
k = min(k, len(metadata))
|
| 93 |
+
top_idx = np.argpartition(scores, -k)[-k:]
|
| 94 |
+
top_idx = top_idx[np.argsort(scores[top_idx])[::-1]]
|
| 95 |
+
return [(int(i), float(scores[i])) for i in top_idx]
|
| 96 |
+
|
| 97 |
+
|
| 98 |
+
def _rrf_fuse(rankings: list[list[int]]) -> list[tuple[int, float]]:
|
| 99 |
+
"""Merge ranked doc-id lists with Reciprocal Rank Fusion; best fused score first."""
|
| 100 |
+
fused: dict[int, float] = {}
|
| 101 |
+
for ranking in rankings:
|
| 102 |
+
for rank, doc_id in enumerate(ranking, start=1):
|
| 103 |
+
fused[doc_id] = fused.get(doc_id, 0.0) + 1.0 / (config.RRF_K + rank)
|
| 104 |
+
return sorted(fused.items(), key=lambda pair: pair[1], reverse=True)
|
| 105 |
+
|
| 106 |
+
|
| 107 |
+
def _as_results(ranked: list[tuple[int, float]], limit: int) -> list[Result]:
|
| 108 |
+
_, metadata = _load_index()
|
| 109 |
+
return [Result(id=i, score=s, recipe=metadata[i]) for i, s in ranked[:limit]]
|
| 110 |
+
|
| 111 |
+
|
| 112 |
+
def retrieve(query: str, top_k: int | None = None, mode: str | None = None) -> list[Result]:
|
| 113 |
+
"""Return the top_k excerpts most relevant to `query`, best first.
|
| 114 |
+
|
| 115 |
+
`mode` defaults to config.RETRIEVAL_MODE (see the module docstring for the three
|
| 116 |
+
modes). The signature and Result shape are stable across modes — the eval harness
|
| 117 |
+
and CLI call this one function regardless of pipeline.
|
| 118 |
+
"""
|
| 119 |
+
top_k = top_k or config.TOP_K
|
| 120 |
+
mode = mode or config.RETRIEVAL_MODE
|
| 121 |
+
if mode not in ("dense", "hybrid", "hybrid_rerank"):
|
| 122 |
+
sys.exit(
|
| 123 |
+
f"Unknown retrieval mode {mode!r} — expected dense, hybrid, or hybrid_rerank."
|
| 124 |
+
)
|
| 125 |
+
trace.detail("retrieve mode", mode)
|
| 126 |
+
trace.detail("embed query (OpenAI API)", repr(query))
|
| 127 |
+
|
| 128 |
+
if mode == "dense":
|
| 129 |
+
out = _as_results(_dense_search(query, top_k), top_k)
|
| 130 |
+
trace.results(out)
|
| 131 |
+
return out
|
| 132 |
+
|
| 133 |
+
# Stage 1 — recall: each retriever nominates candidates, RRF merges the rankings.
|
| 134 |
+
dense_ids = [i for i, _ in _dense_search(query, config.DENSE_K)]
|
| 135 |
+
sparse_ids = [i for i, _ in bm25.search(query, config.SPARSE_K, _corpus_texts())]
|
| 136 |
+
fused = _rrf_fuse([dense_ids, sparse_ids])
|
| 137 |
+
trace.detail("stage 1 recall", f"dense={len(dense_ids)} + bm25={len(sparse_ids)} "
|
| 138 |
+
f"-> RRF-fused to {len(fused)} unique")
|
| 139 |
+
|
| 140 |
+
if mode == "hybrid":
|
| 141 |
+
out = _as_results(fused, top_k)
|
| 142 |
+
trace.results(out)
|
| 143 |
+
return out
|
| 144 |
+
|
| 145 |
+
# Stage 2 — precision: the cross-encoder re-scores the fused shortlist.
|
| 146 |
+
# Imported here so dense/hybrid modes never pay the multi-second torch import.
|
| 147 |
+
from . import rerank
|
| 148 |
+
|
| 149 |
+
shortlist = _as_results(fused, config.RERANK_CANDIDATES)
|
| 150 |
+
trace.detail("stage 2 precision", f"rerank {len(shortlist)} candidates (Cohere API) "
|
| 151 |
+
f"-> top {top_k}")
|
| 152 |
+
out = rerank.rerank(query, shortlist, top_k)
|
| 153 |
+
trace.results(out)
|
| 154 |
+
return out
|
| 155 |
+
|
| 156 |
+
|
| 157 |
+
if __name__ == "__main__":
|
| 158 |
+
# Quick manual check: python -m src.retrieve "prerequisites for CPSC 210"
|
| 159 |
+
# (set UBCAL_RETRIEVAL_MODE to compare modes)
|
| 160 |
+
question = " ".join(sys.argv[1:]) or "prerequisites for CPSC 210"
|
| 161 |
+
for r in retrieve(question):
|
| 162 |
+
print(f"[{r.score:.3f}] (id={r.id}) {r.recipe['title']}")
|
src/router.py
ADDED
|
@@ -0,0 +1,38 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""
|
| 2 |
+
Router — the "adaptive" in adaptive RAG (Phase 4).
|
| 3 |
+
|
| 4 |
+
One small function decides how much machinery a question deserves:
|
| 5 |
+
|
| 6 |
+
simple a single fact or page lookup -> the fast Phase-2 retriever
|
| 7 |
+
complex needs several pages, a -> the agentic loop (decompose ->
|
| 8 |
+
comparison, or multi-step reasoning multi-hop retrieve -> synthesize)
|
| 9 |
+
|
| 10 |
+
The classification is a cheap zero-shot LLM call reusing the generation backend, so there
|
| 11 |
+
is no extra model to configure. Anything the model returns other than a clear "complex"
|
| 12 |
+
falls back to "simple" — the cheaper path is the safe default, so a misparse never sends a
|
| 13 |
+
simple question down the expensive agent.
|
| 14 |
+
"""
|
| 15 |
+
|
| 16 |
+
from __future__ import annotations
|
| 17 |
+
|
| 18 |
+
from . import generate, trace
|
| 19 |
+
|
| 20 |
+
CLASSIFY_SYSTEM = (
|
| 21 |
+
"You are a query router for a UBC academic-calendar Q&A system. Classify the user's "
|
| 22 |
+
"question as either 'simple' or 'complex'.\n"
|
| 23 |
+
" simple — answerable from one calendar page (a single course's prerequisites or "
|
| 24 |
+
"description, one program's requirements, one policy or date).\n"
|
| 25 |
+
" complex — needs multiple courses or programs, a comparison (between programs, "
|
| 26 |
+
"student cohorts, or calendar editions), or a prerequisite/requirement chain "
|
| 27 |
+
"(e.g. 'compare X and Y', or 'what do I need to take before Z').\n"
|
| 28 |
+
"Reply with exactly one word: simple or complex."
|
| 29 |
+
)
|
| 30 |
+
|
| 31 |
+
|
| 32 |
+
def classify(question: str) -> str:
|
| 33 |
+
"""Return "simple" or "complex" for `question`. Defaults to "simple" on any surprise."""
|
| 34 |
+
trace.step("ROUTE - classify (simple vs complex)")
|
| 35 |
+
reply = generate.complete(CLASSIFY_SYSTEM, question).strip().lower()
|
| 36 |
+
route = "complex" if "complex" in reply else "simple"
|
| 37 |
+
trace.detail("decision", route)
|
| 38 |
+
return route
|
src/trace.py
ADDED
|
@@ -0,0 +1,73 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""
|
| 2 |
+
Trace — optional, human-readable logging of every pipeline step and API prompt.
|
| 3 |
+
|
| 4 |
+
Off by default: `enable()` flips a module-global, so normal CLI runs and the eval loop stay
|
| 5 |
+
silent and behave exactly as before. When on (via `ask.py --trace`), each stage prints what
|
| 6 |
+
it is doing and the EXACT (system, user) prompt sent to the LLM, so the whole
|
| 7 |
+
route -> retrieve -> generate loop is visible for learning and debugging.
|
| 8 |
+
|
| 9 |
+
Output is plain ASCII to stdout so it renders in any terminal and interleaves in order with
|
| 10 |
+
the CLI's own prints.
|
| 11 |
+
"""
|
| 12 |
+
|
| 13 |
+
from __future__ import annotations
|
| 14 |
+
|
| 15 |
+
import sys
|
| 16 |
+
|
| 17 |
+
_enabled = False
|
| 18 |
+
|
| 19 |
+
|
| 20 |
+
def enable() -> None:
|
| 21 |
+
"""Turn tracing on for the rest of this process."""
|
| 22 |
+
global _enabled
|
| 23 |
+
_enabled = True
|
| 24 |
+
# Traces print full excerpt context and model replies, which often contain non-ASCII
|
| 25 |
+
# (degree signs, fractions, accents). Switch stdout to UTF-8 so a cp1252 Windows console
|
| 26 |
+
# doesn't raise UnicodeEncodeError; errors="replace" is a last-resort guard.
|
| 27 |
+
try:
|
| 28 |
+
sys.stdout.reconfigure(encoding="utf-8", errors="replace")
|
| 29 |
+
except Exception:
|
| 30 |
+
pass
|
| 31 |
+
|
| 32 |
+
|
| 33 |
+
def enabled() -> bool:
|
| 34 |
+
return _enabled
|
| 35 |
+
|
| 36 |
+
|
| 37 |
+
def step(title: str) -> None:
|
| 38 |
+
"""A top-level stage banner."""
|
| 39 |
+
if _enabled:
|
| 40 |
+
print(f"\n=== {title} ===")
|
| 41 |
+
|
| 42 |
+
|
| 43 |
+
def detail(label: str, value: object = "") -> None:
|
| 44 |
+
"""One indented `label: value` line (or just a label if value is empty)."""
|
| 45 |
+
if _enabled:
|
| 46 |
+
print(f" {label}: {value}" if value != "" else f" {label}")
|
| 47 |
+
|
| 48 |
+
|
| 49 |
+
def results(items) -> None:
|
| 50 |
+
"""Print a list of retrieve.Result rows as [score] (id) title."""
|
| 51 |
+
if _enabled:
|
| 52 |
+
for r in items:
|
| 53 |
+
print(f" [{r.score:.3f}] (id={r.id}) {r.recipe['title']}")
|
| 54 |
+
|
| 55 |
+
|
| 56 |
+
def prompt(system: str, user: str, model: str) -> None:
|
| 57 |
+
"""Show one LLM API call: the system prompt and the user turn sent to `model`."""
|
| 58 |
+
if _enabled:
|
| 59 |
+
print(f" --- API call -> {model} ---")
|
| 60 |
+
_block("SYSTEM", system)
|
| 61 |
+
_block("USER", user)
|
| 62 |
+
|
| 63 |
+
|
| 64 |
+
def response(text: str) -> None:
|
| 65 |
+
"""Show the model's reply for the call printed by `prompt`."""
|
| 66 |
+
if _enabled:
|
| 67 |
+
_block("RESPONSE", text)
|
| 68 |
+
|
| 69 |
+
|
| 70 |
+
def _block(label: str, text: str) -> None:
|
| 71 |
+
print(f" [{label}]")
|
| 72 |
+
for line in (text.splitlines() or [""]):
|
| 73 |
+
print(f" {line}")
|