Devdan Schretlen commited on
Commit
494a4bf
·
1 Parent(s): 46de6a9

Added RAGChef RAG engine and eval harness

Browse files
README.md ADDED
@@ -0,0 +1,175 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # UBCHelper — RAG over the UBC Vancouver Academic Calendar
2
+
3
+ A retrieval-augmented **academic-calendar advisor**. Ask a question about UBC Vancouver
4
+ courses, programs, or policies → the system retrieves the relevant calendar excerpts from a
5
+ local index → an LLM writes a grounded answer that points you at the official calendar page.
6
+ Embeddings run on the **OpenAI API**, reranking on the **Cohere API**, and generation on
7
+ **Claude (Anthropic API)** by default — with an optional local generation backend via LM Studio.
8
+
9
+ A domain re-skin of [RAGChef](../RAGChef): the same from-scratch engine (no
10
+ LangChain/LlamaIndex) — hybrid dense + BM25 retrieval, RRF fusion, hosted rerank, an adaptive
11
+ router with a decompose→multi-hop→synthesize agent, and a two-tier eval harness — pointed at a
12
+ scraped calendar corpus instead of a recipe CSV.
13
+
14
+ ```
15
+ ┌────────────────────────── run once ──────────────────────────────┐
16
+ scrape.py ──► data/pages/ ──► chunk.py ──► ingest.py ──► index/ (numpy + json)
17
+ └──────────────────────────────────────────────────────────────────┘
18
+
19
+ "What are the prerequisites for CPSC 210?"
20
+ │
21
+ ▼
22
+ retrieve.py ──┬─► dense — embed the query (OpenAI API), cosine search ─► ~20 ──┐
23
+ └─► sparse — BM25 keyword search (src/bm25.py) ─► ~20 ──┤
24
+ ▼
25
+ RRF fusion ──► rerank (Cohere API, src/rerank.py) ──► top-k
26
+ │
27
+ ▼
28
+ generate.py ──► grounded prompt ──► Claude (Anthropic API) ──► answer + source URL
29
+ ```
30
+
31
+ Complex questions can take the adaptive path instead (`ask.py --adaptive`): a router
32
+ classifies the question, and **complex** ones go through the agent — decompose into
33
+ sub-questions → retrieve each → merge → synthesize one answer.
34
+
35
+ ---
36
+
37
+ ## The corpus (and why it's deliberately tricky)
38
+
39
+ `src/scrape.py` crawls the calendar into `data/pages/` — one JSON record per page, with
40
+ provenance (canonical URL, edition, fetch time, content hash). Two editions are collected:
41
+
42
+ - **live 2026/27** — `vancouver.calendar.ubc.ca` (sitemap-driven): the Faculty of Science
43
+ subtree, campus-wide policies, all cohort-split programs, and course-description pages.
44
+ - **archive 2025/26** — `archive.calendar.ubc.ca/vancouver/2526/...`: a narrow slice of the
45
+ same subtrees where the editions collide.
46
+
47
+ That gives the corpus two built-in traps that naive top-k RAG falls into: **edition
48
+ collisions** (near-identical pages differing only in calendar year) and **cohort splits**
49
+ (e.g. B.A. requirements for students entering 2023/24-or-earlier vs 2024/25-or-later).
50
+ `src/chunk.py` folds the disambiguators into every chunk's title and text — e.g.
51
+ `Bachelor of Arts — Degree Requirements ... (2024/25 or later) [2026/27]` — and carries them
52
+ as structured metadata, so retrieval, prompts, and the eval can all tell the twins apart.
53
+
54
+ **Scraping etiquette / Terms of Use:** the crawl honors robots.txt (10 s crawl delay), sends
55
+ a descriptive User-Agent, caches every page (re-runs use conditional GETs), and the scraped
56
+ content is **never committed or redistributed** — `data/pages/` is gitignored; only the
57
+ scraper and engine code live in the repo. Rebuild the corpus locally with:
58
+
59
+ ```powershell
60
+ python -m src.scrape --dry-run # enumerate in-scope URLs, fetch nothing
61
+ python -m src.scrape --sample # just the 5 representative sample pages (quick start)
62
+ python -m src.scrape # full in-scope crawl (slow by design: 10 s/request)
63
+ ```
64
+
65
+ ---
66
+
67
+ ## Setup
68
+
69
+ ```powershell
70
+ py -m venv .venv
71
+ .\.venv\Scripts\Activate.ps1
72
+ pip install -r requirements.txt
73
+ cp .env.example .env # then edit .env
74
+ ```
75
+
76
+ - `OPENAI_API_KEY` — **required**; embeddings run on OpenAI (`text-embedding-3-small`).
77
+ - `COHERE_API_KEY` — **required** for the default `hybrid_rerank` mode (`rerank-v3.5`).
78
+ Not needed with `UBCAL_RETRIEVAL_MODE=dense` or `hybrid`.
79
+ - `ANTHROPIC_API_KEY` — **required** for the default generation backend (Claude Haiku 4.5).
80
+
81
+ > The embedding model used to **build** the index and the one used to **query** it must be
82
+ > the same, or retrieval is meaningless. Re-run ingest if you change `UBCAL_EMBED_MODEL`.
83
+
84
+ Optional local generation (embeddings/rerank stay hosted): start any OpenAI-compatible
85
+ server and set `UBCAL_LLM_BACKEND=local`, `UBCAL_BASE_URL_LLM`, `UBCAL_CHAT_MODEL`.
86
+
87
+ ## Run it
88
+
89
+ ```powershell
90
+ python -m src.ingest # chunk + embed the cached pages -> index/
91
+ python ask.py "What are the prerequisites for CPSC 210?"
92
+ python ask.py --show-context "What is the academic standing policy?"
93
+ python ask.py --adaptive --trace "Compare the B.A. degree requirements for students who entered in 2023/24 vs 2024/25"
94
+ ```
95
+
96
+ Example output:
97
+
98
+ ```
99
+ Retrieved excerpts:
100
+ [0.818] (id=8) Computer Science, Faculty of Science [2026/27]
101
+ [0.808] (id=179) Computer Science, Faculty of Science [2025/26]
102
+ ...
103
+
104
+ Answer:
105
+ The prerequisites for CPSC 210 (Software Construction) are one of: CPSC 107 or CPSC 110.
106
+ For the current 2026/27 calendar see:
107
+ https://vancouver.calendar.ubc.ca/course-descriptions/subject/cpscv
108
+ ```
109
+
110
+ ## The code
111
+
112
+ | File | Role |
113
+ |------|------|
114
+ | [src/config.py](src/config.py) | Every tunable + path (env prefix `UBCAL_`). |
115
+ | [src/scrape.py](src/scrape.py) | Polite sitemap-driven crawler → `data/pages/` + manifest. |
116
+ | [src/chunk.py](src/chunk.py) | Page HTML → markdown-ish chunk records (tables kept whole, cohort/edition metadata parsed). |
117
+ | [src/ingest.py](src/ingest.py) | Cached pages → chunks → embeddings → saved index. |
118
+ | [src/embed.py](src/embed.py) | Shared embedding function (same model for index & query). |
119
+ | [src/retrieve.py](src/retrieve.py) | The funnel: dense + BM25 → RRF fuse → rerank → top-k (with ids). |
120
+ | [src/bm25.py](src/bm25.py) | Hand-rolled BM25 Okapi keyword search (numpy). |
121
+ | [src/rerank.py](src/rerank.py) | Reranker via Cohere's hosted rerank API. |
122
+ | [src/generate.py](src/generate.py) | Excerpts + question → grounded prompt → answer with source URLs. |
123
+ | [src/router.py](src/router.py) | Classifies each question simple vs complex. |
124
+ | [src/agent.py](src/agent.py) | Complex path: decompose → multi-hop retrieve → merge → synthesize. |
125
+ | [src/pipeline.py](src/pipeline.py) | The one adaptive entry point (`answer()`). |
126
+ | [ask.py](ask.py) | The CLI that wires it all together. |
127
+
128
+ Design notes carried over from RAGChef: brute-force cosine search on a numpy matrix instead
129
+ of a vector DB (transparent, instant at this scale); BM25 built in memory from the indexed
130
+ metadata; hosted rerank so there's no local torch. New here: calendar pages are long and
131
+ heterogeneous, so unlike recipes they're **chunked** — split on headings, capped at
132
+ ~`UBCAL_CHUNK_MAX_CHARS`, requirement tables never split. BM25 matters even more in this
133
+ domain: course codes ("CPSC 320", "BUCS") are exactly the rare, high-signal tokens dense
134
+ embeddings under-weight.
135
+
136
+ ```powershell
137
+ $env:UBCAL_RETRIEVAL_MODE = "dense" # baseline (pure vector search)
138
+ $env:UBCAL_RETRIEVAL_MODE = "hybrid" # dense + BM25, RRF-fused
139
+ $env:UBCAL_RETRIEVAL_MODE = "hybrid_rerank" # + hosted rerank (the default)
140
+ ```
141
+
142
+ ## Evaluation
143
+
144
+ The two-tier harness scores the frozen test set through the same retrieve → generate
145
+ pipeline: **Tier 1** deterministic retrieval metrics (`hit@k`, `recall@k`, `MRR` —
146
+ [eval/retrieval_metrics.py](eval/retrieval_metrics.py)) and **Tier 2** RAGAS-style judged
147
+ metrics (faithfulness, answer relevancy, context precision/recall —
148
+ [eval/judge.py](eval/judge.py)); the headline number is hallucination rate = 1 − faithfulness.
149
+ [eval/run_ablation.py](eval/run_ablation.py) additionally answers every question closed-book
150
+ to measure what retrieval is actually worth.
151
+
152
+ ```powershell
153
+ # Generate candidate questions from the indexed corpus, then CURATE BY HAND:
154
+ python -m eval.make_testset --num 20
155
+
156
+ # The before/after retrieval sweep (fast, deterministic):
157
+ $env:UBCAL_RETRIEVAL_MODE="dense"; python -m eval.run_eval --retrieval-only --name baseline-dense
158
+ $env:UBCAL_RETRIEVAL_MODE="hybrid"; python -m eval.run_eval --retrieval-only --name hybrid
159
+ $env:UBCAL_RETRIEVAL_MODE="hybrid_rerank"; python -m eval.run_eval --retrieval-only --name hybrid-rerank
160
+
161
+ # Full judged run + ablation (judge defaults to a local LM Studio model — see UBCAL_JUDGE_MODEL):
162
+ python -m eval.run_eval --name full
163
+ python -m eval.run_ablation --name rag-vs-norag
164
+ ```
165
+
166
+ The test set targets three deliberately hard categories: **code-lookup** (hinges on a course
167
+ or program code — BM25's job), **multi-hop** (prerequisite/requirement chains spanning pages —
168
+ the reranker's and agent's job), and **edition/cohort-collision** (near-duplicate pages
169
+ differing only by calendar year or student cohort — the trap this corpus was built to test).
170
+ Chunk ids are positional, so test sets are only valid against the index they were generated
171
+ from — regenerate after any re-ingest that changes the corpus.
172
+
173
+ > **The calendar is authoritative and changes.** Answers always carry the official calendar
174
+ > URL; verify anything that matters there. This is an educational RAG project, not academic
175
+ > advising.
ask.py ADDED
@@ -0,0 +1,94 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """
2
+ ask.py — the tiny CLI that runs the whole naive-RAG loop for one question.
3
+
4
+ retrieve(question) -> generate_answer(question, excerpts) -> print
5
+
6
+ Usage (from the project root, with the venv active):
7
+ python ask.py "What are the prerequisites for CPSC 210?"
8
+ python ask.py --top-k 6 "How many credits does the B.Sc. require?"
9
+ python ask.py --show-context "What is the academic standing policy?"
10
+
11
+ Prerequisites:
12
+ 1. Keys in `.env` (copy `.env.example`): OPENAI_API_KEY for query embeddings and
13
+ ANTHROPIC_API_KEY for generation (the default backend).
14
+ 2. The index built once: python -m src.ingest
15
+ """
16
+
17
+ from __future__ import annotations
18
+
19
+ import argparse
20
+
21
+ from src import config, generate, pipeline, retrieve, trace
22
+
23
+
24
+ def main() -> None:
25
+ parser = argparse.ArgumentParser(description="Ask the UBC academic-calendar RAG advisor a question.")
26
+ parser.add_argument("question", help="your calendar question (wrap it in quotes)")
27
+ parser.add_argument(
28
+ "--top-k",
29
+ type=int,
30
+ default=config.TOP_K,
31
+ help=f"how many calendar excerpts to retrieve (default: {config.TOP_K})",
32
+ )
33
+ parser.add_argument(
34
+ "--show-context",
35
+ action="store_true",
36
+ help="also print the full text of the calendar excerpts sent to the model",
37
+ )
38
+ parser.add_argument(
39
+ "--mode",
40
+ choices=["dense", "hybrid", "hybrid_rerank"],
41
+ default=config.RETRIEVAL_MODE,
42
+ help=f"retrieval pipeline to use (default: {config.RETRIEVAL_MODE})",
43
+ )
44
+ parser.add_argument(
45
+ "--adaptive",
46
+ action="store_true",
47
+ help="route the question (Phase 4): simple -> fast retriever, complex -> agent",
48
+ )
49
+ parser.add_argument(
50
+ "--trace",
51
+ action="store_true",
52
+ help="print every internal step and the exact prompts sent to the APIs "
53
+ "(most informative with --adaptive)",
54
+ )
55
+ args = parser.parse_args()
56
+
57
+ if args.trace:
58
+ trace.enable()
59
+
60
+ # 1. RETRIEVE (+ optionally route). --adaptive sends the question through the Phase-4
61
+ # pipeline, which classifies it and picks the simple or agentic path; otherwise we run
62
+ # the plain retrieve -> generate loop with the chosen --mode.
63
+ if args.adaptive:
64
+ print("\nThinking ...\n")
65
+ result = pipeline.answer(args.question, top_k=args.top_k, mode=args.mode)
66
+ results, answer = result["results"], result["answer"]
67
+ print(f"Route: {result['route']}")
68
+ else:
69
+ results = retrieve.retrieve(args.question, top_k=args.top_k, mode=args.mode)
70
+ answer = None
71
+
72
+ # Show which excerpts grounded the answer (titles + similarity scores). This makes
73
+ # the retrieval step visible, which is the whole point of the learning exercise.
74
+ print("\nRetrieved excerpts:")
75
+ for r in results:
76
+ print(f" [{r.score:.3f}] (id={r.id}) {r.recipe['title']}")
77
+
78
+ if args.show_context:
79
+ print("\n--- context sent to the model ---")
80
+ print(generate.format_context(results))
81
+ print("--- end context ---")
82
+
83
+ # 2. GENERATE — have the LLM answer, grounded in those excerpts (the adaptive path
84
+ # already produced the answer above).
85
+ if answer is None:
86
+ print("\nThinking ...\n")
87
+ answer = generate.generate_answer(args.question, results)
88
+
89
+ print("\nAnswer:")
90
+ print(answer)
91
+
92
+
93
+ if __name__ == "__main__":
94
+ main()
eval/__init__.py ADDED
@@ -0,0 +1 @@
 
 
1
+ """Evaluation harness (Phase 3) — retrieval metrics now, RAGAS-style judge next."""
eval/judge.py ADDED
@@ -0,0 +1,290 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """
2
+ Tier 2 — RAGAS-style generation metrics, judged by the local LLM.
3
+
4
+ These four metrics follow the RAGAS definitions, but instead of pulling in the heavy
5
+ RAGAS + LangChain dependency tree (which fights hard on new Python interpreters), each
6
+ metric is a small, self-contained prompt to the judge model via the same OpenAI client we
7
+ use everywhere. Fully transparent, works on Python 3.14, no extra dependencies.
8
+
9
+ faithfulness fraction of the ANSWER's claims that are supported by the
10
+ retrieved excerpts. Hallucination rate = 1 - faithfulness.
11
+ answer_relevancy how well the answer actually addresses the question (RAGAS trick:
12
+ generate questions the answer fits, embed them, compare to the real
13
+ question via cosine similarity).
14
+ context_precision of the retrieved excerpts, what fraction were relevant — rank-aware,
15
+ so relevant excerpts ranked higher score better.
16
+ context_recall does the retrieved context contain what's needed to reproduce the
17
+ reference answer (fraction of reference claims it supports).
18
+
19
+ Every score is in [0, 1], higher is better. The judge runs at temperature 0 for
20
+ determinism, and bad/unparseable JSON from the model degrades one metric to None for that
21
+ question rather than crashing the whole run.
22
+
23
+ NOTE: these are *directionally faithful* to canonical RAGAS, not numerically identical —
24
+ ideal for tracking relative change as you tune the system, which is the point.
25
+ """
26
+
27
+ from __future__ import annotations
28
+
29
+ import json
30
+ import re
31
+ import time
32
+
33
+ import numpy as np
34
+ from openai import APIError, OpenAI
35
+
36
+ from src import config, embed
37
+
38
+ # Judge settings. Temperature 0 -> as deterministic as the model allows.
39
+ JUDGE_TEMPERATURE = 0.0
40
+ JUDGE_MAX_TOKENS = 1536 # headroom so long claim arrays don't truncate into invalid JSON
41
+ RELEVANCY_NUM_QUESTIONS = 3 # how many questions to reverse-generate for answer_relevancy
42
+
43
+ _JSON_SYSTEM = "You are a strict evaluator. Respond with ONLY valid JSON — no prose, no markdown, no code fences."
44
+
45
+ # LM Studio constrains generation to a JSON *schema* (OpenAI "structured outputs" style),
46
+ # which guarantees both valid JSON and the exact shape each metric expects — so the model
47
+ # can't emit prose, fences, trailing commas, or the wrong keys. One schema per output shape.
48
+ def _claims_schema(verdict_key: str) -> dict:
49
+ """Schema for {"claims": [{"claim": str, <verdict_key>: bool}, ...]} (faithfulness/recall)."""
50
+ return {
51
+ "type": "object",
52
+ "additionalProperties": False,
53
+ "properties": {
54
+ "claims": {
55
+ "type": "array",
56
+ "items": {
57
+ "type": "object",
58
+ "additionalProperties": False,
59
+ "properties": {
60
+ "claim": {"type": "string"},
61
+ verdict_key: {"type": "boolean"},
62
+ },
63
+ "required": ["claim", verdict_key],
64
+ },
65
+ }
66
+ },
67
+ "required": ["claims"],
68
+ }
69
+
70
+
71
+ _QUESTIONS_SCHEMA = {
72
+ "type": "object",
73
+ "additionalProperties": False,
74
+ "properties": {"questions": {"type": "array", "items": {"type": "string"}}},
75
+ "required": ["questions"],
76
+ }
77
+
78
+ _VERDICTS_SCHEMA = {
79
+ "type": "object",
80
+ "additionalProperties": False,
81
+ "properties": {"verdicts": {"type": "array", "items": {"type": "boolean"}}},
82
+ "required": ["verdicts"],
83
+ }
84
+
85
+ _client: OpenAI | None = None
86
+
87
+
88
+ class JudgeParseError(Exception):
89
+ """Raised when the judge's reply can't be parsed into the expected JSON."""
90
+
91
+
92
+ def _get_client() -> OpenAI:
93
+ global _client
94
+ if _client is None:
95
+ _client = OpenAI(base_url=config.OPENAI_BASE_URL, api_key=config.OPENAI_API_KEY)
96
+ return _client
97
+
98
+
99
+ def _extract_json(text: str):
100
+ """Best-effort JSON extraction from a chat reply.
101
+
102
+ Local models often wrap JSON in ```code fences``` or add a sentence around it, so we
103
+ try a direct parse, then strip fences, then fall back to the outermost {...} or [...].
104
+ """
105
+ s = (text or "").strip()
106
+ if s.startswith("```"):
107
+ s = re.sub(r"^```[a-zA-Z]*\s*", "", s)
108
+ s = re.sub(r"\s*```$", "", s).strip()
109
+ try:
110
+ return json.loads(s)
111
+ except json.JSONDecodeError:
112
+ pass
113
+ for open_ch, close_ch in (("{", "}"), ("[", "]")):
114
+ start, end = s.find(open_ch), s.rfind(close_ch)
115
+ if start != -1 and end > start:
116
+ try:
117
+ return json.loads(s[start : end + 1])
118
+ except json.JSONDecodeError:
119
+ continue
120
+ raise JudgeParseError(text)
121
+
122
+
123
+ def _with_retries(fn, attempts: int = 3, base_delay: float = 1.0):
124
+ """Call fn(), retrying transient LM Studio API errors with linear backoff.
125
+
126
+ LM Studio can JIT-unload an idle model and return a transient 400 "Model unloaded"
127
+ (an APIError); the next call reloads it. Without this, one blip sinks a long run.
128
+ """
129
+ for attempt in range(1, attempts + 1):
130
+ try:
131
+ return fn()
132
+ except APIError:
133
+ if attempt == attempts:
134
+ raise
135
+ time.sleep(base_delay * attempt)
136
+
137
+
138
+ def _chat_json(prompt: str, schema: dict, name: str):
139
+ """Send one judging prompt and return the parsed JSON, shape-constrained by `schema`.
140
+
141
+ response_format=json_schema makes LM Studio grammar-constrain the output to that exact
142
+ schema, so the judge can't emit prose/fences/trailing commas or the wrong keys. We keep
143
+ _extract_json as a defensive fallback.
144
+ """
145
+ client = _get_client()
146
+ response = _with_retries(
147
+ lambda: client.chat.completions.create(
148
+ model=config.JUDGE_MODEL,
149
+ messages=[
150
+ {"role": "system", "content": _JSON_SYSTEM},
151
+ {"role": "user", "content": prompt},
152
+ ],
153
+ temperature=JUDGE_TEMPERATURE,
154
+ max_tokens=JUDGE_MAX_TOKENS,
155
+ response_format={
156
+ "type": "json_schema",
157
+ "json_schema": {"name": name, "strict": True, "schema": schema},
158
+ },
159
+ )
160
+ )
161
+ return _extract_json(response.choices[0].message.content or "")
162
+
163
+
164
+ def _join_contexts(contexts: list[str]) -> str:
165
+ """Number the retrieved excerpts so the judge can refer to them by position."""
166
+ return "\n\n".join(f"[{i}] {c}" for i, c in enumerate(contexts, start=1))
167
+
168
+
169
+ # ----------------------------------------------------------------------------------------
170
+ # The four metrics. Each returns a float in [0, 1]; each may raise JudgeParseError (or a
171
+ # KeyError/TypeError on an unexpected JSON shape), which judge_item turns into a None.
172
+ # ----------------------------------------------------------------------------------------
173
+
174
+
175
+ def supported_claims(answer: str, contexts: list[str]) -> tuple[int, int]:
176
+ """Decompose ANSWER into atomic claims and count how many CONTEXT supports.
177
+
178
+ Returns (supported, total). The ablation harness uses the raw counts (so an empty
179
+ answer -> 0/0 is visible) while faithfulness just takes the ratio.
180
+ """
181
+ prompt = (
182
+ "Break the ANSWER into atomic factual claims. For each claim decide whether it is "
183
+ "supported by the CONTEXT (the retrieved calendar excerpts). A claim is supported only if "
184
+ "the CONTEXT contains information that backs it up.\n\n"
185
+ f"CONTEXT:\n{_join_contexts(contexts)}\n\n"
186
+ f"ANSWER:\n{answer}\n\n"
187
+ 'Return JSON: {"claims": [{"claim": "<text>", "supported": true|false}, ...]}. '
188
+ 'If the answer makes no factual claims, return {"claims": []}.'
189
+ )
190
+ claims = _chat_json(prompt, _claims_schema("supported"), "faithfulness")["claims"]
191
+ supported = sum(1 for c in claims if c.get("supported"))
192
+ return supported, len(claims)
193
+
194
+
195
+ def faithfulness(answer: str, contexts: list[str]) -> float:
196
+ """supported_claims / total_claims. Hallucination rate = 1 - this."""
197
+ supported, total = supported_claims(answer, contexts)
198
+ if total == 0:
199
+ return 1.0 # nothing asserted -> nothing unfaithful
200
+ return supported / total
201
+
202
+
203
+ def answer_relevancy(question: str, answer: str) -> float:
204
+ """Mean cosine similarity between the real question and questions the answer fits.
205
+
206
+ If the answer truly addresses the question, questions reverse-generated from it should
207
+ look a lot like the original question in embedding space.
208
+ """
209
+ prompt = (
210
+ f"Given the ANSWER below, generate {RELEVANCY_NUM_QUESTIONS} distinct questions "
211
+ "that this answer would directly and completely answer.\n\n"
212
+ f"ANSWER:\n{answer}\n\n"
213
+ 'Return JSON: {"questions": ["...", "..."]}.'
214
+ )
215
+ generated = _chat_json(prompt, _QUESTIONS_SCHEMA, "answer_relevancy")["questions"]
216
+ if not generated:
217
+ return 0.0
218
+ vecs = embed.embed_texts([question] + list(generated)) # unit vectors
219
+ sims = vecs[1:] @ vecs[0] # cosine == dot, since normalized
220
+ return float(np.clip(np.mean(sims), 0.0, 1.0))
221
+
222
+
223
+ def context_precision(question: str, reference_answer: str, contexts: list[str]) -> float:
224
+ """Rank-aware precision: were relevant excerpts retrieved, and ranked near the top?"""
225
+ prompt = (
226
+ "For each retrieved CONTEXT, decide whether it is useful for answering the "
227
+ "QUESTION, given the REFERENCE ANSWER. Output true if the context contains "
228
+ "information useful to produce the reference answer, false otherwise.\n\n"
229
+ f"QUESTION: {question}\n"
230
+ f"REFERENCE ANSWER: {reference_answer}\n\n"
231
+ f"CONTEXTS:\n{_join_contexts(contexts)}\n\n"
232
+ 'Return JSON: {"verdicts": [true|false, ...]} with one entry per context, in order.'
233
+ )
234
+ verdicts = [bool(v) for v in _chat_json(prompt, _VERDICTS_SCHEMA, "context_precision")["verdicts"]]
235
+ total_relevant = sum(verdicts)
236
+ if total_relevant == 0:
237
+ return 0.0
238
+ # Average precision: average of precision@i taken at each relevant rank i.
239
+ hits = 0
240
+ weighted = 0.0
241
+ for i, relevant in enumerate(verdicts, start=1):
242
+ if relevant:
243
+ hits += 1
244
+ weighted += hits / i
245
+ return weighted / total_relevant
246
+
247
+
248
+ def context_recall(reference_answer: str, contexts: list[str]) -> float:
249
+ """Fraction of the reference answer's claims that the retrieved context supports."""
250
+ prompt = (
251
+ "Break the REFERENCE ANSWER into atomic claims. For each claim decide whether it "
252
+ "can be attributed to (is supported by) the CONTEXT.\n\n"
253
+ f"CONTEXT:\n{_join_contexts(contexts)}\n\n"
254
+ f"REFERENCE ANSWER:\n{reference_answer}\n\n"
255
+ 'Return JSON: {"claims": [{"claim": "<text>", "attributed": true|false}, ...]}.'
256
+ )
257
+ claims = _chat_json(prompt, _claims_schema("attributed"), "context_recall")["claims"]
258
+ if not claims:
259
+ return 1.0
260
+ attributed = sum(1 for c in claims if c.get("attributed"))
261
+ return attributed / len(claims)
262
+
263
+
264
+ # Maps metric name -> a zero-arg callable, so judge_item can run each in a uniform
265
+ # try/except and record which ones failed without aborting the whole evaluation.
266
+ def judge_item(
267
+ question: str, answer: str, contexts: list[str], reference_answer: str
268
+ ) -> dict:
269
+ """Run all four metrics for one (question, answer, contexts) triple.
270
+
271
+ Returns a dict of {metric: float|None, ..., "errors": [names that failed to parse]}.
272
+ A None means the judge's reply for that metric couldn't be parsed — recorded, not
273
+ fatal, so a single bad response never sinks a long run.
274
+ """
275
+ metrics = {
276
+ "faithfulness": lambda: faithfulness(answer, contexts),
277
+ "answer_relevancy": lambda: answer_relevancy(question, answer),
278
+ "context_precision": lambda: context_precision(question, reference_answer, contexts),
279
+ "context_recall": lambda: context_recall(reference_answer, contexts),
280
+ }
281
+ scores: dict = {}
282
+ errors: list[str] = []
283
+ for name, fn in metrics.items():
284
+ try:
285
+ scores[name] = round(float(fn()), 4)
286
+ except (JudgeParseError, KeyError, TypeError, ValueError, IndexError):
287
+ scores[name] = None
288
+ errors.append(name)
289
+ scores["errors"] = errors
290
+ return scores
eval/make_testset.py ADDED
@@ -0,0 +1,127 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """
2
+ make_testset — generate a corpus-grounded test set from the indexed calendar chunks.
3
+
4
+ A test set is only meaningful if its gold ids point at chunks that are actually in the
5
+ index. So instead of hand-writing questions, we sample real indexed chunks and ask the
6
+ local LLM to write, for each one, a specific question that chunk answers plus a concise
7
+ reference answer grounded in it. The chunk's own id becomes the gold id — which is what
8
+ makes the deterministic retrieval metrics (hit@k etc.) work.
9
+
10
+ Landing pages and tiny fragments make useless questions, so sampling skips chunks with
11
+ page_type == "landing" or very short text.
12
+
13
+ This produces *candidates*. The intended workflow is to then CURATE by hand — delete vague
14
+ or wrong questions, fix wording, and (for near-duplicate chunks, e.g. the same requirement
15
+ in the 2025/26 vs 2026/27 edition or across cohort pages) add sibling ids to gold_ids —
16
+ and commit the frozen result. Start small and grow.
17
+
18
+ Run (needs a built index; uses the local LM Studio backend — see config.CHAT_MODEL):
19
+ python -m eval.make_testset --num 15 --out eval/testset.json
20
+ """
21
+
22
+ from __future__ import annotations
23
+
24
+ import argparse
25
+ import json
26
+ import random
27
+
28
+ from openai import OpenAI
29
+
30
+ from src import config, retrieve
31
+ from eval.judge import _extract_json, JudgeParseError # reuse robust JSON parsing
32
+
33
+ _MIN_CHUNK_CHARS = 300 # too little text to ground a specific question + answer
34
+
35
+ _SYSTEM = (
36
+ "You write evaluation data for a UBC academic-calendar question-answering system. "
37
+ "Respond with ONLY valid JSON — no prose, no markdown, no code fences."
38
+ )
39
+
40
+
41
+ def _prompt(chunk: dict) -> str:
42
+ return (
43
+ "Given the CALENDAR EXCERPT below, write ONE specific question that a UBC student "
44
+ "might ask and that THIS excerpt answers well, plus a concise reference answer "
45
+ "grounded only in the excerpt. The question should be specific (name the course, "
46
+ "program, or policy it is about — and the student cohort or calendar year when the "
47
+ "excerpt is cohort- or edition-specific) and answerable from the excerpt.\n\n"
48
+ f"EXCERPT TITLE: {chunk['title']}\n"
49
+ f"SOURCE URL: {chunk['url']}\n"
50
+ f"EXCERPT TEXT: {chunk['text']}\n\n"
51
+ 'Return JSON: {"question": "<question>", "reference_answer": "<answer>"}'
52
+ )
53
+
54
+
55
+ def _eligible(chunk: dict) -> bool:
56
+ return chunk.get("page_type") != "landing" and len(chunk.get("text", "")) >= _MIN_CHUNK_CHARS
57
+
58
+
59
+ def generate(num: int, seed: int) -> list[dict]:
60
+ """Sample `num` eligible indexed chunks and turn each into a test item."""
61
+ _, chunks = retrieve._load_index() # the indexed corpus (id-aligned metadata)
62
+ pool = [c for c in chunks if _eligible(c)]
63
+ print(f"Sampling from {len(pool)} eligible chunks (of {len(chunks)} indexed).")
64
+ if num > len(pool):
65
+ num = len(pool)
66
+ sampled = random.Random(seed).sample(pool, num)
67
+
68
+ client = OpenAI(base_url=config.OPENAI_BASE_URL, api_key=config.OPENAI_API_KEY)
69
+ items: list[dict] = []
70
+ for chunk in sampled:
71
+ response = client.chat.completions.create(
72
+ model=config.CHAT_MODEL,
73
+ messages=[
74
+ {"role": "system", "content": _SYSTEM},
75
+ {"role": "user", "content": _prompt(chunk)},
76
+ ],
77
+ temperature=0.3,
78
+ max_tokens=400,
79
+ )
80
+ try:
81
+ parsed = _extract_json(response.choices[0].message.content or "")
82
+ question = parsed["question"].strip()
83
+ reference_answer = parsed["reference_answer"].strip()
84
+ except (JudgeParseError, KeyError, AttributeError, TypeError):
85
+ print(f" skipped chunk {chunk['id']} ({chunk['title']!r}): bad JSON")
86
+ continue
87
+
88
+ items.append(
89
+ {
90
+ "id": f"q{len(items) + 1:03d}",
91
+ "question": question,
92
+ "reference_answer": reference_answer,
93
+ "gold_ids": [chunk["id"]],
94
+ "source_chunk_id": chunk["id"],
95
+ }
96
+ )
97
+ print(f" [{len(items)}] id={chunk['id']:<4} {chunk['title']}")
98
+ return items
99
+
100
+
101
+ def main() -> None:
102
+ parser = argparse.ArgumentParser(description="Generate a corpus-grounded eval test set.")
103
+ parser.add_argument("--num", type=int, default=15, help="how many questions to generate")
104
+ parser.add_argument("--seed", type=int, default=42, help="sampling seed (reproducible)")
105
+ parser.add_argument("--out", default=str(config.TESTSET_PATH), help="output JSON path")
106
+ args = parser.parse_args()
107
+
108
+ print(f"Generating {args.num} questions from the indexed corpus "
109
+ f"(model={config.CHAT_MODEL}) ...")
110
+ items = generate(args.num, args.seed)
111
+
112
+ payload = {
113
+ "description": (
114
+ "Auto-generated, corpus-grounded test set (eval/make_testset.py). Each gold id "
115
+ "points at an indexed calendar chunk. REVIEW AND CURATE before trusting the "
116
+ "numbers: delete vague/wrong items, fix wording, add sibling ids to gold_ids "
117
+ "for near-duplicate chunks (edition/cohort twins)."
118
+ ),
119
+ "items": items,
120
+ }
121
+ with open(args.out, "w", encoding="utf-8") as f:
122
+ json.dump(payload, f, ensure_ascii=False, indent=2)
123
+ print(f"\nWrote {len(items)} questions to {args.out}")
124
+
125
+
126
+ if __name__ == "__main__":
127
+ main()
eval/reports/history.csv ADDED
@@ -0,0 +1,4 @@
 
 
 
 
 
1
+ timestamp,name,k,num_questions,judged,chat_model,judge_model,hit@k,recall@k,mrr,faithfulness,answer_relevancy,context_precision,context_recall,hallucination_rate
2
+ 2026-08-31T00:30:01,sweep-dense,4,12,False,google/gemma-4-e4b,google/gemma-4-e4b,0.5,0.4583,0.4028,,,,,
3
+ 2026-08-31T00:30:06,sweep-hybrid,4,12,False,google/gemma-4-e4b,google/gemma-4-e4b,0.6667,0.625,0.5417,,,,,
4
+ 2026-08-31T00:32:59,sweep-hybrid-rerank,4,12,False,google/gemma-4-e4b,google/gemma-4-e4b,0.6667,0.625,0.5417,,,,,
eval/reports/report.md ADDED
@@ -0,0 +1,136 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # UBCHelper eval — sweep-hybrid-rerank
2
+
3
+ 2026-08-31T00:32:59 · 12 questions · k=4 · judge off
4
+
5
+ Models: chat=`google/gemma-4-e4b`, judge=`google/gemma-4-e4b`, embed=`text-embedding-3-small` · corpus=3770 · retrieval=hybrid_rerank
6
+
7
+ ## Summary
8
+
9
+ | metric | score |
10
+ |--------|-------|
11
+ | hit@4 | 0.667 |
12
+ | recall@4 | 0.625 |
13
+ | MRR | 0.542 |
14
+
15
+ ## Questions
16
+
17
+ ### q001 — hit — gold [10, 182]
18
+
19
+ **Q:** What are the prerequisites for CPSC 221?
20
+
21
+ **Reference:** One of CPSC 210 or CPEN 221, and either one of CPSC 121, MATH 220 (or MATH_O 220), or a score of 68% or higher in MATH 226.
22
+
23
+ **Retrieved:** [182] Computer Science, Faculty of Science [2025/26] · [10] Computer Science, Faculty of Science [2026/27] · [232] Computer Science, Faculty of Science [2025/26] · [339] Bachelor of Science — Computer Science [2025/26]
24
+
25
+ **Scores:** rr 1.00
26
+
27
+ ### q002 — miss — gold [24, 197]
28
+
29
+ **Q:** What are the prerequisites for CPSC 320?
30
+
31
+ **Reference:** CPSC 221 (or DSCI 221) plus at least 3 credits of MATH or STAT at the 200 level or above (the 2025/26 edition also accepted COMM 291 or BIOL 300 for the second part).
32
+
33
+ **Retrieved:** [42] Computer Science, Faculty of Science [2026/27] · [214] Computer Science, Faculty of Science [2025/26] · [339] Bachelor of Science — Computer Science [2025/26] · [170] Bachelor of Science — Computer Science [2026/27]
34
+
35
+ **Scores:** rr 0.00
36
+
37
+ ### q003 — hit — gold [20, 193]
38
+
39
+ **Q:** What are the prerequisites for CPSC 313?
40
+
41
+ **Reference:** CPSC 213, and either CPSC 221 or DSCI 221.
42
+
43
+ **Retrieved:** [20] Computer Science, Faculty of Science [2026/27] · [193] Computer Science, Faculty of Science [2025/26] · [213] Computer Science, Faculty of Science [2025/26] · [212] Computer Science, Faculty of Science [2025/26]
44
+
45
+ **Scores:** rr 1.00
46
+
47
+ ### q004 — hit — gold [580, 2319]
48
+
49
+ **Q:** Which courses satisfy the prerequisite for MATH 200, Calculus III?
50
+
51
+ **Reference:** One of MATH 101, MATH 103, MATH 105, MATH 121, SCIE 001, MATH_O 101, or MATH_O 103.
52
+
53
+ **Retrieved:** [580] Mathematics, Faculty of Science [2026/27] · [2319] Mathematics, Faculty of Science [2025/26] · [2348] Mathematics, Faculty of Science [2025/26] · [609] Mathematics, Faculty of Science [2026/27]
54
+
55
+ **Scores:** rr 1.00
56
+
57
+ ### q005 — miss — gold [5, 177]
58
+
59
+ **Q:** How many credits is CPSC 110 worth and what does it cover?
60
+
61
+ **Reference:** CPSC 110 (Computation, Programs, and Programming) is worth 4 credits and covers fundamental program and computation structures.
62
+
63
+ **Retrieved:** [146] Bachelor of Science — Computer Science [2026/27] · [2082] Bachelor of Science — Mathematics [2026/27] · [160] Bachelor of Science — Computer Science [2026/27] · [3568] Bachelor of Science — Mathematics [2025/26]
64
+
65
+ **Scores:** rr 0.00
66
+
67
+ ### q006 — miss — gold [114, 128]
68
+
69
+ **Q:** Compare the B.A. degree requirements for students who entered the program in 2023/24 with those who entered in 2024/25 or later.
70
+
71
+ **Reference:** Students entering 2023/24 or earlier must complete seven requirements: Writing and Research, Language, Science, Literature, Outside, Upper-level, and Arts Credit Minimum. Students entering 2024/25 or later complete five: Writing and Research, Ways of Knowing Breadth, Outside, Upper-level, and Arts Credit Minimum — the separate Language, Science, and Literature requirements are folded into the Ways of Knowing Breadth requirement.
72
+
73
+ **Retrieved:** [3039] Bachelor of Arts — Second Degree Studies [2025/26] · [1295] Bachelor of Arts — Second Degree Studies [2026/27] · [2712] Dual Degree Program Option: Bachelor of Arts, UBC and Sciences Po [2025/26] · [963] Dual Degree Program Option: Bachelor of Arts, UBC and Sciences Po [2026/27]
74
+
75
+ **Scores:** rr 0.00
76
+
77
+ ### q007 — hit — gold [1769, 1791]
78
+
79
+ **Q:** How does admission to the Bachelor of Computer Science program differ from B.Sc. admission from secondary school?
80
+
81
+ **Reference:** Bachelor of Computer Science admission is a competitive selection process evaluating applicants on academic and other criteria (not every qualified applicant is admitted); B.Sc. admission from secondary school requires starting first year in September of the Winter session of admission (January starts are not permitted).
82
+
83
+ **Retrieved:** [1768] Bachelor of Computer Science [2026/27] · [1769] Bachelor of Computer Science [2026/27] · [1767] Bachelor of Computer Science [2026/27] · [3275] Bachelor of Science — Admission and Transfer [2025/26]
84
+
85
+ **Scores:** rr 0.50
86
+
87
+ ### q008 — miss — gold [20, 10]
88
+
89
+ **Q:** I have finished CPSC 210. What else do I need to complete before I can take CPSC 313?
90
+
91
+ **Reference:** CPSC 313 requires CPSC 213 and either CPSC 221 or DSCI 221. CPSC 221 in turn requires (beyond CPSC 210) one of CPSC 121, MATH 220, MATH_O 220, or a 68%+ score in MATH 226. So after CPSC 210 you still need CPSC 213 and CPSC 221 (with its discrete-math prerequisite).
92
+
93
+ **Retrieved:** [917] Bachelor of Arts — Computer Science [2026/27] · [2663] Bachelor of Arts — Computer Science [2025/26] · [154] Bachelor of Science — Computer Science [2026/27] · [160] Bachelor of Science — Computer Science [2026/27]
94
+
95
+ **Scores:** rr 0.00
96
+
97
+ ### q009 — hit — gold [3508]
98
+
99
+ **Q:** In the 2025/26 calendar, what is the minimum number of credits required for a B.Sc. degree?
100
+
101
+ **Reference:** A minimum of 120 credits (a major, double major, or General Science option requires at least 120 credits but may require more).
102
+
103
+ **Retrieved:** [3509] Bachelor of Science — General Degree Requirements [2025/26] · [3508] Bachelor of Science — General Degree Requirements [2025/26] · [3438] Bachelor of Science — Credit at UBC and Elsewhere [2025/26] · [2023] Bachelor of Science — General Degree Requirements [2026/27]
104
+
105
+ **Scores:** rr 0.50
106
+
107
+ ### q010 — hit — gold [2023]
108
+
109
+ **Q:** According to the current 2026/27 calendar, what is the minimum number of credits required for a B.Sc. degree?
110
+
111
+ **Reference:** A minimum of 120 credits (a major, double major, or General Science option requires at least 120 credits but may require more).
112
+
113
+ **Retrieved:** [2023] Bachelor of Science — General Degree Requirements [2026/27] · [2024] Bachelor of Science — General Degree Requirements [2026/27] · [1954] Bachelor of Science — Credit at UBC and Elsewhere [2026/27] · [2174] Bachelor of Science — Recognition of Academic Achievement [2026/27]
114
+
115
+ **Scores:** rr 1.00
116
+
117
+ ### q011 — hit — gold [128]
118
+
119
+ **Q:** Does a B.A. student who entered the program in 2024/25 or later have to complete a language requirement?
120
+
121
+ **Reference:** No. Their five degree requirements are Writing and Research, Ways of Knowing Breadth, Outside, Upper-level, and Arts Credit Minimum — there is no separate language requirement (that applies to students who entered in 2023/24 or earlier).
122
+
123
+ **Retrieved:** [2890] Bachelor of Arts — Linguistics [2025/26] · [128] Bachelor of Arts — Degree Requirements for students who enter the program in 2024/25 or later [2026/27] · [2903] Bachelor of Arts — Linguistics [2025/26] · [2285] Bachelor of International Economics — Degree Requirements for students who enter the program in 2024/25 or later [2026/27]
124
+
125
+ **Scores:** rr 0.50
126
+
127
+ ### q012 — hit — gold [0]
128
+
129
+ **Q:** What are the three levels of academic standing at UBC?
130
+
131
+ **Reference:** In Good Standing; On Academic Probation; and Failed, Required to Withdraw. All students are In Good Standing on initial entry to the University.
132
+
133
+ **Retrieved:** [0] Academic Standing [2026/27] · [431] Course Standings [2026/27] · [430] Course Standings [2026/27] · [3658] Bachelor of Science — Recognition of Academic Achievement [2025/26]
134
+
135
+ **Scores:** rr 1.00
136
+
eval/retrieval_metrics.py ADDED
@@ -0,0 +1,138 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """
2
+ Tier 1 — retrieval metrics. Deterministic, no LLM, instant, free.
3
+
4
+ This is the fast inner loop for tuning retrieval: it asks "did the retriever surface the
5
+ right excerpt?" purely by comparing the ids it returned against the gold ids in the test
6
+ set. No judge model, no embeddings beyond the query embedding the retriever already does.
7
+
8
+ Three standard metrics, each computed per question over the ranked list of returned ids,
9
+ then averaged across all questions:
10
+
11
+ hit@k did ANY gold id land in the top-k? (1 or 0) -> mean
12
+ recall@k what FRACTION of gold ids landed in the top-k? -> mean
13
+ MRR 1 / rank of the FIRST gold hit (0 if none) -> mean
14
+
15
+ Run it (needs LM Studio for the query embeddings + a built index):
16
+ python -m eval.retrieval_metrics
17
+ python -m eval.retrieval_metrics --k 3 --limit 5
18
+ """
19
+
20
+ from __future__ import annotations
21
+
22
+ import argparse
23
+ import json
24
+
25
+ from src import config, retrieve
26
+
27
+ # ----------------------------------------------------------------------------------------
28
+ # Pure metric functions. These take plain lists of ids, so they are fully deterministic
29
+ # and unit-testable with no LM Studio, no index, no network.
30
+ # ----------------------------------------------------------------------------------------
31
+
32
+
33
+ def hit_at_k(retrieved_ids: list[int], gold_ids: list[int], k: int) -> float:
34
+ """1.0 if at least one gold id appears in the top-k retrieved ids, else 0.0."""
35
+ top_k = set(retrieved_ids[:k])
36
+ return 1.0 if top_k & set(gold_ids) else 0.0
37
+
38
+
39
+ def recall_at_k(retrieved_ids: list[int], gold_ids: list[int], k: int) -> float:
40
+ """Fraction of the gold ids that appear in the top-k retrieved ids."""
41
+ if not gold_ids:
42
+ return 0.0
43
+ top_k = set(retrieved_ids[:k])
44
+ found = top_k & set(gold_ids)
45
+ return len(found) / len(gold_ids)
46
+
47
+
48
+ def reciprocal_rank(retrieved_ids: list[int], gold_ids: list[int]) -> float:
49
+ """1 / (rank of the first gold hit). 0.0 if no gold id is in the list.
50
+
51
+ Averaging this across questions gives MRR (Mean Reciprocal Rank).
52
+ """
53
+ gold = set(gold_ids)
54
+ for rank, doc_id in enumerate(retrieved_ids, start=1):
55
+ if doc_id in gold:
56
+ return 1.0 / rank
57
+ return 0.0
58
+
59
+
60
+ # ----------------------------------------------------------------------------------------
61
+ # Runner: drive the retriever over the test set and aggregate the metrics above.
62
+ # (This part DOES need LM Studio — to embed each query — and a built index.)
63
+ # ----------------------------------------------------------------------------------------
64
+
65
+
66
+ def load_testset(path=config.TESTSET_PATH) -> list[dict]:
67
+ with open(path, "r", encoding="utf-8") as f:
68
+ return json.load(f)["items"]
69
+
70
+
71
+ def evaluate_retrieval(testset: list[dict], k: int) -> dict:
72
+ """Run retrieval for every question and return aggregate + per-item metrics.
73
+
74
+ `k` controls both how many excerpts we retrieve and the @k in the metrics, so the eval
75
+ measures retrieval at the depth it reports.
76
+ """
77
+ per_item = []
78
+ for item in testset:
79
+ results = retrieve.retrieve(item["question"], top_k=k)
80
+ retrieved_ids = [r.id for r in results]
81
+ gold_ids = item["gold_ids"]
82
+ per_item.append(
83
+ {
84
+ "id": item["id"],
85
+ "gold_ids": gold_ids,
86
+ "retrieved_ids": retrieved_ids,
87
+ "hit": hit_at_k(retrieved_ids, gold_ids, k),
88
+ "recall": recall_at_k(retrieved_ids, gold_ids, k),
89
+ "rr": reciprocal_rank(retrieved_ids, gold_ids),
90
+ }
91
+ )
92
+
93
+ n = len(per_item) or 1 # avoid divide-by-zero on an empty test set
94
+ aggregate = {
95
+ "k": k,
96
+ "num_questions": len(per_item),
97
+ f"hit@{k}": sum(x["hit"] for x in per_item) / n,
98
+ f"recall@{k}": sum(x["recall"] for x in per_item) / n,
99
+ "mrr": sum(x["rr"] for x in per_item) / n,
100
+ }
101
+ return {"aggregate": aggregate, "per_item": per_item}
102
+
103
+
104
+ def _print_report(report: dict) -> None:
105
+ agg = report["aggregate"]
106
+ k = agg["k"]
107
+ hit = agg[f"hit@{k}"]
108
+ recall = agg[f"recall@{k}"]
109
+ print(f"\nRetrieval metrics over {agg['num_questions']} questions (k={k}):")
110
+ print(f" hit@{k} : {hit:.3f}")
111
+ print(f" recall@{k} : {recall:.3f}")
112
+ print(f" MRR : {agg['mrr']:.3f}")
113
+
114
+ print("\nPer-question ([+] = a gold id was retrieved):")
115
+ for x in report["per_item"]:
116
+ mark = "[+]" if x["hit"] else "[ ]"
117
+ print(
118
+ f" {mark} {x['id']} gold={x['gold_ids']} "
119
+ f"retrieved={x['retrieved_ids']} rr={x['rr']:.3f}"
120
+ )
121
+
122
+
123
+ def main() -> None:
124
+ parser = argparse.ArgumentParser(description="Run Tier-1 retrieval metrics.")
125
+ parser.add_argument("--k", type=int, default=config.EVAL_K, help="retrieval depth / @k")
126
+ parser.add_argument("--limit", type=int, default=None, help="only the first N questions")
127
+ args = parser.parse_args()
128
+
129
+ testset = load_testset()
130
+ if args.limit:
131
+ testset = testset[: args.limit]
132
+
133
+ report = evaluate_retrieval(testset, k=args.k)
134
+ _print_report(report)
135
+
136
+
137
+ if __name__ == "__main__":
138
+ main()
eval/run_ablation.py ADDED
@@ -0,0 +1,249 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """
2
+ Ablation — how much is the RAG actually helping?
3
+
4
+ Answers every test question two ways and scores both with the SAME metric:
5
+
6
+ RAG : retrieve relevant excerpts -> grounded generation (the real pipeline)
7
+ no-RAG : closed-book, no context -> the model answers from memory (the baseline)
8
+
9
+ Both answers are judged for GROUNDEDNESS against the question's *gold excerpt* — the fraction
10
+ of the answer's atomic claims that the true excerpt supports (reusing judge.supported_claims).
11
+ Hallucination rate = 1 - groundedness. Because the judge compares against the gold excerpt (not
12
+ the retrieved context), the metric is well-defined for the no-RAG arm too, so the two are
13
+ directly comparable. The gap between the arms is the value retrieval adds.
14
+
15
+ The test set is the frozen specific-source questions (eval/testset.json) plus a few broad
16
+ general calendar questions (eval/testset.general.json); results are broken down by type, since
17
+ RAG helps most on specific-source questions and least on general knowledge.
18
+
19
+ NOTE: "hallucination" here means "unsupported by the gold excerpt", so a closed-book answer
20
+ that gives plausible but *different* requirements still counts as unsupported. That is intended
21
+ for specific-source questions; it is why general questions are reported separately.
22
+
23
+ Run (needs an index built; the judge defaults to a local LM Studio server — see
24
+ config.JUDGE_MODEL):
25
+ python -m eval.run_ablation
26
+ python -m eval.run_ablation --limit 2 # quick smoke test
27
+ """
28
+
29
+ from __future__ import annotations
30
+
31
+ import argparse
32
+ import json
33
+ from datetime import datetime
34
+ from statistics import mean
35
+
36
+ from src import config, generate, retrieve
37
+ from eval import judge
38
+
39
+
40
+ def _load_items(path, kind: str) -> list[dict]:
41
+ """Load a test set file and tag every item with its `type` (specific/general)."""
42
+ if not path.exists():
43
+ return []
44
+ data = json.loads(path.read_text(encoding="utf-8"))
45
+ items = data["items"]
46
+ for it in items:
47
+ it["type"] = kind
48
+ return items
49
+
50
+
51
+ def _groundedness(answer: str, gold_texts: list[str]) -> dict:
52
+ """Score one answer against the gold excerpt(s). Returns score + raw claim counts.
53
+
54
+ For near-dup questions with several valid golds, a claim supported by ANY gold counts —
55
+ so the answer is judged against the union of all gold excerpts.
56
+ """
57
+ supported, total = judge.supported_claims(answer, gold_texts)
58
+ score = 1.0 if total == 0 else supported / total
59
+ return {"score": round(score, 4), "supported": supported, "total": total}
60
+
61
+
62
+ def run(items: list[dict], k: int) -> dict:
63
+ """Run both arms for every item and return the full ablation report dict."""
64
+ _, metadata = retrieve._load_index()
65
+ chunk_by_id = {r["id"]: r for r in metadata}
66
+
67
+ per_item = []
68
+ for n, item in enumerate(items, start=1):
69
+ gold_ids = item["gold_ids"]
70
+ golds = [chunk_by_id[g] for g in gold_ids if g in chunk_by_id]
71
+ if not golds:
72
+ print(f" [{n}/{len(items)}] {item['id']} SKIPPED — no gold ids {gold_ids} in index")
73
+ continue
74
+ gold_texts = [g["text"] for g in golds]
75
+ gold_title = golds[0]["title"] + (f" (+{len(golds) - 1} more)" if len(golds) > 1 else "")
76
+ question = item["question"]
77
+
78
+ # Arm A: RAG (retrieve -> grounded generation). Arm B: closed-book (no context).
79
+ rag_answer = generate.generate_answer(question, retrieve.retrieve(question, top_k=k))
80
+ cb_answer = generate.generate_answer_closed_book(question)
81
+
82
+ rag = _groundedness(rag_answer, gold_texts)
83
+ cb = _groundedness(cb_answer, gold_texts)
84
+
85
+ per_item.append({
86
+ "id": item["id"],
87
+ "type": item["type"],
88
+ "question": question,
89
+ "reference_answer": item.get("reference_answer", ""),
90
+ "gold_id": gold_ids,
91
+ "gold_title": gold_title,
92
+ "rag": {"answer": rag_answer, **rag},
93
+ "no_rag": {"answer": cb_answer, **cb},
94
+ })
95
+ print(
96
+ f" [{n}/{len(items)}] {item['id']} ({item['type']}) "
97
+ f"RAG ground={rag['score']:.2f} ({rag['supported']}/{rag['total']}) "
98
+ f"no-RAG ground={cb['score']:.2f} ({cb['supported']}/{cb['total']})"
99
+ )
100
+
101
+ return _aggregate(per_item, k)
102
+
103
+
104
+ def _aggregate(per_item: list[dict], k: int) -> dict:
105
+ def summarize(rows: list[dict]) -> dict | None:
106
+ if not rows:
107
+ return None
108
+ rag_g = mean(r["rag"]["score"] for r in rows)
109
+ cb_g = mean(r["no_rag"]["score"] for r in rows)
110
+ return {
111
+ "n": len(rows),
112
+ "rag_groundedness": round(rag_g, 4),
113
+ "no_rag_groundedness": round(cb_g, 4),
114
+ "rag_hallucination": round(1.0 - rag_g, 4),
115
+ "no_rag_hallucination": round(1.0 - cb_g, 4),
116
+ "hallucination_delta": round((1.0 - rag_g) - (1.0 - cb_g), 4), # neg = RAG helps
117
+ }
118
+
119
+ by_type = {
120
+ "overall": summarize(per_item),
121
+ "specific": summarize([r for r in per_item if r["type"] == "specific"]),
122
+ "general": summarize([r for r in per_item if r["type"] == "general"]),
123
+ }
124
+ return {
125
+ "run": {
126
+ "timestamp": datetime.now().isoformat(timespec="seconds"),
127
+ "k": k,
128
+ "num_questions": len(per_item),
129
+ "chat_model": config.CHAT_MODEL,
130
+ "judge_model": config.JUDGE_MODEL,
131
+ "embedding_model": config.EMBEDDING_MODEL,
132
+ "corpus_size": len(retrieve._load_index()[1]), # indexed chunk count
133
+ },
134
+ "summary": by_type,
135
+ "per_item": per_item,
136
+ }
137
+
138
+
139
+ # ----------------------------------------------------------------------------------------
140
+ # Report: a single self-contained markdown doc (summary table + every answer side by side).
141
+ # ----------------------------------------------------------------------------------------
142
+
143
+ def _fmt(v) -> str:
144
+ return "-" if v is None else f"{v:.3f}"
145
+
146
+
147
+ def write_report(report: dict, name: str):
148
+ config.REPORTS_DIR.mkdir(parents=True, exist_ok=True)
149
+ config.ABLATION_REPORT_PATH.write_text(_render_markdown(report, name), encoding="utf-8")
150
+ return config.ABLATION_REPORT_PATH
151
+
152
+
153
+ def _render_markdown(report: dict, name: str) -> str:
154
+ run = report["run"]
155
+ s = report["summary"]
156
+ n_specific = sum(1 for r in report["per_item"] if r["type"] == "specific")
157
+ n_general = sum(1 for r in report["per_item"] if r["type"] == "general")
158
+
159
+ lines = [
160
+ f"# UBCHelper ablation — {name}",
161
+ "",
162
+ f"{run['timestamp']} · {run['num_questions']} questions "
163
+ f"({n_specific} specific + {n_general} general) · k={run['k']}",
164
+ "",
165
+ f"Models: chat=`{run['chat_model']}`, judge=`{run['judge_model']}`, "
166
+ f"embed=`{run['embedding_model']}` · corpus={run['corpus_size']}",
167
+ "",
168
+ "Groundedness = fraction of the answer's claims supported by the gold excerpt "
169
+ "(higher is better). Hallucination = 1 - groundedness. Both arms are judged against "
170
+ "the same gold excerpt, so a closed-book answer that gives plausible but *different* "
171
+ "requirements still counts as unsupported — which is why general questions are listed "
172
+ "separately.",
173
+ "",
174
+ "## Summary — RAG vs no-RAG",
175
+ "",
176
+ "| set | n | RAG groundedness | no-RAG groundedness | RAG halluc. | no-RAG halluc. | halluc. delta (RAG-noRAG) |",
177
+ "|-----|---|------------------|---------------------|-------------|----------------|---------------------------|",
178
+ ]
179
+ for label in ("overall", "specific", "general"):
180
+ row = s.get(label)
181
+ if row is None:
182
+ continue
183
+ lines.append(
184
+ f"| {label} | {row['n']} | {_fmt(row['rag_groundedness'])} | "
185
+ f"{_fmt(row['no_rag_groundedness'])} | {_fmt(row['rag_hallucination'])} | "
186
+ f"{_fmt(row['no_rag_hallucination'])} | {_fmt(row['hallucination_delta'])} |"
187
+ )
188
+
189
+ lines += ["", "## Questions", ""]
190
+ for x in report["per_item"]:
191
+ rag, cb = x["rag"], x["no_rag"]
192
+ lines += [
193
+ f"### {x['id']} — {x['type']} — gold {x['gold_id']} {x['gold_title']}",
194
+ "",
195
+ f"**Q:** {x['question']}",
196
+ "",
197
+ f"**Reference:** {x['reference_answer']}",
198
+ "",
199
+ f"**RAG answer** — groundedness {rag['score']:.3f} "
200
+ f"({rag['supported']}/{rag['total']} claims):",
201
+ "",
202
+ rag["answer"],
203
+ "",
204
+ f"**No-RAG answer** — groundedness {cb['score']:.3f} "
205
+ f"({cb['supported']}/{cb['total']} claims):",
206
+ "",
207
+ cb["answer"],
208
+ "",
209
+ ]
210
+ return "\n".join(lines) + "\n"
211
+
212
+
213
+ def _print_summary(report: dict) -> None:
214
+ print("\n=== RAG vs no-RAG (groundedness, higher is better) ===")
215
+ for label in ("overall", "specific", "general"):
216
+ row = report["summary"].get(label)
217
+ if row is None:
218
+ continue
219
+ print(
220
+ f"{label:<9} (n={row['n']:>2}): "
221
+ f"RAG={_fmt(row['rag_groundedness'])} no-RAG={_fmt(row['no_rag_groundedness'])} "
222
+ f"| halluc RAG={_fmt(row['rag_hallucination'])} vs no-RAG={_fmt(row['no_rag_hallucination'])}"
223
+ )
224
+
225
+
226
+ def main() -> None:
227
+ parser = argparse.ArgumentParser(description="Run the RAG vs no-RAG ablation.")
228
+ parser.add_argument("--name", default="rag-vs-norag", help="label for this run")
229
+ parser.add_argument("--k", type=int, default=config.EVAL_K, help="retrieval depth for the RAG arm")
230
+ parser.add_argument("--limit", type=int, default=None, help="only the first N questions")
231
+ args = parser.parse_args()
232
+
233
+ items = _load_items(config.TESTSET_PATH, "specific") + _load_items(
234
+ config.GENERAL_TESTSET_PATH, "general"
235
+ )
236
+ if args.limit:
237
+ items = items[: args.limit]
238
+
239
+ print(f"Running ablation '{args.name}' over {len(items)} questions "
240
+ f"(RAG k={args.k} + closed-book) ...")
241
+ report = run(items, k=args.k)
242
+
243
+ _print_summary(report)
244
+ report_path = write_report(report, args.name)
245
+ print(f"\nWrote:\n {report_path}")
246
+
247
+
248
+ if __name__ == "__main__":
249
+ main()
eval/run_eval.py ADDED
@@ -0,0 +1,293 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """
2
+ Orchestrator — run the whole evaluation and record the result.
3
+
4
+ For every question in the frozen test set it does ONE retrieval pass, then (unless
5
+ --retrieval-only) generates an answer and judges it, computing both metric tiers:
6
+
7
+ Tier 1 (retrieval) : hit@k, recall@k, MRR -- deterministic, from eval/retrieval_metrics
8
+ Tier 2 (generation): faithfulness, answer_relevancy, context_precision/recall -- eval/judge
9
+
10
+ It writes two things to eval/reports/:
11
+ report.md ONE self-contained doc: summary metrics + every question's full test
12
+ data (question, reference answer, retrieved excerpts, answer, scores).
13
+ Overwritten each run.
14
+ history.csv ONE row per run -- the cross-run performance ledger (kept so you can
15
+ compare runs, e.g. phase1 vs phase2, since report.md only holds the last).
16
+
17
+ Run (needs an index built; the judge defaults to a local LM Studio server — see
18
+ config.JUDGE_MODEL):
19
+ python -m eval.run_eval --name phase1-dense
20
+ python -m eval.run_eval --retrieval-only # fast: skip generation + judge
21
+ python -m eval.run_eval --limit 3 # quick smoke test
22
+ """
23
+
24
+ from __future__ import annotations
25
+
26
+ import argparse
27
+ import csv
28
+ from datetime import datetime
29
+ from statistics import mean
30
+
31
+ from src import config, generate, retrieve
32
+ from eval import judge
33
+ from eval import retrieval_metrics as rm
34
+
35
+ GEN_METRICS = ["faithfulness", "answer_relevancy", "context_precision", "context_recall"]
36
+
37
+
38
+ def _mean_or_none(values: list[float]):
39
+ vals = [v for v in values if v is not None]
40
+ return round(mean(vals), 4) if vals else None
41
+
42
+
43
+ def run(testset: list[dict], k: int, do_judge: bool) -> dict:
44
+ """Evaluate every test item and return the full report dict."""
45
+ per_item = []
46
+ for n, item in enumerate(testset, start=1):
47
+ # --- Tier 1: retrieve once, score deterministically -----------------------------
48
+ results = retrieve.retrieve(item["question"], top_k=k)
49
+ retrieved_ids = [r.id for r in results]
50
+ gold_ids = item["gold_ids"]
51
+ record = {
52
+ "id": item["id"],
53
+ "question": item["question"],
54
+ "reference_answer": item.get("reference_answer", ""),
55
+ "gold_ids": gold_ids,
56
+ "retrieved_ids": retrieved_ids,
57
+ "retrieved_titles": [r.recipe["title"] for r in results],
58
+ "retrieval": {
59
+ "hit": rm.hit_at_k(retrieved_ids, gold_ids, k),
60
+ "recall": rm.recall_at_k(retrieved_ids, gold_ids, k),
61
+ "rr": rm.reciprocal_rank(retrieved_ids, gold_ids),
62
+ },
63
+ }
64
+
65
+ # --- Tier 2: generate an answer, then judge it ----------------------------------
66
+ if do_judge:
67
+ contexts = [r.recipe["text"] for r in results]
68
+ answer = generate.generate_answer(item["question"], results)
69
+ record["answer"] = answer
70
+ record["judge"] = judge.judge_item(
71
+ item["question"], answer, contexts, item["reference_answer"]
72
+ )
73
+ faith = record["judge"]["faithfulness"]
74
+ print(
75
+ f" [{n}/{len(testset)}] {item['id']} hit={record['retrieval']['hit']:.0f}"
76
+ f" faithfulness={'-' if faith is None else f'{faith:.2f}'}"
77
+ )
78
+ else:
79
+ print(f" [{n}/{len(testset)}] {item['id']} hit={record['retrieval']['hit']:.0f}")
80
+
81
+ per_item.append(record)
82
+
83
+ return _aggregate(per_item, k, do_judge)
84
+
85
+
86
+ def _aggregate(per_item: list[dict], k: int, do_judge: bool) -> dict:
87
+ n = len(per_item) or 1
88
+ retrieval = {
89
+ "hit@k": round(sum(x["retrieval"]["hit"] for x in per_item) / n, 4),
90
+ "recall@k": round(sum(x["retrieval"]["recall"] for x in per_item) / n, 4),
91
+ "mrr": round(sum(x["retrieval"]["rr"] for x in per_item) / n, 4),
92
+ }
93
+
94
+ generation = None
95
+ hallucination_rate = None
96
+ if do_judge:
97
+ generation = {
98
+ m: _mean_or_none([x["judge"][m] for x in per_item]) for m in GEN_METRICS
99
+ }
100
+ if generation["faithfulness"] is not None:
101
+ hallucination_rate = round(1.0 - generation["faithfulness"], 4)
102
+
103
+ return {
104
+ "run": {
105
+ "timestamp": datetime.now().isoformat(timespec="seconds"),
106
+ "k": k,
107
+ "num_questions": len(per_item),
108
+ "judged": do_judge,
109
+ "retrieval_mode": config.RETRIEVAL_MODE,
110
+ "chat_model": config.CHAT_MODEL,
111
+ "judge_model": config.JUDGE_MODEL,
112
+ "embedding_model": config.EMBEDDING_MODEL,
113
+ "corpus_size": len(retrieve._load_index()[1]), # indexed chunk count
114
+ },
115
+ "aggregate": {
116
+ "retrieval": retrieval,
117
+ "generation": generation,
118
+ "hallucination_rate": hallucination_rate,
119
+ },
120
+ "per_item": per_item,
121
+ }
122
+
123
+
124
+ # ----------------------------------------------------------------------------------------
125
+ # Output: a single, self-contained markdown report, plus the history.csv ledger.
126
+ # ----------------------------------------------------------------------------------------
127
+
128
+ HISTORY_COLUMNS = [
129
+ "timestamp", "name", "k", "num_questions", "judged",
130
+ "chat_model", "judge_model",
131
+ "hit@k", "recall@k", "mrr",
132
+ "faithfulness", "answer_relevancy", "context_precision", "context_recall",
133
+ "hallucination_rate",
134
+ ]
135
+
136
+
137
+ def _fmt(value) -> str:
138
+ return "-" if value is None else f"{value:.3f}"
139
+
140
+
141
+ def write_report(report: dict, name: str):
142
+ """Write the single markdown report and append one row to the history ledger."""
143
+ config.REPORTS_DIR.mkdir(parents=True, exist_ok=True)
144
+ config.REPORT_PATH.write_text(_render_markdown(report, name), encoding="utf-8")
145
+ _append_history(report, name)
146
+ return config.REPORT_PATH
147
+
148
+
149
+ def _render_markdown(report: dict, name: str) -> str:
150
+ """One concise markdown doc: a summary table, then every question's full test data."""
151
+ run = report["run"]
152
+ agg = report["aggregate"]
153
+ ret = agg["retrieval"]
154
+ gen = agg["generation"]
155
+ k = run["k"]
156
+
157
+ # --- header + summary metric table ---
158
+ lines = [
159
+ f"# UBCHelper eval — {name}",
160
+ "",
161
+ f"{run['timestamp']} · {run['num_questions']} questions · k={k} · "
162
+ f"judge {'on' if run['judged'] else 'off'}",
163
+ "",
164
+ f"Models: chat=`{run['chat_model']}`, judge=`{run['judge_model']}`, "
165
+ f"embed=`{run['embedding_model']}` · corpus={run['corpus_size']} · "
166
+ f"retrieval={run['retrieval_mode']}",
167
+ "",
168
+ "## Summary",
169
+ "",
170
+ "| metric | score |",
171
+ "|--------|-------|",
172
+ f"| hit@{k} | {_fmt(ret['hit@k'])} |",
173
+ f"| recall@{k} | {_fmt(ret['recall@k'])} |",
174
+ f"| MRR | {_fmt(ret['mrr'])} |",
175
+ ]
176
+ if gen is not None:
177
+ lines += [
178
+ f"| faithfulness | {_fmt(gen['faithfulness'])} |",
179
+ f"| answer_relevancy | {_fmt(gen['answer_relevancy'])} |",
180
+ f"| context_precision | {_fmt(gen['context_precision'])} |",
181
+ f"| context_recall | {_fmt(gen['context_recall'])} |",
182
+ f"| **hallucination_rate** | **{_fmt(agg['hallucination_rate'])}** |",
183
+ ]
184
+
185
+ # --- per-question detail: all test data, one block each ---
186
+ lines += ["", "## Questions", ""]
187
+ for x in report["per_item"]:
188
+ r = x["retrieval"]
189
+ marker = "hit" if r["hit"] else "miss"
190
+ retrieved = " · ".join(
191
+ f"[{i}] {t}" for i, t in zip(x["retrieved_ids"], x["retrieved_titles"])
192
+ )
193
+ lines += [
194
+ f"### {x['id']} — {marker} — gold {x['gold_ids']}",
195
+ "",
196
+ f"**Q:** {x['question']}",
197
+ "",
198
+ f"**Reference:** {x['reference_answer']}",
199
+ "",
200
+ f"**Retrieved:** {retrieved}",
201
+ ]
202
+ if "judge" in x:
203
+ j = x["judge"]
204
+ lines += [
205
+ "",
206
+ f"**Answer:** {x['answer']}",
207
+ "",
208
+ f"**Scores:** rr {r['rr']:.2f} · faithfulness {_fmt(j['faithfulness'])} "
209
+ f"· answer_relevancy {_fmt(j['answer_relevancy'])} "
210
+ f"· context_precision {_fmt(j['context_precision'])} "
211
+ f"· context_recall {_fmt(j['context_recall'])}"
212
+ + (f" · parse-failed: {', '.join(j['errors'])}" if j["errors"] else ""),
213
+ ]
214
+ else:
215
+ lines += ["", f"**Scores:** rr {r['rr']:.2f}"]
216
+ lines.append("")
217
+
218
+ return "\n".join(lines) + "\n"
219
+
220
+
221
+ def _append_history(report: dict, name: str) -> None:
222
+ run = report["run"]
223
+ ret = report["aggregate"]["retrieval"]
224
+ gen = report["aggregate"]["generation"] or {}
225
+ row = {
226
+ "timestamp": run["timestamp"],
227
+ "name": name,
228
+ "k": run["k"],
229
+ "num_questions": run["num_questions"],
230
+ "judged": run["judged"],
231
+ "chat_model": run["chat_model"],
232
+ "judge_model": run["judge_model"],
233
+ "hit@k": ret["hit@k"],
234
+ "recall@k": ret["recall@k"],
235
+ "mrr": ret["mrr"],
236
+ "faithfulness": gen.get("faithfulness"),
237
+ "answer_relevancy": gen.get("answer_relevancy"),
238
+ "context_precision": gen.get("context_precision"),
239
+ "context_recall": gen.get("context_recall"),
240
+ "hallucination_rate": report["aggregate"]["hallucination_rate"],
241
+ }
242
+ is_new = not config.HISTORY_PATH.exists()
243
+ with open(config.HISTORY_PATH, "a", encoding="utf-8", newline="") as f:
244
+ writer = csv.DictWriter(f, fieldnames=HISTORY_COLUMNS)
245
+ if is_new:
246
+ writer.writeheader()
247
+ writer.writerow(row)
248
+
249
+
250
+ def _print_summary(report: dict) -> None:
251
+ agg = report["aggregate"]
252
+ ret = agg["retrieval"]
253
+ k = report["run"]["k"]
254
+ print("\n=== Summary ===")
255
+ print(f"Retrieval : hit@{k}={_fmt(ret['hit@k'])} "
256
+ f"recall@{k}={_fmt(ret['recall@k'])} mrr={_fmt(ret['mrr'])}")
257
+ gen = agg["generation"]
258
+ if gen is not None:
259
+ print(f"Generation: faithfulness={_fmt(gen['faithfulness'])} "
260
+ f"answer_relevancy={_fmt(gen['answer_relevancy'])}")
261
+ print(f" context_precision={_fmt(gen['context_precision'])} "
262
+ f"context_recall={_fmt(gen['context_recall'])}")
263
+ print(f"Hallucination rate (1 - faithfulness): "
264
+ f"{_fmt(agg['hallucination_rate'])}")
265
+
266
+
267
+ def main() -> None:
268
+ parser = argparse.ArgumentParser(description="Run the UBCHelper evaluation harness.")
269
+ parser.add_argument("--name", default="run", help="label for this run (e.g. phase1-dense)")
270
+ parser.add_argument("--k", type=int, default=config.EVAL_K, help="retrieval depth / @k")
271
+ parser.add_argument("--limit", type=int, default=None, help="only the first N questions")
272
+ parser.add_argument(
273
+ "--retrieval-only",
274
+ action="store_true",
275
+ help="skip generation + the LLM judge (fast, deterministic, no chat model needed)",
276
+ )
277
+ args = parser.parse_args()
278
+
279
+ testset = rm.load_testset()
280
+ if args.limit:
281
+ testset = testset[: args.limit]
282
+
283
+ print(f"Running '{args.name}' over {len(testset)} questions (k={args.k}, "
284
+ f"mode={config.RETRIEVAL_MODE}, judge={'off' if args.retrieval_only else 'on'}) ...")
285
+ report = run(testset, k=args.k, do_judge=not args.retrieval_only)
286
+
287
+ _print_summary(report)
288
+ report_path = write_report(report, args.name)
289
+ print(f"\nWrote:\n {report_path}\n {config.HISTORY_PATH}")
290
+
291
+
292
+ if __name__ == "__main__":
293
+ main()
eval/testset.general.json ADDED
@@ -0,0 +1,33 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "description": "Broad questions a closed-book model can reasonably attempt, each still bound to a gold calendar chunk — used by eval/run_ablation.py as the 'general' arm. gold_ids are valid only for the 3,770-chunk index built 2026-08-31; regenerate after any re-ingest.",
3
+ "items": [
4
+ {
5
+ "id": "g001",
6
+ "question": "What happens to a UBC student's academic standing, and what standings exist?",
7
+ "reference_answer": "There are three levels of academic standing: In Good Standing, On Academic Probation, and Failed, Required to Withdraw. Students enter the University In Good Standing.",
8
+ "gold_ids": [0],
9
+ "source_chunk_id": 0
10
+ },
11
+ {
12
+ "id": "g002",
13
+ "question": "Roughly how many credits does a UBC B.Sc. degree require?",
14
+ "reference_answer": "A minimum of 120 credits; some options require more.",
15
+ "gold_ids": [2023],
16
+ "source_chunk_id": 2023
17
+ },
18
+ {
19
+ "id": "g003",
20
+ "question": "What is CPSC 110 at UBC about?",
21
+ "reference_answer": "CPSC 110, Computation, Programs, and Programming (4 credits), covers fundamental program and computation structures.",
22
+ "gold_ids": [5],
23
+ "source_chunk_id": 5
24
+ },
25
+ {
26
+ "id": "g004",
27
+ "question": "What kinds of requirements does a UBC Bachelor of Arts student have to complete?",
28
+ "reference_answer": "B.A. students complete a set of faculty-wide degree requirements — for current entrants (2024/25 or later): Writing and Research, Ways of Knowing Breadth, Outside, Upper-level, and Arts Credit Minimum; earlier cohorts also had separate Language, Science, and Literature requirements.",
29
+ "gold_ids": [128, 114],
30
+ "source_chunk_id": 128
31
+ }
32
+ ]
33
+ }
eval/testset.json ADDED
@@ -0,0 +1,101 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "description": "Hand-curated, corpus-grounded test set (authored directly from indexed chunks; regenerate candidates with eval/make_testset.py once a local LLM backend is available). Categories: code-lookup (hinges on a course code — BM25's job), multi-hop (spans chunks — the agent/reranker's job), collision (edition- or cohort-qualified — the corpus's trap), policy. gold_ids are valid ONLY for the 3,770-chunk index built 2026-08-31 from the full live-2627 + archive-2526 crawl; regenerate after any re-ingest.",
3
+ "items": [
4
+ {
5
+ "id": "q001",
6
+ "category": "code-lookup",
7
+ "question": "What are the prerequisites for CPSC 221?",
8
+ "reference_answer": "One of CPSC 210 or CPEN 221, and either one of CPSC 121, MATH 220 (or MATH_O 220), or a score of 68% or higher in MATH 226.",
9
+ "gold_ids": [10, 182],
10
+ "source_chunk_id": 10
11
+ },
12
+ {
13
+ "id": "q002",
14
+ "category": "code-lookup",
15
+ "question": "What are the prerequisites for CPSC 320?",
16
+ "reference_answer": "CPSC 221 (or DSCI 221) plus at least 3 credits of MATH or STAT at the 200 level or above (the 2025/26 edition also accepted COMM 291 or BIOL 300 for the second part).",
17
+ "gold_ids": [24, 197],
18
+ "source_chunk_id": 24
19
+ },
20
+ {
21
+ "id": "q003",
22
+ "category": "code-lookup",
23
+ "question": "What are the prerequisites for CPSC 313?",
24
+ "reference_answer": "CPSC 213, and either CPSC 221 or DSCI 221.",
25
+ "gold_ids": [20, 193],
26
+ "source_chunk_id": 20
27
+ },
28
+ {
29
+ "id": "q004",
30
+ "category": "code-lookup",
31
+ "question": "Which courses satisfy the prerequisite for MATH 200, Calculus III?",
32
+ "reference_answer": "One of MATH 101, MATH 103, MATH 105, MATH 121, SCIE 001, MATH_O 101, or MATH_O 103.",
33
+ "gold_ids": [580, 2319],
34
+ "source_chunk_id": 580
35
+ },
36
+ {
37
+ "id": "q005",
38
+ "category": "code-lookup",
39
+ "question": "How many credits is CPSC 110 worth and what does it cover?",
40
+ "reference_answer": "CPSC 110 (Computation, Programs, and Programming) is worth 4 credits and covers fundamental program and computation structures.",
41
+ "gold_ids": [5, 177],
42
+ "source_chunk_id": 5
43
+ },
44
+ {
45
+ "id": "q006",
46
+ "category": "multi-hop",
47
+ "question": "Compare the B.A. degree requirements for students who entered the program in 2023/24 with those who entered in 2024/25 or later.",
48
+ "reference_answer": "Students entering 2023/24 or earlier must complete seven requirements: Writing and Research, Language, Science, Literature, Outside, Upper-level, and Arts Credit Minimum. Students entering 2024/25 or later complete five: Writing and Research, Ways of Knowing Breadth, Outside, Upper-level, and Arts Credit Minimum — the separate Language, Science, and Literature requirements are folded into the Ways of Knowing Breadth requirement.",
49
+ "gold_ids": [114, 128],
50
+ "source_chunk_id": 114
51
+ },
52
+ {
53
+ "id": "q007",
54
+ "category": "multi-hop",
55
+ "question": "How does admission to the Bachelor of Computer Science program differ from B.Sc. admission from secondary school?",
56
+ "reference_answer": "Bachelor of Computer Science admission is a competitive selection process evaluating applicants on academic and other criteria (not every qualified applicant is admitted); B.Sc. admission from secondary school requires starting first year in September of the Winter session of admission (January starts are not permitted).",
57
+ "gold_ids": [1769, 1791],
58
+ "source_chunk_id": 1769
59
+ },
60
+ {
61
+ "id": "q008",
62
+ "category": "multi-hop",
63
+ "question": "I have finished CPSC 210. What else do I need to complete before I can take CPSC 313?",
64
+ "reference_answer": "CPSC 313 requires CPSC 213 and either CPSC 221 or DSCI 221. CPSC 221 in turn requires (beyond CPSC 210) one of CPSC 121, MATH 220, MATH_O 220, or a 68%+ score in MATH 226. So after CPSC 210 you still need CPSC 213 and CPSC 221 (with its discrete-math prerequisite).",
65
+ "gold_ids": [20, 10],
66
+ "source_chunk_id": 20
67
+ },
68
+ {
69
+ "id": "q009",
70
+ "category": "collision",
71
+ "question": "In the 2025/26 calendar, what is the minimum number of credits required for a B.Sc. degree?",
72
+ "reference_answer": "A minimum of 120 credits (a major, double major, or General Science option requires at least 120 credits but may require more).",
73
+ "gold_ids": [3508],
74
+ "source_chunk_id": 3508
75
+ },
76
+ {
77
+ "id": "q010",
78
+ "category": "collision",
79
+ "question": "According to the current 2026/27 calendar, what is the minimum number of credits required for a B.Sc. degree?",
80
+ "reference_answer": "A minimum of 120 credits (a major, double major, or General Science option requires at least 120 credits but may require more).",
81
+ "gold_ids": [2023],
82
+ "source_chunk_id": 2023
83
+ },
84
+ {
85
+ "id": "q011",
86
+ "category": "collision",
87
+ "question": "Does a B.A. student who entered the program in 2024/25 or later have to complete a language requirement?",
88
+ "reference_answer": "No. Their five degree requirements are Writing and Research, Ways of Knowing Breadth, Outside, Upper-level, and Arts Credit Minimum — there is no separate language requirement (that applies to students who entered in 2023/24 or earlier).",
89
+ "gold_ids": [128],
90
+ "source_chunk_id": 128
91
+ },
92
+ {
93
+ "id": "q012",
94
+ "category": "policy",
95
+ "question": "What are the three levels of academic standing at UBC?",
96
+ "reference_answer": "In Good Standing; On Academic Probation; and Failed, Required to Withdraw. All students are In Good Standing on initial entry to the University.",
97
+ "gold_ids": [0],
98
+ "source_chunk_id": 0
99
+ }
100
+ ]
101
+ }
src/__init__.py ADDED
@@ -0,0 +1,6 @@
 
 
 
 
 
 
 
1
+ """UBCHelper — a retrieval-augmented UBC Vancouver Academic Calendar advisor.
2
+
3
+ A domain re-skin of RAGChef: the same generic retrieval/generation engine over a scraped
4
+ Academic Calendar corpus. This package currently holds the data layer — scrape -> chunk ->
5
+ ingest — which fills the engine's document index.
6
+ """
src/agent.py ADDED
@@ -0,0 +1,96 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """
2
+ Agent — the complex path of the adaptive pipeline (Phase 4), framework-free.
3
+
4
+ A plain Python loop, no LangChain/LangGraph, so the whole thing stays readable:
5
+
6
+ decompose(question) -> sub-questions one LLM call
7
+ for each sub-question: retrieve.retrieve(...) multi-hop recall
8
+ round-robin merge + dedup the Result lists each hop's best hit rises to the top
9
+ synthesize(question, merged) one grounded answer over the union
10
+
11
+ The merge is what earns the complex path its keep: a single retrieval tends to satisfy one
12
+ part of a multi-hop question and starve the other, whereas retrieving each sub-question
13
+ separately and interleaving the rankings surfaces an excerpt for *every* hop — the recall
14
+ win the eval measures.
15
+
16
+ Synthesis deliberately reuses `generate.generate_answer`: the grounded SYSTEM_PROMPT and
17
+ `format_context` already turn a list of excerpts + the original question into one clean
18
+ answer, so there is nothing to reinvent here.
19
+ """
20
+
21
+ from __future__ import annotations
22
+
23
+ from . import config, generate, retrieve, trace
24
+ from .retrieve import Result
25
+
26
+ DECOMPOSE_SYSTEM = (
27
+ "You break a complex question about the UBC Academic Calendar into a few focused "
28
+ "sub-questions, each of which can be answered from ONE calendar page (one course, "
29
+ "one program's requirements, one policy). Cover every distinct course, program, "
30
+ "student cohort, or side of a comparison the question asks about. Write one "
31
+ "sub-question per line, no numbering, and produce at most {max_subq}. If the "
32
+ "question is already single-focus, just restate it as the only line."
33
+ )
34
+
35
+
36
+ def decompose(question: str) -> list[str]:
37
+ """Split `question` into up to AGENT_MAX_SUBQ sub-questions; fall back to the original."""
38
+ trace.step("AGENT - decompose into sub-questions")
39
+ system = DECOMPOSE_SYSTEM.format(max_subq=config.AGENT_MAX_SUBQ)
40
+ reply = generate.complete(system, question)
41
+ subqs = [_strip_bullet(line) for line in reply.splitlines()]
42
+ subqs = [s for s in subqs if s]
43
+ subqs = subqs[: config.AGENT_MAX_SUBQ] if subqs else [question]
44
+ for i, sq in enumerate(subqs, start=1):
45
+ trace.detail(f"sub-question {i}", sq)
46
+ return subqs
47
+
48
+
49
+ def _strip_bullet(line: str) -> str:
50
+ """Drop any leading list marker ("1.", "-", "*") the model may have added."""
51
+ return line.strip().lstrip("0123456789.)-*• \t").strip()
52
+
53
+
54
+ def _merge(result_lists: list[list[Result]]) -> list[Result]:
55
+ """Round-robin interleave per-sub-question results, deduped by chunk id (first wins).
56
+
57
+ Taking rank-1 from every sub-question before any rank-2 keeps each hop's strongest hit
58
+ near the top, which is exactly what recall@k on a multi-hop question rewards.
59
+ """
60
+ merged: list[Result] = []
61
+ seen: set[int] = set()
62
+ for rank in range(max((len(rl) for rl in result_lists), default=0)):
63
+ for rl in result_lists:
64
+ if rank < len(rl) and rl[rank].id not in seen:
65
+ seen.add(rl[rank].id)
66
+ merged.append(rl[rank])
67
+ return merged
68
+
69
+
70
+ def synthesize(question: str, results: list[Result]) -> str:
71
+ """Answer the original question grounded in the merged excerpts (capped for prompt size)."""
72
+ return generate.generate_answer(question, results[: config.SYNTH_CONTEXT_K])
73
+
74
+
75
+ def run(question: str, top_k: int | None = None) -> tuple[list[Result], str]:
76
+ """Decompose -> multi-hop retrieve -> merge -> synthesize. Returns (merged_results, answer).
77
+
78
+ `top_k` sets how many excerpts to pull per sub-question (defaults to AGENT_SUBQ_TOP_K);
79
+ the merged list can be longer, since it unions every hop.
80
+ """
81
+ per_hop = top_k or config.AGENT_SUBQ_TOP_K
82
+ subqs = decompose(question)
83
+
84
+ result_lists = []
85
+ for i, sq in enumerate(subqs, start=1):
86
+ trace.step(f"AGENT - multi-hop retrieve, sub-question {i}/{len(subqs)}: {sq!r}")
87
+ result_lists.append(retrieve.retrieve(sq, top_k=per_hop))
88
+
89
+ merged = _merge(result_lists)
90
+ trace.step("AGENT - merge sub-question results (round-robin, dedup by chunk id)")
91
+ trace.detail("merged excerpts", len(merged))
92
+ trace.results(merged)
93
+
94
+ trace.step("AGENT - synthesize final answer over the merged excerpts")
95
+ answer = synthesize(question, merged)
96
+ return merged, answer
src/bm25.py ADDED
@@ -0,0 +1,115 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """
2
+ BM25 — hand-rolled sparse (keyword) retrieval, the other half of hybrid search.
3
+
4
+ Dense embeddings capture *meaning* but under-weight rare, high-signal tokens: a query
5
+ about "CPSC 320" lands near generic computer-science text because the code barely moves
6
+ the vector. BM25 is the opposite — it scores documents by exact term overlap, weighting
7
+ each term by how rare it is in the corpus (IDF), saturating repeated occurrences (k1), and
8
+ penalizing long documents (b). Rare course-code/program tokens are exactly where it
9
+ shines, which is why fusing it with dense retrieval helps.
10
+
11
+ This is the classic Okapi BM25 formula, built with numpy so the mechanism stays visible
12
+ (same spirit as the hand-rolled vector store):
13
+
14
+ score(q, d) = sum over query terms t of
15
+ idf(t) * tf(t,d) * (k1 + 1) / (tf(t,d) + k1 * (1 - b + b * |d| / avgdl))
16
+ idf(t) = ln(1 + (N - df(t) + 0.5) / (df(t) + 0.5))
17
+
18
+ Everything inside the sum depends only on the corpus, so each term's per-document
19
+ contribution is precomputed at build time; scoring a query is then just summing a few
20
+ postings arrays. The index is built in memory from the text the dense index already
21
+ stores (a few seconds even for tens of thousands of chunks, once per process) — no new artifact on disk and
22
+ no re-ingest needed.
23
+ """
24
+
25
+ from __future__ import annotations
26
+
27
+ import math
28
+ import re
29
+ from collections import Counter
30
+ from dataclasses import dataclass
31
+
32
+ import numpy as np
33
+
34
+ # Standard Okapi parameters.
35
+ K1 = 1.5 # term-frequency saturation: repeats help, with diminishing returns
36
+ B = 0.75 # document-length normalization strength
37
+
38
+ _TOKEN_RE = re.compile(r"[a-z0-9]+")
39
+
40
+
41
+ def tokenize(text: str) -> list[str]:
42
+ """Lowercase word/number tokens. Used for BOTH documents and queries — the sparse
43
+ analogue of "the same embedding model must embed the index and the query"."""
44
+ return _TOKEN_RE.findall(text.lower())
45
+
46
+
47
+ @dataclass
48
+ class _Index:
49
+ # term -> (doc indices, that term's precomputed idf*tf-saturation score per doc)
50
+ postings: dict[str, tuple[np.ndarray, np.ndarray]]
51
+ n_docs: int
52
+
53
+
54
+ def _build(texts: list[str]) -> _Index:
55
+ docs = [Counter(tokenize(t)) for t in texts]
56
+ doc_lens = np.array([sum(c.values()) for c in docs], dtype=np.float64)
57
+ avgdl = doc_lens.mean() or 1.0
58
+ n = len(texts)
59
+
60
+ # Document frequency: in how many docs does each term appear?
61
+ df: Counter[str] = Counter()
62
+ for counts in docs:
63
+ df.update(counts.keys())
64
+
65
+ # Per-document length penalty (the k1 * (1 - b + b*|d|/avgdl) denominator part).
66
+ norm = K1 * (1 - B + B * doc_lens / avgdl)
67
+
68
+ by_term: dict[str, tuple[list[int], list[float]]] = {t: ([], []) for t in df}
69
+ for i, counts in enumerate(docs):
70
+ for term, tf in counts.items():
71
+ indices, weights = by_term[term]
72
+ indices.append(i)
73
+ weights.append(tf * (K1 + 1) / (tf + norm[i]))
74
+
75
+ postings = {}
76
+ for term, (indices, weights) in by_term.items():
77
+ idf = math.log(1 + (n - df[term] + 0.5) / (df[term] + 0.5))
78
+ postings[term] = (
79
+ np.array(indices, dtype=np.int32),
80
+ idf * np.array(weights, dtype=np.float32),
81
+ )
82
+ return _Index(postings=postings, n_docs=n)
83
+
84
+
85
+ # Built once per process on first search, cached for the corpus it was built from.
86
+ _index: _Index | None = None
87
+ _source: list[str] | None = None
88
+
89
+
90
+ def _get_index(texts: list[str]) -> _Index:
91
+ global _index, _source
92
+ if _index is None or _source is not texts:
93
+ _index = _build(texts)
94
+ _source = texts
95
+ return _index
96
+
97
+
98
+ def search(query: str, k: int, texts: list[str]) -> list[tuple[int, float]]:
99
+ """Return the top-k (doc index, BM25 score) pairs for `query`, best first.
100
+
101
+ Only documents sharing at least one term with the query are returned, so the list
102
+ can be shorter than k.
103
+ """
104
+ index = _get_index(texts)
105
+ scores = np.zeros(index.n_docs, dtype=np.float32)
106
+ for term in tokenize(query):
107
+ entry = index.postings.get(term)
108
+ if entry is not None:
109
+ indices, weights = entry
110
+ scores[indices] += weights
111
+
112
+ k = min(k, index.n_docs)
113
+ top_idx = np.argpartition(scores, -k)[-k:]
114
+ top_idx = top_idx[np.argsort(scores[top_idx])[::-1]]
115
+ return [(int(i), float(scores[i])) for i in top_idx if scores[i] > 0]
src/generate.py ADDED
@@ -0,0 +1,174 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """
2
+ Generate — the "G" in RAG. Turn retrieved calendar excerpts + a question into a grounded answer.
3
+
4
+ The whole idea of RAG: instead of asking the LLM to answer from memory (where it may
5
+ hallucinate), we hand it real excerpts from the UBC Academic Calendar as context and
6
+ instruct it to answer *from that context*. The model becomes a reader/synthesizer over
7
+ retrieved facts.
8
+
9
+ `generate_answer` returns a string and never prints. That keeps it usable both from the
10
+ CLI (which prints the return value) and, later, from the Phase 3 eval loop (which runs
11
+ 100 questions and must not spew to stdout).
12
+
13
+ Two backends produce the answer, selected by `config.LLM_BACKEND`:
14
+ - "anthropic": Claude via the Anthropic API (the default, so the app runs without a
15
+ local model server).
16
+ - "local": any OpenAI-compatible local server (LM Studio, llama.cpp, vLLM, Ollama's
17
+ OpenAI shim) via `/v1/chat/completions`.
18
+
19
+ Both paths take the same (system prompt, user turn) and return a plain string, so the
20
+ callers and the eval loop don't care which backend is active.
21
+ """
22
+
23
+ from __future__ import annotations
24
+
25
+ from . import config, trace
26
+ from .retrieve import Result
27
+
28
+ # System prompt: defines the assistant's role and the grounding rule.
29
+ # The user only sees their question and the final answer — never the retrieved context —
30
+ # so the answer must read as standalone advice and must not point back at the numbered
31
+ # excerpt blocks. Two calendar-specific twists: (1) the calendar is authoritative and
32
+ # changes yearly, so the answer should hand the user the official page URL(s) carried on
33
+ # each excerpt; (2) the corpus deliberately contains colliding editions (2026/27 live vs
34
+ # 2025/26 archive) and cohort-split requirement pages, so the model must disambiguate.
35
+ SYSTEM_PROMPT = (
36
+ "You are a UBC academic advisor. Answer the user's question using ONLY the excerpts "
37
+ "from the UBC Vancouver Academic Calendar provided as context, but write the answer "
38
+ "as if you simply know it. Do NOT mention or number the excerpts: the user cannot "
39
+ 'see them, so phrases like "Excerpt 3" are meaningless to them. You SHOULD, however, '
40
+ "point the user to the official calendar page URL(s) shown with the excerpts for the "
41
+ "authoritative and most current wording. Excerpts may come from different calendar "
42
+ "editions (e.g. 2026/27 vs 2025/26) or different student cohorts — answer for the "
43
+ "edition and cohort the question asks about, defaulting to the current 2026/27 "
44
+ "calendar, and note when requirements differ between them. Keep the answer concise "
45
+ "and focused directly on what was asked. If the excerpts do not contain enough "
46
+ "information to answer, say so plainly rather than inventing requirements."
47
+ )
48
+
49
+ # Closed-book prompt: the "no-RAG" baseline for the ablation. The model answers from its
50
+ # own parametric memory with no retrieved context. We push it to commit to concrete
51
+ # specifics (rather than refuse) so its answer makes measurable claims to score for
52
+ # hallucination against the gold calendar excerpt.
53
+ CLOSED_BOOK_SYSTEM = (
54
+ "You are a UBC academic advisor. Answer the user's question about UBC Vancouver "
55
+ "courses, programs, and policies from your own knowledge, with concrete specifics "
56
+ "(course codes, credit counts, requirements), kept concise. Give your single best "
57
+ "answer; do not refuse or say you lack access to the official calendar."
58
+ )
59
+
60
+ # Both clients are created lazily and cached, so importing this module is cheap and only
61
+ # the backend actually in use gets constructed.
62
+ _openai_client = None
63
+ _anthropic_client = None
64
+
65
+
66
+ def _get_openai_client():
67
+ """The OpenAI client pointed at the local server (LLM_BACKEND == 'local')."""
68
+ global _openai_client
69
+ if _openai_client is None:
70
+ from openai import OpenAI
71
+
72
+ _openai_client = OpenAI(
73
+ base_url=config.OPENAI_BASE_URL,
74
+ api_key=config.OPENAI_API_KEY,
75
+ )
76
+ return _openai_client
77
+
78
+
79
+ def _get_anthropic_client():
80
+ """The Anthropic client (LLM_BACKEND == 'anthropic'); reads ANTHROPIC_API_KEY."""
81
+ global _anthropic_client
82
+ if _anthropic_client is None:
83
+ import anthropic
84
+
85
+ _anthropic_client = anthropic.Anthropic()
86
+ return _anthropic_client
87
+
88
+
89
+ def format_context(results: list[Result]) -> str:
90
+ """Render retrieved calendar excerpts into a readable, numbered context block for the
91
+ prompt. Each chunk's `text` already leads with its page title + section breadcrumb;
92
+ the header line and Source URL make the excerpt scannable and citable."""
93
+ blocks = []
94
+ for n, r in enumerate(results, start=1):
95
+ chunk = r.recipe # engine-wide name for "the document record"
96
+ blocks.append(
97
+ f"[Excerpt {n}] {chunk['title']}\n"
98
+ f"Source: {chunk['url']}\n"
99
+ f"{chunk['text']}"
100
+ )
101
+ return "\n\n".join(blocks)
102
+
103
+
104
+ def build_user_turn(query: str, results: list[Result]) -> str:
105
+ """Render the grounded user turn: the retrieved excerpts followed by the question."""
106
+ context = format_context(results)
107
+ return (
108
+ f"Here are some excerpts from the UBC Vancouver Academic Calendar that may be "
109
+ f"relevant:\n\n{context}\n\n"
110
+ f"Question: {query}"
111
+ )
112
+
113
+
114
+ def complete(system: str, user: str) -> str:
115
+ """One-shot completion via the configured backend, for callers that build their own
116
+ prompt (the Phase-4 router and agent) rather than the grounded calendar turn."""
117
+ return _complete(system, user)
118
+
119
+
120
+ def _complete(system: str, user: str) -> str:
121
+ """Dispatch one (system, user) turn to the configured backend and return the text.
122
+
123
+ Every LLM call in the project funnels through here, so this is the one place that traces
124
+ the exact prompt sent to the API (when tracing is on; a no-op otherwise).
125
+ """
126
+ model = config.ANTHROPIC_MODEL if config.LLM_BACKEND == "anthropic" else config.CHAT_MODEL
127
+ trace.prompt(system, user, model)
128
+ if config.LLM_BACKEND == "anthropic":
129
+ text = _complete_anthropic(system, user)
130
+ else:
131
+ text = _complete_local(system, user)
132
+ trace.response(text)
133
+ return text
134
+
135
+
136
+ def _complete_local(system: str, user: str) -> str:
137
+ """OpenAI-compatible local server via /v1/chat/completions."""
138
+ client = _get_openai_client()
139
+ response = client.chat.completions.create(
140
+ model=config.CHAT_MODEL,
141
+ messages=[
142
+ {"role": "system", "content": system},
143
+ {"role": "user", "content": user},
144
+ ],
145
+ temperature=config.TEMPERATURE,
146
+ max_tokens=config.MAX_TOKENS,
147
+ )
148
+ return (response.choices[0].message.content or "").strip()
149
+
150
+
151
+ def _complete_anthropic(system: str, user: str) -> str:
152
+ """Claude via the Anthropic API. `system` is a top-level arg, not a message.
153
+
154
+ Note: current Claude models don't take sampling params (no `temperature`/`top_p`),
155
+ so grounding is steered by SYSTEM_PROMPT rather than by config.TEMPERATURE.
156
+ """
157
+ client = _get_anthropic_client()
158
+ response = client.messages.create(
159
+ model=config.ANTHROPIC_MODEL,
160
+ max_tokens=config.MAX_TOKENS,
161
+ system=system,
162
+ messages=[{"role": "user", "content": user}],
163
+ )
164
+ return "".join(b.text for b in response.content if b.type == "text").strip()
165
+
166
+
167
+ def generate_answer(query: str, results: list[Result]) -> str:
168
+ """Answer the question grounded in the retrieved calendar excerpts."""
169
+ return _complete(SYSTEM_PROMPT, build_user_turn(query, results))
170
+
171
+
172
+ def generate_answer_closed_book(query: str) -> str:
173
+ """Answer with NO retrieved context — the no-RAG baseline for the ablation harness."""
174
+ return _complete(CLOSED_BOOK_SYSTEM, query)
src/pipeline.py ADDED
@@ -0,0 +1,42 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """
2
+ Pipeline — the single adaptive entry point (Phase 4).
3
+
4
+ `answer()` is the one function the CLI, the API, and the eval all call. It routes each
5
+ question, then runs the matching path and returns a uniform result:
6
+
7
+ router.classify(q) -> "simple" retrieve + generate a grounded answer
8
+ "complex" hand off to the agent (decompose -> multi-hop
9
+ retrieve -> merge -> synthesize)
10
+
11
+ Returning {answer, results, route} keeps callers backend-agnostic: the CLI prints it, the
12
+ API serializes it, and the eval scores `results` (retrieval) and `answer` (generation).
13
+ """
14
+
15
+ from __future__ import annotations
16
+
17
+ from . import agent, generate, retrieve, router, trace
18
+ from .retrieve import Result
19
+
20
+
21
+ def answer(
22
+ question: str,
23
+ top_k: int | None = None,
24
+ mode: str | None = None,
25
+ route: str | None = None,
26
+ ) -> dict:
27
+ """Route `question` and answer it. Pass `route` to force a path (skips classification)."""
28
+ trace.step(f"PIPELINE - answer: {question!r}")
29
+ route = route or router.classify(question)
30
+ if route == "complex":
31
+ results, ans = agent.run(question, top_k=top_k)
32
+ else:
33
+ trace.step("RETRIEVE - simple route (single query)")
34
+ results = retrieve.retrieve(question, top_k=top_k, mode=mode)
35
+ trace.step("GENERATE - grounded answer over the retrieved excerpts")
36
+ ans = generate.generate_answer(question, results)
37
+ trace.step(f"PIPELINE - done (route={route}, {len(results)} excerpts)")
38
+ return {"answer": ans, "results": results, "route": route}
39
+
40
+
41
+ # Re-exported so callers can annotate/inspect results without importing retrieve directly.
42
+ __all__ = ["answer", "Result"]
src/rerank.py ADDED
@@ -0,0 +1,68 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """
2
+ Rerank — the precision stage of the retrieval funnel.
3
+
4
+ The bi-encoder (embeddings) scores query and document *separately*, so it can only
5
+ compare their summaries. A cross-encoder reads the query and a document *together* and
6
+ scores the pair jointly — far sharper, but far too slow to run against the whole corpus.
7
+ So it only ever sees the short candidate list that the cheap recall stage (dense + BM25)
8
+ already produced.
9
+
10
+ Reranking runs on Cohere's hosted rerank API (config.RERANKER_MODEL). Keeping it hosted
11
+ means no local torch / sentence-transformers dependency, so the service deploys on an
12
+ ordinary CPU host. Excerpts longer than the model's window are truncated server-side,
13
+ which is fine: the title/section header + the start of the body carry the ranking signal.
14
+ """
15
+
16
+ from __future__ import annotations
17
+
18
+ import time
19
+
20
+ from . import config
21
+ from .retrieve import Result
22
+
23
+ # The Cohere client is created lazily and cached: only the hybrid_rerank mode needs it,
24
+ # so dense/hybrid modes never construct it.
25
+ _client = None
26
+
27
+
28
+ def _get_client():
29
+ global _client
30
+ if _client is None:
31
+ import cohere
32
+
33
+ _client = cohere.Client(config.COHERE_API_KEY)
34
+ return _client
35
+
36
+
37
+ def rerank(query: str, candidates: list[Result], top_k: int) -> list[Result]:
38
+ """Re-score `candidates` jointly against `query`; return the top_k, best first.
39
+
40
+ The returned Results carry the Cohere relevance score (0-1, higher is better),
41
+ replacing whatever score the recall stage assigned.
42
+ """
43
+ import cohere
44
+
45
+ client = _get_client()
46
+ # Trial keys are capped at 10 calls/minute; back off and retry on 429 so a batch
47
+ # (e.g. the eval loop) doesn't die mid-run.
48
+ for attempt in range(4):
49
+ try:
50
+ response = client.rerank(
51
+ model=config.RERANKER_MODEL,
52
+ query=query,
53
+ documents=[r.recipe["text"] for r in candidates],
54
+ top_n=top_k,
55
+ )
56
+ break
57
+ except cohere.errors.TooManyRequestsError:
58
+ if attempt == 3:
59
+ raise
60
+ time.sleep(15 * (attempt + 1))
61
+ return [
62
+ Result(
63
+ id=candidates[result.index].id,
64
+ score=result.relevance_score,
65
+ recipe=candidates[result.index].recipe,
66
+ )
67
+ for result in response.results
68
+ ]
src/retrieve.py ADDED
@@ -0,0 +1,162 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """
2
+ Retrieve — given a question, return the most relevant calendar excerpts.
3
+
4
+ This is the "R" in RAG. Phase 2 turns it into a two-stage funnel:
5
+
6
+ stage 1 — RECALL dense (vector) and sparse (BM25) search each nominate ~20
7
+ candidates; Reciprocal Rank Fusion merges the two rankings.
8
+ stage 2 — PRECISION a cross-encoder re-scores the fused shortlist jointly with
9
+ the query; the final top-k come out the top.
10
+
11
+ Which stages run is a config knob (UBCAL_RETRIEVAL_MODE), so the eval can compare:
12
+
13
+ dense Phase-1 pure vector search (the baseline)
14
+ hybrid stage 1 only (dense + BM25, RRF-fused)
15
+ hybrid_rerank both stages (the Phase-2 default)
16
+
17
+ Why RRF for fusion? Cosine similarities and BM25 scores live on totally different
18
+ scales, so averaging them is meaningless. RRF ignores the scores entirely and combines
19
+ *ranks* — a document's fused score is sum(1 / (RRF_K + rank)) across the lists it
20
+ appears in — sidestepping the calibration problem.
21
+
22
+ Exposing the doc `id` on every result is deliberate: the Phase 3 evaluation harness
23
+ matches retrieved ids against "gold" chunk ids to compute hit@k / recall@k.
24
+ """
25
+
26
+ from __future__ import annotations
27
+
28
+ import json
29
+ import sys
30
+ from dataclasses import dataclass
31
+
32
+ import numpy as np
33
+
34
+ from . import bm25, config, embed, trace
35
+
36
+
37
+ @dataclass
38
+ class Result:
39
+ """One retrieved excerpt (chunk record) plus its retrieval score.
40
+
41
+ What `score` means depends on the mode that produced it: cosine similarity (dense),
42
+ RRF fused score (hybrid), or a cross-encoder logit (hybrid_rerank). Higher is always
43
+ better within one result list, but values are not comparable across modes.
44
+ """
45
+
46
+ id: int
47
+ score: float
48
+ recipe: dict # the full metadata record from the index
49
+
50
+
51
+ # The index is loaded once and cached for the process lifetime.
52
+ _embeddings: np.ndarray | None = None
53
+ _metadata: list[dict] | None = None
54
+ _texts: list[str] | None = None
55
+
56
+
57
+ def _load_index() -> tuple[np.ndarray, list[dict]]:
58
+ global _embeddings, _metadata
59
+ if _embeddings is None or _metadata is None:
60
+ if not config.EMBEDDINGS_PATH.exists() or not config.METADATA_PATH.exists():
61
+ sys.exit(
62
+ "No index found. Build it first with: python -m src.ingest"
63
+ )
64
+ _embeddings = np.load(config.EMBEDDINGS_PATH)
65
+ with open(config.METADATA_PATH, "r", encoding="utf-8") as f:
66
+ _metadata = json.load(f)
67
+ return _embeddings, _metadata
68
+
69
+
70
+ def _corpus_texts() -> list[str]:
71
+ """The raw document texts, id-aligned with the index (what BM25 searches over)."""
72
+ global _texts
73
+ if _texts is None:
74
+ _, metadata = _load_index()
75
+ _texts = [r["text"] for r in metadata]
76
+ return _texts
77
+
78
+
79
+ def _dense_search(query: str, k: int) -> list[tuple[int, float]]:
80
+ """Embed the query and rank every chunk by cosine similarity; top-k, best first."""
81
+ embeddings, metadata = _load_index()
82
+
83
+ # Embed the query (shape (1, dim)) and reduce to a 1-D vector.
84
+ query_vec = embed.embed_texts([query])[0]
85
+
86
+ # Cosine similarity == dot product, since every vector is unit-length.
87
+ # scores[i] is the similarity between the query and chunk i.
88
+ scores = embeddings @ query_vec
89
+
90
+ # Indices of the top_k highest scores. argpartition finds them without fully
91
+ # sorting the whole array; we then sort just those k by score, descending.
92
+ k = min(k, len(metadata))
93
+ top_idx = np.argpartition(scores, -k)[-k:]
94
+ top_idx = top_idx[np.argsort(scores[top_idx])[::-1]]
95
+ return [(int(i), float(scores[i])) for i in top_idx]
96
+
97
+
98
+ def _rrf_fuse(rankings: list[list[int]]) -> list[tuple[int, float]]:
99
+ """Merge ranked doc-id lists with Reciprocal Rank Fusion; best fused score first."""
100
+ fused: dict[int, float] = {}
101
+ for ranking in rankings:
102
+ for rank, doc_id in enumerate(ranking, start=1):
103
+ fused[doc_id] = fused.get(doc_id, 0.0) + 1.0 / (config.RRF_K + rank)
104
+ return sorted(fused.items(), key=lambda pair: pair[1], reverse=True)
105
+
106
+
107
+ def _as_results(ranked: list[tuple[int, float]], limit: int) -> list[Result]:
108
+ _, metadata = _load_index()
109
+ return [Result(id=i, score=s, recipe=metadata[i]) for i, s in ranked[:limit]]
110
+
111
+
112
+ def retrieve(query: str, top_k: int | None = None, mode: str | None = None) -> list[Result]:
113
+ """Return the top_k excerpts most relevant to `query`, best first.
114
+
115
+ `mode` defaults to config.RETRIEVAL_MODE (see the module docstring for the three
116
+ modes). The signature and Result shape are stable across modes — the eval harness
117
+ and CLI call this one function regardless of pipeline.
118
+ """
119
+ top_k = top_k or config.TOP_K
120
+ mode = mode or config.RETRIEVAL_MODE
121
+ if mode not in ("dense", "hybrid", "hybrid_rerank"):
122
+ sys.exit(
123
+ f"Unknown retrieval mode {mode!r} — expected dense, hybrid, or hybrid_rerank."
124
+ )
125
+ trace.detail("retrieve mode", mode)
126
+ trace.detail("embed query (OpenAI API)", repr(query))
127
+
128
+ if mode == "dense":
129
+ out = _as_results(_dense_search(query, top_k), top_k)
130
+ trace.results(out)
131
+ return out
132
+
133
+ # Stage 1 — recall: each retriever nominates candidates, RRF merges the rankings.
134
+ dense_ids = [i for i, _ in _dense_search(query, config.DENSE_K)]
135
+ sparse_ids = [i for i, _ in bm25.search(query, config.SPARSE_K, _corpus_texts())]
136
+ fused = _rrf_fuse([dense_ids, sparse_ids])
137
+ trace.detail("stage 1 recall", f"dense={len(dense_ids)} + bm25={len(sparse_ids)} "
138
+ f"-> RRF-fused to {len(fused)} unique")
139
+
140
+ if mode == "hybrid":
141
+ out = _as_results(fused, top_k)
142
+ trace.results(out)
143
+ return out
144
+
145
+ # Stage 2 — precision: the cross-encoder re-scores the fused shortlist.
146
+ # Imported here so dense/hybrid modes never pay the multi-second torch import.
147
+ from . import rerank
148
+
149
+ shortlist = _as_results(fused, config.RERANK_CANDIDATES)
150
+ trace.detail("stage 2 precision", f"rerank {len(shortlist)} candidates (Cohere API) "
151
+ f"-> top {top_k}")
152
+ out = rerank.rerank(query, shortlist, top_k)
153
+ trace.results(out)
154
+ return out
155
+
156
+
157
+ if __name__ == "__main__":
158
+ # Quick manual check: python -m src.retrieve "prerequisites for CPSC 210"
159
+ # (set UBCAL_RETRIEVAL_MODE to compare modes)
160
+ question = " ".join(sys.argv[1:]) or "prerequisites for CPSC 210"
161
+ for r in retrieve(question):
162
+ print(f"[{r.score:.3f}] (id={r.id}) {r.recipe['title']}")
src/router.py ADDED
@@ -0,0 +1,38 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """
2
+ Router — the "adaptive" in adaptive RAG (Phase 4).
3
+
4
+ One small function decides how much machinery a question deserves:
5
+
6
+ simple a single fact or page lookup -> the fast Phase-2 retriever
7
+ complex needs several pages, a -> the agentic loop (decompose ->
8
+ comparison, or multi-step reasoning multi-hop retrieve -> synthesize)
9
+
10
+ The classification is a cheap zero-shot LLM call reusing the generation backend, so there
11
+ is no extra model to configure. Anything the model returns other than a clear "complex"
12
+ falls back to "simple" — the cheaper path is the safe default, so a misparse never sends a
13
+ simple question down the expensive agent.
14
+ """
15
+
16
+ from __future__ import annotations
17
+
18
+ from . import generate, trace
19
+
20
+ CLASSIFY_SYSTEM = (
21
+ "You are a query router for a UBC academic-calendar Q&A system. Classify the user's "
22
+ "question as either 'simple' or 'complex'.\n"
23
+ " simple — answerable from one calendar page (a single course's prerequisites or "
24
+ "description, one program's requirements, one policy or date).\n"
25
+ " complex — needs multiple courses or programs, a comparison (between programs, "
26
+ "student cohorts, or calendar editions), or a prerequisite/requirement chain "
27
+ "(e.g. 'compare X and Y', or 'what do I need to take before Z').\n"
28
+ "Reply with exactly one word: simple or complex."
29
+ )
30
+
31
+
32
+ def classify(question: str) -> str:
33
+ """Return "simple" or "complex" for `question`. Defaults to "simple" on any surprise."""
34
+ trace.step("ROUTE - classify (simple vs complex)")
35
+ reply = generate.complete(CLASSIFY_SYSTEM, question).strip().lower()
36
+ route = "complex" if "complex" in reply else "simple"
37
+ trace.detail("decision", route)
38
+ return route
src/trace.py ADDED
@@ -0,0 +1,73 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """
2
+ Trace — optional, human-readable logging of every pipeline step and API prompt.
3
+
4
+ Off by default: `enable()` flips a module-global, so normal CLI runs and the eval loop stay
5
+ silent and behave exactly as before. When on (via `ask.py --trace`), each stage prints what
6
+ it is doing and the EXACT (system, user) prompt sent to the LLM, so the whole
7
+ route -> retrieve -> generate loop is visible for learning and debugging.
8
+
9
+ Output is plain ASCII to stdout so it renders in any terminal and interleaves in order with
10
+ the CLI's own prints.
11
+ """
12
+
13
+ from __future__ import annotations
14
+
15
+ import sys
16
+
17
+ _enabled = False
18
+
19
+
20
+ def enable() -> None:
21
+ """Turn tracing on for the rest of this process."""
22
+ global _enabled
23
+ _enabled = True
24
+ # Traces print full excerpt context and model replies, which often contain non-ASCII
25
+ # (degree signs, fractions, accents). Switch stdout to UTF-8 so a cp1252 Windows console
26
+ # doesn't raise UnicodeEncodeError; errors="replace" is a last-resort guard.
27
+ try:
28
+ sys.stdout.reconfigure(encoding="utf-8", errors="replace")
29
+ except Exception:
30
+ pass
31
+
32
+
33
+ def enabled() -> bool:
34
+ return _enabled
35
+
36
+
37
+ def step(title: str) -> None:
38
+ """A top-level stage banner."""
39
+ if _enabled:
40
+ print(f"\n=== {title} ===")
41
+
42
+
43
+ def detail(label: str, value: object = "") -> None:
44
+ """One indented `label: value` line (or just a label if value is empty)."""
45
+ if _enabled:
46
+ print(f" {label}: {value}" if value != "" else f" {label}")
47
+
48
+
49
+ def results(items) -> None:
50
+ """Print a list of retrieve.Result rows as [score] (id) title."""
51
+ if _enabled:
52
+ for r in items:
53
+ print(f" [{r.score:.3f}] (id={r.id}) {r.recipe['title']}")
54
+
55
+
56
+ def prompt(system: str, user: str, model: str) -> None:
57
+ """Show one LLM API call: the system prompt and the user turn sent to `model`."""
58
+ if _enabled:
59
+ print(f" --- API call -> {model} ---")
60
+ _block("SYSTEM", system)
61
+ _block("USER", user)
62
+
63
+
64
+ def response(text: str) -> None:
65
+ """Show the model's reply for the call printed by `prompt`."""
66
+ if _enabled:
67
+ _block("RESPONSE", text)
68
+
69
+
70
+ def _block(label: str, text: str) -> None:
71
+ print(f" [{label}]")
72
+ for line in (text.splitlines() or [""]):
73
+ print(f" {line}")