Fix grounding: qwen2.5:1.5b + embedding re-rank + topic-relevance prompt

The 0.5b model hallucinated on questions whose answer isn't the top BM25 hit
(e.g. invented a "car importation tax" figure, parroted excerpt headers). Fixes:
- Re-enable embedding re-rank (auto) — bridges lexical gaps (car tax -> excise/
customs); chunks pre-warmed in a background thread so queries stay fast.
- qwen2.5:1.5b: grounds reliably where 1b/0.5b hallucinated (e.g. deficit).
- Prompt now requires same-subject excerpts and to decline (not invent) when the
topic isn't covered. Verified: car tax correctly declines; maternity/revenue/
deficit/tobacco/motor-vehicle measures all correct.
- Restore recall (top_k 4, budget 4000, num_ctx 4096); EMBED_MODEL pulled at boot.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Dockerfile

@@ -20,9 +20,9 @@ ENV HOME=/home/user \
     HOST=0.0.0.0 \
     PORT=7860 \
     OLLAMA_URL=http://localhost:11434 \
-    OLLAMA_MODEL=qwen2.5:0.5b \
+    OLLAMA_MODEL=qwen2.5:1.5b \
+    EMBED_MODEL=nomic-embed-text \
     INDEX_PATH=/tmp/mru_budget_index.parquet \
-    RERANK=off \
     OMP_NUM_THREADS=2 \
     OLLAMA_NUM_PARALLEL=1
 

README.md

@@ -34,7 +34,7 @@ embeddings** for retrieval, **Ollama** for the LLM, and **Gradio** for the UI.
 Requires [Ollama](https://ollama.com) running on your machine.
 
 ```bash
-ollama pull qwen2.5:0.5b          # the answer model (any chat model works)
+ollama pull qwen2.5:1.5b          # the answer model (any chat model works)
 ollama pull nomic-embed-text      # optional: enables embedding re-rank
 pip install -r requirements.txt
 python main.py                    # serves http://127.0.0.1:7860

app/ollama_client.py

@@ -38,12 +38,16 @@ def _extra() -> dict:
 
 _SYSTEM = (
     "You are a precise analyst of the Republic of Mauritius National Budget 2026-2027. "
-    "Answer the user's question using ONLY the budget excerpts in the latest message. "
-    "Each excerpt is prefixed with a number like [1]; cite the ones you use inline, e.g. [1]. "
-    "Monetary figures are in Mauritian Rupees (Rs) — quote them exactly with their unit "
-    "(million/billion). Be concise and direct: lead with the answer, no preamble. "
+    "Answer the user's question using ONLY the numbered budget excerpts in the latest message. "
+    "Before using an excerpt, check it is about the SAME subject as the question — e.g. do NOT answer "
+    "a question about cars or vehicles using an excerpt about plastic bottles or sugar. "
+    "Cite the excerpts you actually use inline, e.g. [1]. Monetary figures are in Mauritian Rupees "
+    "(Rs) — quote them exactly with their unit (million/billion). Be concise and direct: lead with "
+    "the answer, no preamble, and do NOT repeat the excerpt headers (the 'Document: …' lines). "
     "Use earlier conversation turns to resolve follow-up questions (e.g. 'how does it change' "
-    "refers to the topic just discussed). If the excerpts do not contain the answer, say so plainly."
+    "refers to the topic just discussed). If no excerpt specifically addresses the asked subject, "
+    "reply that the budget documents provided do not appear to address it — NEVER invent, guess or "
+    "estimate a figure."
 )
 
 

app/rag.py

@@ -13,7 +13,7 @@ from typing import Iterator
 from app.config import CONFIG
 from app.ingest import build_index, corpus_summary
 from app.ollama_client import chat_stream, pick_model, warm_model
-from app.retrieval import build_context
+from app.retrieval import build_context, prewarm_embeddings
 
 log = logging.getLogger("rag")
 
@@ -31,6 +31,8 @@ def init() -> None:
     _STATE["summary"] = corpus_summary(df)
     if CONFIG["ollama"].get("warm_on_start", True):
         threading.Thread(target=warm_model, daemon=True).start()
+    # Embed all chunks in the background so re-ranked queries are fast.
+    threading.Thread(target=prewarm_embeddings, args=(df,), daemon=True).start()
 
 
 def summary() -> list[dict]:

app/retrieval.py

@@ -65,6 +65,22 @@ def _embed_docs(texts: list, model: str) -> list:
     return [_emb_store[k] for k in keys]
 
 
+def prewarm_embeddings(df: pd.DataFrame) -> None:
+    """Embed every chunk up front (in batches) so the first re-ranked queries
+    don't pay a per-chunk embedding cost. No-op when re-rank is disabled."""
+    if not _embeddings_enabled():
+        return
+    model = _R.get("embed_model", "nomic-embed-text")
+    texts = [(r["text"][:1500] or "_") for r in df.to_dict("records")]
+    B = 64
+    try:
+        for i in range(0, len(texts), B):
+            _embed_docs(texts[i:i + B], model)
+        log.info("prewarmed %d chunk embeddings", len(texts))
+    except Exception as e:
+        log.warning("embedding prewarm stopped early: %s", e)
+
+
 def _rerank(question: str, rows: list, idxs: list) -> list:
     import numpy as np
 

config.yaml

@@ -15,15 +15,15 @@ ollama:
   model: null              # null = first installed model
   request_timeout: 300
   keep_alive: 30m          # keep the model resident between questions
-  num_ctx: 2048            # context window: smaller = much faster prefill on CPU
+  num_ctx: 4096            # context window: fits a few re-ranked excerpts + answer
   num_predict: 400         # cap answer tokens so long answers don't stall on CPU
   temperature: 0           # greedy decoding: far more reliable for factual extraction
   warm_on_start: true      # preload the model at startup so the first answer is fast
   history_turns: 4         # prior chat messages fed back for follow-up questions
 
 retrieval:
-  top_k: 3
-  context_budget: 2600     # chars of excerpt context handed to the model (smaller = faster prefill on CPU)
-  rerank: off              # embedding re-rank of BM25 hits: auto | on | off (off = no extra round-trip)
+  top_k: 4
+  context_budget: 4000     # chars of excerpt context handed to the model
+  rerank: auto             # embedding re-rank of BM25 hits: auto | on | off (auto = on when embed model installed)
   embed_model: nomic-embed-text  # Ollama embedding model (auto-used only if installed)
   rerank_k: 30             # BM25 candidates to re-rank by embedding similarity