Step 1: scoring engine, improvement-prompt generator, API, 44 green tests

- Scorer adapter interface + LocalScorer (judge mode plugs in later)
- taxonomy: task type -> research-grounded parameters with default weights
- parameter evaluators with quoted evidence and honesty flags
- flagship generator: failure mode x task x model family, complete
paste-ready replacement prompts, no-generic-advice contract under test
- FastAPI app (/evaluate, /health, /warmup)
- curated planted-failure test pairs; classifier/constraint/regex fixes
found by the suite; question coverage reframed as honest topical proxy
after empirical probe showed bi-encoders cannot detect answerhood

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

CLAUDE.md

@@ -74,6 +74,7 @@ model), and the entire improvement-prompt template library.
 | No ROUGE/BLEU | n-gram metrics excluded everywhere | SummEval showed weak human correlation; embedding + NLI methods used instead. |
 | No code execution in v1 | static analysis ladder for code tasks | Executing arbitrary pasted code is a safety/dependency problem; HumanEval-style pass@k logged as judge-mode-era future work. |
 | No toxicity metric in v1 | deferred | Lexicon approaches false-positive heavily; a proper local classifier is future work. |
+| Question coverage is a topical proxy | flag a question part only when its topic/keywords are absent from the output | Empirically verified (2026-07-05) that bi-encoder similarity cannot distinguish "mentioned" from "answered" (unanswered part scored 0.65 vs answered 0.59). Labeled is_proxy, limitation documented in a dedicated test; true answerhood is judge-mode work. |
 | Scoring adapter | `Scorer` interface with `LocalScorer` now; `LLMJudgeScorer` later (BYO-key free tier, our-key paid tier) | Required by brief. Interface lands in Step 1. |
 | Frontend | Decided at Step 2 (leaning React/Next.js + Vercel) | Not needed before Checkpoint 2. |
 | Document evaluation | Deferred decision until the text pipeline exists | Include only if nearly free per brief. |
@@ -122,9 +123,16 @@ Work autonomously between checkpoints; stop and wait at each one.
   CLAUDE.md created, names proposed.
 - 2026-07-05 — **Checkpoint 1 passed**: name = Rubriq. Folder rename deferred
   by user (still `LLM Quality` on disk — harmless, git doesn't care). Repo
-  `devadit1515/rubriq` created and pushed. **Step 1 underway**: research →
-  RESEARCH.md → classifier → taxonomy → scoring engine → prompt generator →
-  tests. Next stop: Checkpoint 2 (user review + Supabase credentials).
+  `devadit1515/rubriq` created and pushed.
+- 2026-07-05 — **Step 1 complete**: RESEARCH.md (all sources logged),
+  classifier (rules + bge prototypes + source split), IFEval-style constraint
+  engine, metric modules (SummaC-ZS faithfulness, relevance, readability,
+  lexical, hallucination signals, static code ladder), Scorer adapter +
+  LocalScorer, improvement-prompt generator (failure × task × model family),
+  FastAPI app, 44 tests green on curated planted-failure pairs. Demo pair
+  verified end-to-end: sabotaged summary scores 33/100 with all four planted
+  fabrications quoted. **Waiting at Checkpoint 2**: user reviews, sets up
+  Supabase, hands over credentials; Step 2 (frontend + deploy) after that.
 
 ## Future work log
 

README.md

@@ -0,0 +1,63 @@
+# Rubriq
+
+Paste the prompt you gave an LLM and the output you got back. Rubriq scores
+the output against evaluation parameters drawn from published research,
+quotes the evidence for every finding, and generates ready-to-paste
+replacement prompts that target the specific failures it found — adapted to
+the model you name (GPT, Claude, Gemini, or open-weights).
+
+Everything runs locally. No API calls, no keys, no data leaving your machine.
+
+## How it works
+
+1. **Classify** — rule signals plus embedding prototypes decide the task
+   type (summarization, code generation, extraction, Q&A, creative writing,
+   reasoning, rewriting) and split pasted source material from the
+   instruction.
+2. **Score** — each task type gets the parameters the literature assigns it:
+   NLI-based faithfulness when source text exists (SummaC method),
+   deterministic instruction-following checks (IFEval method), embedding
+   relevance (RAGAS-style), readability against the stated audience,
+   hallucination-risk signals, static code analysis, style statistics.
+3. **Report** — per-parameter verdicts with quotes from the output itself.
+   Parameters that local methods cannot judge say so instead of guessing:
+   open-world fact checking and subjective quality are labeled as
+   judge-mode territory, and proxies are labeled proxies.
+4. **Repair** — the flagship. Every diagnosed failure generates a complete
+   replacement prompt using the named model family's own documented
+   mechanisms (XML tags and quote-first grounding for Claude, delimiters and
+   literal phrasing for GPT, few-shot and positive instructions for Gemini,
+   constraint repetition for open-weights models).
+
+Every parameter and template traces to a named source — see
+[RESEARCH.md](RESEARCH.md).
+
+## Run it
+
+```bash
+python -m venv .venv
+.venv/Scripts/pip install -e ".[dev]"     # first run downloads two small models (~350 MB)
+.venv/Scripts/uvicorn rubriq.api.main:app --reload
+```
+
+`POST /evaluate` with `{"prompt": ..., "output": ..., "model_name": "gpt-4o"}`
+returns the full report. `GET /health` shows engine and model status.
+
+Models: `BAAI/bge-small-en-v1.5` (embeddings) and
+`cross-encoder/nli-deberta-v3-small` (entailment). Both run CPU-only.
+
+## Tests
+
+```bash
+.venv/Scripts/python -m pytest tests -q
+```
+
+The suite includes curated prompt/output pairs with known planted failures
+(invented facts, broken code, fabricated citations, ignored constraints) and
+asserts Rubriq finds each one — plus a documented-limitation test that keeps
+the honest boundary of the coverage proxy visible.
+
+## Status
+
+Step 1 (evaluation engine) complete. Frontend and deployment are Step 2.
+An LLM-as-judge engine plugs in behind the same `Scorer` interface later.

rubriq/api/main.py

@@ -0,0 +1,51 @@
+"""FastAPI surface. One evaluation endpoint; the engine behind it is chosen
+by the Scorer adapter, so judge mode later is a query param, not a rewrite.
+
+Run:  uvicorn rubriq.api.main:app --reload
+"""
+
+from __future__ import annotations
+
+from fastapi import FastAPI, HTTPException
+from fastapi.middleware.cors import CORSMiddleware
+
+from .. import models
+from ..schemas import EvalReport, EvalRequest
+from ..scoring import LocalScorer, Scorer
+
+app = FastAPI(
+    title="Rubriq",
+    version="0.1.0",
+    description="Local, research-grounded LLM output quality evaluation.",
+)
+
+app.add_middleware(
+    CORSMiddleware,
+    allow_origins=["*"],        # tightened at deployment (Step 2)
+    allow_methods=["*"],
+    allow_headers=["*"],
+)
+
+_SCORERS: dict[str, Scorer] = {"local": LocalScorer()}
+# LLMJudgeScorer registers here later: {"judge-byok": ..., "judge-managed": ...}
+
+
+@app.post("/evaluate", response_model=EvalReport)
+def evaluate(request: EvalRequest, engine: str = "local") -> EvalReport:
+    scorer = _SCORERS.get(engine)
+    if scorer is None:
+        raise HTTPException(status_code=400, detail=f"unknown engine '{engine}'; available: {list(_SCORERS)}")
+    if not request.prompt.strip() or not request.output.strip():
+        raise HTTPException(status_code=422, detail="both prompt and output are required")
+    return scorer.evaluate(request)
+
+
+@app.get("/health")
+def health() -> dict:
+    return {"status": "ok", "engines": list(_SCORERS), "models": models.availability()}
+
+
+@app.post("/warmup")
+def warmup() -> dict:
+    """Load both local models so the first evaluation doesn't pay the cost."""
+    return models.availability()

rubriq/classify.py

@@ -28,7 +28,8 @@ _DELIM_BLOCK = re.compile(
 )
 _SOURCE_INTRO = re.compile(
     r"(?i)(?:following (?:text|article|document|passage|email|report|code|excerpt|transcript|paragraph)s?|"
-    r"text below|below is|here is the (?:text|article|document|passage|code)|this (?:text|article|document))\s*[:.]?\s*\n"
+    r"text below|below is|here is the (?:text|article|document|passage|code)|this (?:text|article|document))"
+    r"[^\n]*\n"
 )
 
 
@@ -53,14 +54,14 @@ def split_instruction_and_source(prompt: str) -> tuple[str, str | None]:
     if m:
         source = prompt[m.end():].strip()
         instruction = prompt[:m.end()].strip()
-        if word_count(source) >= 40:
+        if word_count(source) >= 30:
             return instruction, source
 
     # Length heuristic: short first paragraph, long remainder.
     parts = prompt.split("\n", 1)
     if len(parts) == 2:
         head, rest = parts[0].strip(), parts[1].strip()
-        if head and word_count(head) <= 60 and word_count(rest) >= 150 and _looks_instructional(head):
+        if head and word_count(head) <= 60 and word_count(rest) >= 60 and _looks_instructional(head):
             return head, rest
 
     return prompt.strip(), None
@@ -95,6 +96,10 @@ _RULES: list[tuple[TaskType, str, re.Pattern]] = [
      re.compile(r"(?i)\b(rewrite|rephrase|paraphrase|edit|proofread|improve|polish|make (?:this|it) (?:more|less|sound)|change the tone|translate)\b")),
     (TaskType.QA_OPEN, "prompt asks a direct question",
      re.compile(r"(?i)^(?:please\s+)?(what|who|when|where|why|how|which|is|are|does|do|can|could|should|explain)\b")),
+    (TaskType.QA_OPEN, "prompt asks to answer questions",
+     re.compile(r"(?i)\banswer (?:the |these |the following |my )?questions?\b")),
+    (TaskType.QA_OPEN, "prompt contains interrogative sentences",
+     re.compile(r"(?:^|[.!?]\s+)(?:What|Who|When|Where|Why|How|Which)\b[^.\n?]*\?")),
 ]
 
 _CODE_FENCE = re.compile(r"```")

rubriq/constraints.py

@@ -80,7 +80,7 @@ _INCLUDE_PAT = re.compile(
 )
 _EXCLUDE_PAT = re.compile(
     r"(?i)\b(?:do not|don'?t|never|avoid)\s+"
-    r"(?:use|mention|include|say)(?:\s+the\s+(?:word|term|phrase)s?)?\s+[\"“']([^\"”']{2,60})[\"”']"
+    r"(?:us(?:e|ing)|mention(?:ing)?|includ(?:e|ing)|say(?:ing)?)(?:\s+the\s+(?:word|term|phrase)s?)?\s+[\"“']([^\"”']{2,60})[\"”']"
 )
 _START_PAT = re.compile(r"(?i)\b(?:begin|start)\s+(?:your\s+)?(?:response|answer|output|reply|with)?\s*with\s+[\"“']([^\"”']{2,80})[\"”']")
 _END_PAT = re.compile(r"(?i)\bend\s+(?:your\s+)?(?:response|answer|output|reply)?\s*with\s+[\"“']([^\"”']{2,80})[\"”']")

rubriq/metrics/hallucination_risk.py

@@ -17,7 +17,7 @@ from ..textutils import snippet
 
 _SIGNALS: list[tuple[str, re.Pattern, str]] = [
     ("citation_pattern",
-     re.compile(r"\([A-Z][A-Za-z-]+(?: (?:et al\.?|&|and) [A-Z][A-Za-z-]+)?,? (?:19|20)\d{2}\)|\[[0-9]{1,3}\]"),
+     re.compile(r"\([A-Z][A-Za-z-]+(?:\s+et al\.?|\s+(?:&|and)\s+[A-Z][A-Za-z-]+)?,?\s+(?:19|20)\d{2}\)|\[[0-9]{1,3}\]"),
      "citation-shaped reference — LLMs fabricate plausible-looking citations; verify each one exists"),
     ("case_or_statute",
      re.compile(r"\b[A-Z][a-z]+ v\.? [A-Z][a-z]+\b|\b\d+ U\.S\.C?\.? §? ?\d+\b"),

rubriq/prompts/generator.py

@@ -0,0 +1,454 @@
+"""Improvement-prompt generation — the flagship.
+
+Contract (RESEARCH.md Part 4, enforced by tests):
+(a) the diagnosis names the specific failure with the evidence that triggered it,
+(b) prompt_text is a complete, ready-to-paste replacement prompt,
+(c) the fix uses the named model family's documented mechanism.
+"Be more specific" is a bug, not a template.
+
+Builders receive the user's original instruction and rebuild it with the
+repair applied, so the user pastes one block, not advice.
+"""
+
+from __future__ import annotations
+
+from ..schemas import (Evidence, FailureMode, ImprovementPrompt, ModelFamily,
+                       ParameterScore, TaskClassification, TaskType)
+from ..textutils import snippet
+from .model_profiles import Profile, get_profile
+
+_SEVERITY = [
+    FailureMode.UNFAITHFUL_CONTENT,
+    FailureMode.INVALID_CODE,
+    FailureMode.INVENTED_FIELDS,
+    FailureMode.MISSED_CONSTRAINT,
+    FailureMode.FORMAT_VIOLATION,
+    FailureMode.INCOMPLETE_COVERAGE,
+    FailureMode.HALLUCINATION_RISK,
+    FailureMode.OFF_TOPIC_DRIFT,
+    FailureMode.WEAK_REASONING,
+    FailureMode.VERBOSITY,
+    FailureMode.WRONG_REGISTER,
+    FailureMode.POOR_STRUCTURE,
+    FailureMode.LOW_DIVERSITY,
+]
+
+MAX_PROMPTS = 5
+
+
+def _quotes(findings, limit=3) -> list[str]:
+    out = []
+    for f in findings:
+        for e in f.evidence:
+            if e.source == "output" and e.quote:
+                out.append(e.quote)
+                break
+        if len(out) >= limit:
+            break
+    return out
+
+
+def _source_block(profile: Profile, task: TaskClassification) -> str:
+    if not task.has_source_text:
+        return ""
+    return profile.wrap_source() + "\n\n"
+
+
+def _rules_block(profile: Profile, rules: list[str]) -> str:
+    """Render hard rules in the family's preferred structure."""
+    if profile.family == ModelFamily.CLAUDE:
+        inner = "\n".join(f"- {r}" for r in rules)
+        return f"<rules>\n{inner}\n</rules>"
+    if profile.family == ModelFamily.GPT:
+        inner = "\n".join(f"{i+1}. {r}" for i, r in enumerate(rules))
+        return f"### Requirements\n{inner}"
+    inner = "\n".join(f"- {r}" for r in rules)
+    return f"Rules:\n{inner}"
+
+
+def _tail_emphasis(profile: Profile, critical: str) -> str:
+    """GPT/open-weights/generic get the critical constraint repeated last
+    (lost-in-the-middle, RESEARCH.md 2.7). Claude/Gemini get their own levers
+    elsewhere, so no noisy repetition."""
+    if profile.family in (ModelFamily.OPEN_WEIGHTS, ModelFamily.GPT, ModelFamily.GENERIC):
+        return f"\n\nIMPORTANT — before finishing, verify: {critical}"
+    return ""
+
+
+# --------------------------------------------------------------- builders
+
+def _build_unfaithful(findings, task, profile: Profile) -> ImprovementPrompt:
+    bad = _quotes(findings)
+    bad_list = "\n".join(f'- "{q}"' for q in bad) if bad else "- (see report)"
+    grounding_rules = [
+        "Use ONLY information stated in the source text. If the source does not say it, do not write it.",
+        "If something the task needs is missing from the source, write \"the source does not specify\" rather than filling the gap.",
+    ]
+    if profile.family == ModelFamily.CLAUDE:
+        pre = ("First, copy the exact sentences from the source that support your answer into <quotes> tags. "
+               "Then write the final answer in <answer> tags using only what you quoted.\n\n")
+    elif profile.family == ModelFamily.GEMINI:
+        pre = "Base every sentence of your answer on the text above. Quote or closely paraphrase the source.\n\n"
+    else:
+        pre = "Answer using only the provided text. Quote the source where possible.\n\n"
+    body = (
+        f"{_source_block(profile, task)}"
+        f"{task.instruction.strip()}\n\n"
+        f"{pre}"
+        f"{_rules_block(profile, grounding_rules)}\n\n"
+        f"A previous attempt invented content the source does not support, including:\n{bad_list}\n"
+        f"Do not repeat these claims unless the source states them."
+        f"{_tail_emphasis(profile, 'every claim traces to a sentence in the source text')}"
+    )
+    return ImprovementPrompt(
+        failure_mode=FailureMode.UNFAITHFUL_CONTENT,
+        title="Pin the output to the source",
+        diagnosis=(f"The output makes claims the source text does not support"
+                   + (f', e.g. "{bad[0]}"' if bad else "") + ". "
+                   f"Repair: {profile.grounding_mechanism}"),
+        prompt_text=body,
+        model_family=profile.family,
+        technique_source=profile.source,
+    )
+
+
+def _build_constraints(findings, task, profile: Profile) -> ImprovementPrompt:
+    missed = []
+    for f in findings:
+        kind = f.data.get("kind", "")
+        actual = f.data.get("actual", "")
+        missed.append((f.detail, kind, actual))
+    rules = []
+    for f in findings:
+        prompt_ev = next((e.quote for e in f.evidence if e.source == "prompt"), None)
+        rules.append(f.detail.split(" — ")[0].removeprefix("Prompt asked for ").strip().capitalize()
+                     + (f" (your prompt said: \"{prompt_ev}\")" if prompt_ev else ""))
+    if profile.family == ModelFamily.GEMINI:
+        example = ("\n\nHere is an example of a response with the right shape (replace the content, keep the form):\n"
+                   "[write one short example that satisfies every rule above]")
+    else:
+        example = ""
+    critical = rules[0] if rules else "all stated constraints are met"
+    body = (
+        f"{_source_block(profile, task)}"
+        f"{task.instruction.strip()}\n\n"
+        f"{_rules_block(profile, rules)}"
+        f"{example}"
+        f"\n\nAfter drafting, count and check each requirement above before giving your final answer."
+        f"{_tail_emphasis(profile, critical)}"
+    )
+    first = findings[0]
+    return ImprovementPrompt(
+        failure_mode=first.failure_mode,
+        title="Make the constraints unmissable",
+        diagnosis=(f"The output missed {len(findings)} verifiable constraint(s): "
+                   + "; ".join(f.detail for f in findings[:3]) + " "
+                   f"Repair: {profile.format_mechanism}"),
+        prompt_text=body,
+        model_family=profile.family,
+        technique_source=profile.source,
+    )
+
+
+def _build_coverage(findings, task, profile: Profile) -> ImprovementPrompt:
+    parts = []
+    for f in findings:
+        q = next((e.quote for e in f.evidence if e.source == "prompt"), None)
+        if q:
+            parts.append(q)
+    part_list = "\n".join(f"{i+1}. {p}" for i, p in enumerate(parts)) or "1. (see report)"
+    body = (
+        f"{_source_block(profile, task)}"
+        f"{task.instruction.strip()}\n\n"
+        f"Answer every numbered part separately, under its own heading:\n{part_list}\n\n"
+        f"If you cannot answer a part, say so under that heading rather than skipping it."
+        f"{_tail_emphasis(profile, f'all {len(parts)} parts have their own answer')}"
+    )
+    return ImprovementPrompt(
+        failure_mode=FailureMode.INCOMPLETE_COVERAGE,
+        title="Force full coverage of the question",
+        diagnosis=(f"{len(parts)} part(s) of the request went unanswered, e.g. \"{parts[0] if parts else ''}\". "
+                   "Repair: enumerate the parts and demand one answer per part — models skip sub-questions "
+                   "buried in prose but not numbered lists."),
+        prompt_text=body,
+        model_family=profile.family,
+        technique_source=profile.source,
+    )
+
+
+def _build_hallucination(findings, task, profile: Profile) -> ImprovementPrompt:
+    flagged = _quotes(findings)
+    fl = "\n".join(f'- "{q}"' for q in flagged) if flagged else ""
+    rules = [
+        "For every statistic, name its source inline, or present it as an estimate in words (\"roughly\", \"on the order of\").",
+        "Do not cite papers, cases, or URLs unless you are certain they exist; prefer \"research on X suggests\" with no fake citation.",
+        "Mark claims you are unsure about explicitly (\"I may be wrong about the exact figure\").",
+    ]
+    body = (
+        f"{task.instruction.strip()}\n\n"
+        f"{_rules_block(profile, rules)}\n\n"
+        + (f"A previous attempt stated the following as fact; each is unverifiable as written:\n{fl}\n"
+           f"Restate these with sources or with honest uncertainty, or drop them.\n" if fl else "")
+        + "\nEnd your answer with a one-line note: which claims above should I double-check independently?"
+        f"{_tail_emphasis(profile, 'no statistic or citation appears without a named source or an uncertainty marker')}"
+    )
+    return ImprovementPrompt(
+        failure_mode=FailureMode.HALLUCINATION_RISK,
+        title="Demand attribution or honest uncertainty",
+        diagnosis=(f"The output states unverifiable specifics as fact"
+                   + (f', e.g. "{flagged[0]}"' if flagged else "") + ". "
+                   "No local check can verify these; the repair makes the model attribute or hedge them, "
+                   "and hands you a verification checklist."),
+        prompt_text=body,
+        model_family=profile.family,
+        technique_source=profile.source,
+    )
+
+
+def _build_reasoning(findings, task, profile: Profile) -> ImprovementPrompt:
+    if profile.family == ModelFamily.CLAUDE:
+        think = ("Work through the problem step by step inside <thinking> tags first. "
+                 "Then give the final answer in <answer> tags.")
+    else:
+        think = ("Solve this step by step. Show each step of your working on its own line, "
+                 "then state the final answer on a last line beginning \"Answer:\".")
+    body = (
+        f"{task.instruction.strip()}\n\n"
+        f"{think}\n\n"
+        f"After stating the answer, re-derive it a second way if possible and confirm both routes agree; "
+        f"if they disagree, say which one you trust and why."
+        f"{_tail_emphasis(profile, 'working shown, final answer explicitly labeled')}"
+    )
+    return ImprovementPrompt(
+        failure_mode=FailureMode.WEAK_REASONING,
+        title="Make the model show its working",
+        diagnosis=("The output gives a conclusion without checkable working. "
+                   "Repair: request the reasoning *before* the answer (chain-of-thought), plus a second "
+                   "independent derivation — the manual form of self-consistency."),
+        prompt_text=body,
+        model_family=profile.family,
+        technique_source="Wei et al. 2022 (CoT); Wang et al. 2023 (self-consistency); " + profile.source,
+    )
+
+
+def _build_verbosity(findings, task, profile: Profile) -> ImprovementPrompt:
+    fillers = _quotes(findings)
+    limit_hint = next((f.data.get("ratio") for f in findings if "ratio" in f.data), None)
+    rules = [
+        "Open with the answer itself — no preamble, no restating the question.",
+        "Cut framing phrases (e.g. " + ", ".join(f'"{f}"' for f in fillers[:3]) + ")." if fillers
+        else "Cut framing phrases and meta-commentary.",
+        "One idea per sentence; delete any sentence that repeats an earlier one.",
+    ]
+    if profile.family == ModelFamily.CLAUDE:
+        opener = "\nBegin your response directly with the substance (no introduction)."
+    else:
+        opener = ""
+    body = (
+        f"{_source_block(profile, task)}"
+        f"{task.instruction.strip()}\n\n"
+        f"{_rules_block(profile, rules)}{opener}"
+        f"{_tail_emphasis(profile, 'the response starts with substance and contains no filler phrases')}"
+    )
+    return ImprovementPrompt(
+        failure_mode=FailureMode.VERBOSITY,
+        title="Strip the padding",
+        diagnosis=("The output pads its length" + (f" ({limit_hint}x the asked-for size)" if limit_hint else "")
+                   + (f' with filler like "{fillers[0]}"' if fillers else "") + ". "
+                   "Repair: answer-first structure plus an explicit ban on the exact filler found."),
+        prompt_text=body,
+        model_family=profile.family,
+        technique_source=profile.source,
+    )
+
+
+def _build_register(findings, task, profile: Profile) -> ImprovementPrompt:
+    data = findings[0].data
+    fk, band = data.get("fk_grade"), data.get("band", [6, 14])
+    audience = data.get("audience", "the intended audience")
+    too_hard = fk is not None and fk > band[1]
+    style = ("short sentences, everyday words, one idea at a time"
+             if too_hard else "precise terminology and compact, information-dense prose")
+    role = f"You are writing for {audience}."
+    body = (
+        f"{role}\n\n"
+        f"{_source_block(profile, task)}"
+        f"{task.instruction.strip()}\n\n"
+        f"Write in a register that fits that audience: {style}. "
+        f"Read your draft as that reader before finalizing; rewrite any sentence they would stumble on."
+        f"{_tail_emphasis(profile, f'the text reads naturally for {audience}')}"
+    )
+    return ImprovementPrompt(
+        failure_mode=FailureMode.WRONG_REGISTER,
+        title="Match the audience's reading level",
+        diagnosis=(f"The output reads at US grade level {fk}, outside the {band[0]:g}-{band[1]:g} band for {audience} — "
+                   f"{'harder' if too_hard else 'simpler'} than it should. "
+                   "Repair: an explicit audience role. Role prompting reliably shifts register (it does not fix facts)."),
+        prompt_text=body,
+        model_family=profile.family,
+        technique_source="role prompting for register (RESEARCH.md 2.8); " + profile.source,
+    )
+
+
+def _build_structure(findings, task, profile: Profile) -> ImprovementPrompt:
+    repeats = any("repeat" in f.detail.lower() or "duplicate" in f.detail.lower() for f in findings)
+    rules = [
+        "Before writing, produce a 3-6 point outline; then write one section per point.",
+        "Each section must add new information — no restating earlier sections." if repeats
+        else "Order sections so each follows from the previous one.",
+    ]
+    body = (
+        f"{_source_block(profile, task)}"
+        f"{task.instruction.strip()}\n\n"
+        f"{_rules_block(profile, rules)}"
+        f"{_tail_emphasis(profile, 'sections follow the outline and none repeats another')}"
+    )
+    return ImprovementPrompt(
+        failure_mode=FailureMode.POOR_STRUCTURE,
+        title="Impose an outline",
+        diagnosis=("Adjacent sections " + ("repeat each other" if repeats else "jump between topics without connection")
+                   + " (flagged by flow analysis). Repair: outline-first generation pins the structure before prose exists."),
+        prompt_text=body,
+        model_family=profile.family,
+        technique_source=profile.source,
+    )
+
+
+def _build_drift(findings, task, profile: Profile) -> ImprovementPrompt:
+    drifting = _quotes(findings, 1)
+    body = (
+        f"{_source_block(profile, task)}"
+        f"{task.instruction.strip()}\n\n"
+        f"Stay strictly on this request. Do not add background, related topics, or general advice unless asked."
+        + (f"\n\nA previous attempt drifted into: \"{drifting[0]}\" — leave this out unless the request explicitly needs it."
+           if drifting else "")
+        + f"{_tail_emphasis(profile, 'every paragraph directly serves the request')}"
+    )
+    return ImprovementPrompt(
+        failure_mode=FailureMode.OFF_TOPIC_DRIFT,
+        title="Fence the scope",
+        diagnosis=(f"Part of the output drifts off the request"
+                   + (f' ("{drifting[0]}")' if drifting else "") +
+                   ". Repair: an explicit scope fence naming the drift to exclude."),
+        prompt_text=body,
+        model_family=profile.family,
+        technique_source=profile.source,
+    )
+
+
+def _build_code(findings, task, profile: Profile) -> ImprovementPrompt:
+    problems = [f.detail for f in findings]
+    plist = "\n".join(f"- {p}" for p in problems[:4])
+    rules = [
+        "Return complete, runnable code — no TODOs, placeholders, or elided sections.",
+        "If the code needs assumptions (versions, inputs), state them in a comment at the top.",
+        "After the code, list 2-3 quick test cases I can run to verify it.",
+    ]
+    body = (
+        f"{task.instruction.strip()}\n\n"
+        f"{_rules_block(profile, rules)}\n\n"
+        f"A previous attempt had these problems — fix each:\n{plist}"
+        f"{_tail_emphasis(profile, 'the code parses, is complete, and includes the requested names')}"
+    )
+    return ImprovementPrompt(
+        failure_mode=FailureMode.INVALID_CODE,
+        title="Demand complete, verifiable code",
+        diagnosis=("Static analysis found concrete defects: " + "; ".join(problems[:2]) + " "
+                   "Repair: completeness rules plus self-supplied test cases give you an immediate way to check the redo."),
+        prompt_text=body,
+        model_family=profile.family,
+        technique_source=profile.source,
+    )
+
+
+def _build_invented(findings, task, profile: Profile) -> ImprovementPrompt:
+    invented = _quotes(findings)
+    il = "\n".join(f'- "{q}"' for q in invented) if invented else ""
+    rules = [
+        "Copy every extracted value verbatim from the source text — no paraphrasing, no normalizing.",
+        "If a requested field is not in the text, output null for it. Never fill gaps from general knowledge.",
+    ]
+    body = (
+        f"{_source_block(profile, task)}"
+        f"{task.instruction.strip()}\n\n"
+        f"{_rules_block(profile, rules)}\n\n"
+        + (f"A previous attempt output values that are not in the source:\n{il}\n" if il else "")
+        + f"{_tail_emphasis(profile, 'every value string-matches the source text or is null')}"
+    )
+    return ImprovementPrompt(
+        failure_mode=FailureMode.INVENTED_FIELDS,
+        title="Verbatim-only extraction",
+        diagnosis=("Extracted values don't appear in the source text"
+                   + (f', e.g. "{invented[0]}"' if invented else "") +
+                   ". Repair: verbatim-copy rule plus explicit null for missing fields — the two rules that stop "
+                   "extraction models from 'helpfully' completing data."),
+        prompt_text=body,
+        model_family=profile.family,
+        technique_source=profile.source,
+    )
+
+
+def _build_diversity(findings, task, profile: Profile) -> ImprovementPrompt:
+    cliches = _quotes(findings)
+    cl = ", ".join(f'"{c}"' for c in cliches[:5])
+    rules = [
+        (f"Banned phrases: {cl}. Find a fresh image instead." if cliches else
+         "Avoid stock phrases; when a familiar expression appears in your draft, replace it with a concrete image."),
+        "Vary sentence length deliberately: mix sentences under 8 words with longer ones.",
+        "Prefer specific, sensory detail over abstract description.",
+    ]
+    body = (
+        f"{task.instruction.strip()}\n\n"
+        f"{_rules_block(profile, rules)}"
+        f"{_tail_emphasis(profile, 'no banned phrase appears and sentence lengths vary')}"
+    )
+    return ImprovementPrompt(
+        failure_mode=FailureMode.LOW_DIVERSITY,
+        title="Ban the clichés it reached for",
+        diagnosis=("Style statistics flagged " + (f"stock phrases ({cl})" if cliches else "flat, repetitive prose") +
+                   ". Repair: ban the exact phrases found and constrain rhythm — negative examples work when they are this specific."),
+        prompt_text=body,
+        model_family=profile.family,
+        technique_source=profile.source,
+    )
+
+
+_BUILDERS = {
+    FailureMode.UNFAITHFUL_CONTENT: _build_unfaithful,
+    FailureMode.MISSED_CONSTRAINT: _build_constraints,
+    FailureMode.FORMAT_VIOLATION: _build_constraints,
+    FailureMode.INCOMPLETE_COVERAGE: _build_coverage,
+    FailureMode.HALLUCINATION_RISK: _build_hallucination,
+    FailureMode.WEAK_REASONING: _build_reasoning,
+    FailureMode.VERBOSITY: _build_verbosity,
+    FailureMode.WRONG_REGISTER: _build_register,
+    FailureMode.POOR_STRUCTURE: _build_structure,
+    FailureMode.OFF_TOPIC_DRIFT: _build_drift,
+    FailureMode.INVALID_CODE: _build_code,
+    FailureMode.INVENTED_FIELDS: _build_invented,
+    FailureMode.LOW_DIVERSITY: _build_diversity,
+}
+
+
+def generate_improvement_prompts(
+    model_name: str,
+    task: TaskClassification,
+    parameters: list[ParameterScore],
+) -> list[ImprovementPrompt]:
+    profile = get_profile(model_name)
+
+    by_mode: dict[FailureMode, list] = {}
+    for ps in parameters:
+        for f in ps.findings:
+            by_mode.setdefault(f.failure_mode, []).append(f)
+    # constraint + format failures repair through the same rebuilt prompt
+    if FailureMode.FORMAT_VIOLATION in by_mode and FailureMode.MISSED_CONSTRAINT in by_mode:
+        by_mode[FailureMode.MISSED_CONSTRAINT].extend(by_mode.pop(FailureMode.FORMAT_VIOLATION))
+
+    prompts: list[ImprovementPrompt] = []
+    for mode in _SEVERITY:
+        if mode in by_mode and mode in _BUILDERS:
+            prompts.append(_BUILDERS[mode](by_mode[mode], task, profile))
+        if len(prompts) >= MAX_PROMPTS:
+            break
+    return prompts

rubriq/prompts/model_profiles.py

@@ -0,0 +1,89 @@
+"""Model-family detection and prompting mechanisms (RESEARCH.md 2.1-2.3).
+
+Each family gets the *mechanisms its own provider documents*: XML tags and
+prefills for Claude, delimiters and literal phrasing for GPT, few-shot and
+positive instructions for Gemini, repetition and explicit format for open
+weights. The generic profile is the fallback when no model is named.
+"""
+
+from __future__ import annotations
+
+import re
+from dataclasses import dataclass
+
+from ..schemas import ModelFamily
+
+_FAMILY_PATTERNS = [
+    (ModelFamily.CLAUDE, re.compile(r"(?i)claude|anthropic|sonnet|opus|haiku|fable")),
+    (ModelFamily.GPT, re.compile(r"(?i)gpt|openai|o[134][\b-]|chatgpt|davinci")),
+    (ModelFamily.GEMINI, re.compile(r"(?i)gemini|bard|google|palm")),
+    (ModelFamily.OPEN_WEIGHTS, re.compile(r"(?i)llama|mistral|mixtral|qwen|deepseek|phi-|falcon|vicuna|ollama")),
+]
+
+
+def detect_family(model_name: str) -> ModelFamily:
+    for family, pat in _FAMILY_PATTERNS:
+        if pat.search(model_name or ""):
+            return family
+    return ModelFamily.GENERIC
+
+
+@dataclass(frozen=True)
+class Profile:
+    family: ModelFamily
+    display: str
+    source: str                      # provider guidance citation (RESEARCH.md key)
+    # mechanism snippets used by the builders:
+    structure_hint: str              # how this family likes prompts organized
+    format_mechanism: str            # strongest format-control lever
+    grounding_mechanism: str         # strongest faithfulness lever
+
+    def wrap_source(self, placeholder: str = "<paste your source text here>") -> str:
+        if self.family == ModelFamily.CLAUDE:
+            return f"<source>\n{placeholder}\n</source>"
+        if self.family == ModelFamily.GPT:
+            return f'"""\n{placeholder}\n"""'
+        return f"--- SOURCE TEXT ---\n{placeholder}\n--- END SOURCE ---"
+
+
+PROFILES: dict[ModelFamily, Profile] = {
+    ModelFamily.CLAUDE: Profile(
+        family=ModelFamily.CLAUDE, display="Claude",
+        source="Anthropic prompt engineering docs (RESEARCH.md 2.1)",
+        structure_hint="Claude responds well to XML-tagged sections and an assigned role.",
+        format_mechanism="Tell Claude exactly how to open its reply (a prefill-style instruction) and tag the rules in XML.",
+        grounding_mechanism="Ask Claude to extract supporting quotes into <quotes> tags before writing, and to use only what it quoted.",
+    ),
+    ModelFamily.GPT: Profile(
+        family=ModelFamily.GPT, display="GPT",
+        source="OpenAI prompt engineering guide (RESEARCH.md 2.2)",
+        structure_hint="Newer GPT models follow instructions literally — state every requirement explicitly, separated by ### delimiters.",
+        format_mechanism="Number the requirements and restate the output format as the final line of the prompt.",
+        grounding_mechanism="Provide the reference text inside triple quotes and instruct: answer using only the provided text; say \"not in the text\" otherwise.",
+    ),
+    ModelFamily.GEMINI: Profile(
+        family=ModelFamily.GEMINI, display="Gemini",
+        source="Google Gemini prompting guidance (RESEARCH.md 2.3)",
+        structure_hint="Gemini's guidance: context first, question last, and always include a worked example (few-shot).",
+        format_mechanism="Show one example of the exact output you want — Gemini docs treat few-shot as the primary format lever. Phrase rules positively (\"do X\", not \"don't do Y\").",
+        grounding_mechanism="Place the source text before the question and add a positive rule: base every sentence on the text above.",
+    ),
+    ModelFamily.OPEN_WEIGHTS: Profile(
+        family=ModelFamily.OPEN_WEIGHTS, display="open-weights model",
+        source="lost-in-the-middle + instruction-following headroom notes (RESEARCH.md 2.7/Part 4)",
+        structure_hint="Smaller instruction-following headroom: keep the prompt short, one requirement per line, no nested asks.",
+        format_mechanism="Spell out the format mechanically and REPEAT the critical constraint as the last line — end-of-prompt text gets the most attention.",
+        grounding_mechanism="Keep the source short and close to the question; instruct it to copy exact phrases from the source rather than paraphrase.",
+    ),
+    ModelFamily.GENERIC: Profile(
+        family=ModelFamily.GENERIC, display="the model",
+        source="cross-provider consensus (RESEARCH.md Part 2)",
+        structure_hint="Separate instructions from content with clear delimiters.",
+        format_mechanism="State the output format explicitly and restate the critical constraint at the end of the prompt.",
+        grounding_mechanism="Label the source text clearly and instruct the model to use only that text.",
+    ),
+}
+
+
+def get_profile(model_name: str) -> Profile:
+    return PROFILES[detect_family(model_name)]

rubriq/scoring/__init__.py

@@ -0,0 +1,4 @@
+from .base import Scorer
+from .local import LocalScorer
+
+__all__ = ["Scorer", "LocalScorer"]

rubriq/scoring/base.py

@@ -0,0 +1,25 @@
+"""The Scorer adapter interface (required by brief).
+
+LocalScorer implements it today with deterministic/statistical methods.
+LLMJudgeScorer will implement the same contract later, in two configurations:
+BYO-key (free tier) and our-key (paid tier). Nothing downstream — API,
+frontend, tests — may import anything but this interface and the schemas.
+"""
+
+from __future__ import annotations
+
+from abc import ABC, abstractmethod
+
+from ..schemas import EvalReport, EvalRequest
+
+
+class Scorer(ABC):
+    """Turns an EvalRequest into an EvalReport. Implementations must set
+    EvalReport.engine and fill honesty_notes with whatever the engine
+    cannot judge."""
+
+    name: str = "abstract"
+
+    @abstractmethod
+    def evaluate(self, request: EvalRequest) -> EvalReport:
+        ...

rubriq/scoring/local.py

@@ -0,0 +1,72 @@
+"""LocalScorer: the v1 engine. Deterministic + statistical, no API calls."""
+
+from __future__ import annotations
+
+from ..classify import classify
+from ..prompts.generator import generate_improvement_prompts
+from ..schemas import EvalReport, EvalRequest, OverallScore, TaskType
+from .base import Scorer
+from .parameters import EvalContext
+from .taxonomy import TAXONOMY
+
+_CARVEOUT_NOTES = {
+    "open_world": ("Factual accuracy against world knowledge is not verifiable locally; "
+                   "Rubriq reports fabrication-risk signals instead. LLM-judge mode adds verification."),
+    "proxy": ("Parameters marked 'proxy' measure statistical stand-ins (diversity, rhythm, flow), "
+              "not the subjective quality itself."),
+    "static_code": "Code is analyzed statically, never executed; parsing cleanly does not mean it runs correctly.",
+}
+
+
+class LocalScorer(Scorer):
+    name = "local"
+
+    def evaluate(self, request: EvalRequest) -> EvalReport:
+        task = classify(request.prompt, request.output)
+        ctx = EvalContext(
+            prompt=request.prompt,
+            output=request.output,
+            task=task,
+            audience=request.options.audience,
+        )
+
+        specs = TAXONOMY[task.task_type]
+        params = []
+        weights: dict[str, float] = {}
+        for evaluator, default_w in specs:
+            score = evaluator(ctx)
+            params.append(score)
+            weights[score.parameter] = request.options.weights.get(score.parameter, default_w)
+
+        scored = [ps for ps in params if ps.score is not None and weights.get(ps.parameter, 0) > 0]
+        if scored:
+            total_w = sum(weights[ps.parameter] for ps in scored)
+            overall_val = sum(ps.score * weights[ps.parameter] for ps in scored) / total_w
+        else:
+            overall_val = 0.0
+        skipped = [ps.display_name for ps in params if ps.score is None]
+        note = (f"Weighted over {len(scored)} locally scoreable parameters"
+                + (f"; not included: {', '.join(skipped)}" if skipped else "") + ".")
+
+        honesty: list[str] = []
+        if task.task_type in (TaskType.QA_OPEN, TaskType.GENERAL, TaskType.REASONING) or not task.has_source_text:
+            honesty.append(_CARVEOUT_NOTES["open_world"])
+        if any(ps.is_proxy for ps in params):
+            honesty.append(_CARVEOUT_NOTES["proxy"])
+        if task.task_type == TaskType.CODE_GENERATION:
+            honesty.append(_CARVEOUT_NOTES["static_code"])
+
+        prompts = generate_improvement_prompts(
+            model_name=request.model_name,
+            task=task,
+            parameters=params,
+        )
+
+        return EvalReport(
+            task=task,
+            parameters=params,
+            overall=OverallScore(score=round(overall_val, 1), note=note),
+            improvement_prompts=prompts,
+            honesty_notes=honesty,
+            engine=self.name,
+        )

rubriq/scoring/parameters.py

@@ -0,0 +1,533 @@
+"""Parameter evaluators: metric results -> ParameterScore with verdicts,
+strengths, findings, and evidence quotes. Each evaluator names its rubric
+source (RESEARCH.md entry). Scores are 0-100; None means "could not judge",
+with the reason stated — never a silently invented number.
+"""
+
+from __future__ import annotations
+
+from dataclasses import dataclass, field
+
+from .. import constraints as con
+from ..metrics import (code_static, faithfulness, hallucination_risk, lexical,
+                       readability, relevance, structure)
+from ..schemas import (Evidence, FailureMode, Finding, ParameterScore,
+                       TaskClassification, TaskType)
+from ..textutils import snippet, split_sentences, word_count, words
+
+
+@dataclass
+class EvalContext:
+    """Shared inputs + lazily cached metric results for one evaluation."""
+    prompt: str
+    output: str
+    task: TaskClassification
+    audience: str = ""
+    _cache: dict = field(default_factory=dict)
+
+    @property
+    def instruction(self) -> str:
+        return self.task.instruction or self.prompt
+
+    @property
+    def source(self) -> str | None:
+        return self.task.source_text
+
+    def get(self, key: str, fn):
+        if key not in self._cache:
+            self._cache[key] = fn()
+        return self._cache[key]
+
+
+def _clamp(x: float) -> float:
+    return max(0.0, min(100.0, round(x, 1)))
+
+
+def _band(x: float, floor: float, ceil: float) -> float:
+    """Linear map [floor..ceil] -> [0..100]."""
+    if ceil <= floor:
+        return 50.0
+    return _clamp(100 * (x - floor) / (ceil - floor))
+
+
+# ------------------------------------------------------------ evaluators
+
+def instruction_following(ctx: EvalContext) -> ParameterScore:
+    results = ctx.get("constraints", lambda: con.run_checks(ctx.instruction, ctx.output))
+    src = "IFEval (Zhou et al. 2023)"
+    if not results:
+        return ParameterScore(
+            parameter="instruction_following", display_name="Instruction following",
+            score=None, verdict="No verifiable constraints (length, format, required content) detected in the prompt.",
+            skipped_reason="nothing checkable was asked", rubric_source=src)
+    passed = [r for r in results if r.passed]
+    findings = con.to_findings(results)
+    strengths = [f"Met: {r.constraint.description} ({r.detail})" for r in passed]
+    score = _clamp(100 * len(passed) / len(results))
+    verdict = (f"Met {len(passed)} of {len(results)} verifiable constraints extracted from the prompt."
+               if findings else f"All {len(results)} verifiable constraints met.")
+    return ParameterScore(
+        parameter="instruction_following", display_name="Instruction following",
+        score=score, verdict=verdict, strengths=strengths, findings=findings,
+        rubric_source=src)
+
+
+def faithfulness_param(ctx: EvalContext) -> ParameterScore:
+    src = "SummaC (Laban et al. 2022) / RAGAS (Es et al. 2024)"
+    if not ctx.source:
+        return ParameterScore(
+            parameter="faithfulness", display_name="Faithfulness to source",
+            score=None, verdict="No source text found in the prompt to check against.",
+            skipped_reason="no source text", rubric_source=src)
+    result = ctx.get("faithfulness", lambda: faithfulness.compute(ctx.source, ctx.output))
+    if result is None:
+        return ParameterScore(
+            parameter="faithfulness", display_name="Faithfulness to source",
+            score=None, verdict="NLI model unavailable on this machine.",
+            skipped_reason="nli model not loaded", rubric_source=src)
+    findings: list[Finding] = []
+    for s in result.unsupported:
+        findings.append(Finding(
+            failure_mode=FailureMode.UNFAITHFUL_CONTENT,
+            detail=f"This claim has no support in the source (entailment {s.entailment:.2f}).",
+            evidence=[Evidence(quote=snippet(s.sentence)),
+                      Evidence(quote=s.best_chunk, source="prompt", note="closest source passage")],
+            data={"entailment": s.entailment}))
+    for s in result.contradicted:
+        if all(s.sentence != f.evidence[0].quote for f in findings if f.evidence):
+            findings.append(Finding(
+                failure_mode=FailureMode.UNFAITHFUL_CONTENT,
+                detail=f"The source appears to contradict this claim (contradiction {s.contradiction:.2f}).",
+                evidence=[Evidence(quote=snippet(s.sentence))],
+                data={"contradiction": s.contradiction}))
+    n = len(result.sentences)
+    supported = n - len(result.unsupported)
+    strengths = [f"{supported} of {n} output claims are entailed by the source."] if supported else []
+    return ParameterScore(
+        parameter="faithfulness", display_name="Faithfulness to source",
+        score=_clamp(result.mean_entailment * 100),
+        verdict=(f"{supported}/{n} claims supported by the source; "
+                 f"{len(result.unsupported)} unsupported" +
+                 (f", {len(result.contradicted)} possibly contradicted." if result.contradicted else ".")),
+        strengths=strengths, findings=findings, rubric_source=src)
+
+
+def answer_relevance(ctx: EvalContext) -> ParameterScore:
+    src = "RAGAS answer relevancy (Es et al. 2024) / TruLens RAG triad"
+    result = ctx.get("relevance", lambda: relevance.compute(ctx.instruction, ctx.output))
+    if result is None:
+        return ParameterScore(
+            parameter="answer_relevance", display_name="Relevance to the request",
+            score=None, verdict="Embedding model unavailable on this machine.",
+            skipped_reason="embedding model not loaded", rubric_source=src)
+    findings: list[Finding] = []
+    if result.worst_paragraph is not None and result.worst_paragraph_sim is not None \
+            and result.worst_paragraph_sim < 0.45 and result.overall >= 0.45:
+        findings.append(Finding(
+            failure_mode=FailureMode.OFF_TOPIC_DRIFT,
+            detail=f"One section drifts from the request (similarity {result.worst_paragraph_sim:.2f} vs {result.overall:.2f} overall).",
+            evidence=[Evidence(quote=result.worst_paragraph)],
+            data={"paragraph_sim": result.worst_paragraph_sim}))
+    if result.overall < 0.45:
+        findings.append(Finding(
+            failure_mode=FailureMode.OFF_TOPIC_DRIFT,
+            detail=f"The output as a whole tracks the request weakly (semantic similarity {result.overall:.2f}).",
+            evidence=[Evidence(quote=snippet(ctx.instruction), source="prompt", note="the request")],
+            data={"overall_sim": result.overall}))
+    score = _band(result.overall, 0.35, 0.80)
+    strengths = ["Output stays on the topic of the request."] if not findings else []
+    return ParameterScore(
+        parameter="answer_relevance", display_name="Relevance to the request",
+        score=score,
+        verdict=f"Semantic similarity between request and output: {result.overall:.2f}.",
+        strengths=strengths, findings=findings, rubric_source=src)
+
+
+def coherence_proxy(ctx: EvalContext) -> ParameterScore:
+    src = "Coh-Metrix tradition (Graesser et al. 2004), embedding adaptation"
+    result = ctx.get("coherence", lambda: structure.coherence_proxy(ctx.output))
+    if result is None:
+        return ParameterScore(
+            parameter="coherence", display_name="Coherence (proxy)",
+            score=None, verdict="Output too short for flow analysis, or embedding model unavailable.",
+            skipped_reason="too short or model unavailable", is_proxy=True, rubric_source=src)
+    findings: list[Finding] = []
+    for a, b, sim in result.flagged_low:
+        findings.append(Finding(
+            failure_mode=FailureMode.POOR_STRUCTURE,
+            detail=f"Abrupt topic jump between adjacent sections (similarity {sim}).",
+            evidence=[Evidence(quote=a, note="…followed by…"), Evidence(quote=b)]))
+    for a, b, sim in result.flagged_high:
+        findings.append(Finding(
+            failure_mode=FailureMode.POOR_STRUCTURE,
+            detail=f"Near-duplicate adjacent sections (similarity {sim}) — the output repeats itself.",
+            evidence=[Evidence(quote=a), Evidence(quote=b)]))
+    score = _clamp(100 - 18 * len(result.flagged_low) - 12 * len(result.flagged_high))
+    return ParameterScore(
+        parameter="coherence", display_name="Coherence (proxy)",
+        score=score,
+        verdict=("Flow between sections reads consistent." if not findings
+                 else f"{len(findings)} flow problems between adjacent sections."),
+        strengths=["No abrupt jumps or repeated sections detected."] if not findings else [],
+        findings=findings, is_proxy=True, rubric_source=src)
+
+
+def conciseness(ctx: EvalContext) -> ParameterScore:
+    src = "MT-Bench verbosity-bias finding (Zheng et al. 2023), inverted"
+    checks = ctx.get("constraints", lambda: con.run_checks(ctx.instruction, ctx.output))
+    expected = None
+    for r in checks:
+        p = r.constraint.params
+        if p.get("unit") == "word" and p.get("bound") in ("max", "approx", "exact"):
+            expected = p["n"]
+    result = structure.conciseness(ctx.instruction, ctx.output, expected)
+    findings: list[Finding] = []
+    if result.filler_hits:
+        findings.append(Finding(
+            failure_mode=FailureMode.VERBOSITY,
+            detail=f"Filler phrasing adds words without content: {', '.join(repr(h) for h in result.filler_hits[:4])}.",
+            evidence=[Evidence(quote=h) for h in result.filler_hits[:3]]))
+    if result.ratio is not None and result.ratio > 1.3:
+        findings.append(Finding(
+            failure_mode=FailureMode.VERBOSITY,
+            detail=f"Output is {result.output_words} words against an expected ~{result.expected_max} ({result.ratio}x).",
+            data={"ratio": result.ratio}))
+    score = _clamp(100 - 8 * len(result.filler_hits) - (25 if result.ratio and result.ratio > 1.3 else 0))
+    return ParameterScore(
+        parameter="conciseness", display_name="Conciseness",
+        score=score,
+        verdict=(f"{result.output_words} words, no filler detected." if not findings
+                 else f"{result.output_words} words with padding present."),
+        strengths=["No filler phrases; length fits the ask."] if not findings else [],
+        findings=findings, rubric_source=src)
+
+
+def readability_audience(ctx: EvalContext) -> ParameterScore:
+    src = "Flesch (1948) / Kincaid et al. (1975)"
+    stats = readability.compute(ctx.output)
+    if stats is None:
+        return ParameterScore(
+            parameter="readability", display_name="Readability vs. audience",
+            score=None, verdict="Output too short for readability statistics.",
+            skipped_reason="too short", rubric_source=src)
+    (lo, hi), label = readability.audience_band(ctx.audience, ctx.instruction)
+    fk = stats.fk_grade
+    findings: list[Finding] = []
+    if fk < lo or fk > hi:
+        direction = "harder" if fk > hi else "simpler"
+        findings.append(Finding(
+            failure_mode=FailureMode.WRONG_REGISTER,
+            detail=(f"Reads at US grade level {fk}, outside the {lo:g}-{hi:g} band for {label} — "
+                    f"{direction} than the audience needs."),
+            data={"fk_grade": fk, "band": [lo, hi], "audience": label}))
+        dist = (lo - fk) if fk < lo else (fk - hi)
+        score = _clamp(100 - 12 * dist)
+    else:
+        score = 100.0
+    return ParameterScore(
+        parameter="readability", display_name="Readability vs. audience",
+        score=score,
+        verdict=f"Grade level {fk} (Flesch ease {stats.flesch_reading_ease}); target band {lo:g}-{hi:g} for {label}.",
+        strengths=[f"Reading level fits {label}."] if not findings else [],
+        findings=findings, rubric_source=src)
+
+
+def hallucination_signals(ctx: EvalContext) -> ParameterScore:
+    src = "FActScore precondition (Min et al. 2023); fabricated-citation literature"
+    result = hallucination_risk.compute(ctx.output, word_count(ctx.output))
+    findings: list[Finding] = []
+    for s in result.signals:
+        findings.append(Finding(
+            failure_mode=FailureMode.HALLUCINATION_RISK,
+            detail=f"{s.why} (signal: {s.kind}).",
+            evidence=[Evidence(quote=s.quote)],
+            data={"kind": s.kind}))
+    score = _clamp(100 - 14 * len(result.signals))
+    verdict = ("No fabrication-prone patterns detected. This does NOT verify the facts — see honesty notes."
+               if not findings else
+               f"{len(result.signals)} unverifiable-claim signals found. These are risk flags, not verdicts of falsehood.")
+    return ParameterScore(
+        parameter="hallucination_risk", display_name="Hallucination risk signals",
+        score=score, verdict=verdict,
+        strengths=["Output avoids unattributed statistics, invented-looking citations, and vague authority claims."] if not findings else [],
+        findings=findings, rubric_source=src)
+
+
+def factual_accuracy_placeholder(ctx: EvalContext) -> ParameterScore:
+    """The carve-out, rendered as a first-class parameter so the UI shows
+    what local scoring cannot do and what judge mode adds."""
+    return ParameterScore(
+        parameter="factual_accuracy", display_name="Factual accuracy",
+        score=None,
+        verdict="Verifying claims against world knowledge requires a judge model with external knowledge. "
+                "Local mode reports risk signals instead (see Hallucination risk).",
+        skipped_reason="requires LLM-judge mode", requires_judge=True,
+        rubric_source="FActScore (Min et al. 2023)")
+
+
+def creative_proxies(ctx: EvalContext) -> ParameterScore:
+    src = "MTLD (McCarthy & Jarvis 2010); cliché/rhythm proxies"
+    stats = lexical.compute(ctx.output)
+    findings: list[Finding] = []
+    score = 100.0
+    if stats.cliche_hits:
+        findings.append(Finding(
+            failure_mode=FailureMode.LOW_DIVERSITY,
+            detail=f"Stock phrases weaken the writing: {', '.join(repr(c) for c in stats.cliche_hits[:5])}.",
+            evidence=[Evidence(quote=c) for c in stats.cliche_hits[:3]]))
+        score -= 10 * min(5, len(stats.cliche_hits))
+    if stats.mtld is not None and stats.mtld < 50:
+        findings.append(Finding(
+            failure_mode=FailureMode.LOW_DIVERSITY,
+            detail=f"Low lexical diversity (MTLD {stats.mtld}; fluent varied prose typically sits above ~70).",
+            data={"mtld": stats.mtld}))
+        score -= 20
+    if stats.sentence_len_cv is not None and stats.sentence_len_cv < 0.25:
+        findings.append(Finding(
+            failure_mode=FailureMode.LOW_DIVERSITY,
+            detail=f"Flat sentence rhythm (length variation {stats.sentence_len_cv}) — sentences are all the same size.",
+            data={"cv": stats.sentence_len_cv}))
+        score -= 15
+    if stats.repeated_ngrams:
+        findings.append(Finding(
+            failure_mode=FailureMode.LOW_DIVERSITY,
+            detail="Repeated phrasing: " + "; ".join(f'"{g}"' for g in stats.repeated_ngrams[:3]),
+            evidence=[Evidence(quote=g) for g in stats.repeated_ngrams[:3]]))
+        score -= 10
+    strengths = []
+    if stats.mtld and stats.mtld >= 70:
+        strengths.append(f"Strong lexical variety (MTLD {stats.mtld}).")
+    if stats.sentence_len_cv and stats.sentence_len_cv >= 0.4:
+        strengths.append("Varied sentence rhythm.")
+    if not stats.cliche_hits:
+        strengths.append("No stock phrases detected.")
+    return ParameterScore(
+        parameter="creative_proxies", display_name="Style proxies (diversity, clichés, rhythm)",
+        score=_clamp(score),
+        verdict="Statistical style proxies only — creativity itself needs a human or judge model.",
+        strengths=strengths, findings=findings, is_proxy=True, rubric_source=src)
+
+
+def code_validity(ctx: EvalContext) -> ParameterScore:
+    src = "static ladder; execution reserved per HumanEval (Chen et al. 2021)"
+    result = ctx.get("code", lambda: code_static.compute(ctx.instruction, ctx.output))
+    findings: list[Finding] = []
+    if result.prose_only:
+        findings.append(Finding(
+            failure_mode=FailureMode.INVALID_CODE,
+            detail="The prompt asks for code but the output contains none.",
+            evidence=[Evidence(quote=snippet(ctx.instruction), source="prompt")]))
+        return ParameterScore(
+            parameter="code_validity", display_name="Code validity (static)",
+            score=0.0, verdict="No code found in the output.", findings=findings, rubric_source=src)
+    score = 100.0
+    for b in result.blocks:
+        if b.parses is False:
+            findings.append(Finding(
+                failure_mode=FailureMode.INVALID_CODE,
+                detail=f"{b.language} block does not parse: {b.parse_error}.",
+                evidence=[Evidence(quote=snippet(b.code, 200))]))
+            score -= 40
+    for name in result.requested_names_missing:
+        findings.append(Finding(
+            failure_mode=FailureMode.INCOMPLETE_COVERAGE,
+            detail=f'The prompt asks for "{name}" but it never appears in the code.',
+            data={"missing_name": name}))
+        score -= 20
+    if result.placeholders:
+        findings.append(Finding(
+            failure_mode=FailureMode.INVALID_CODE,
+            detail="Placeholder stubs left in the code: " + "; ".join(result.placeholders[:3]),
+            evidence=[Evidence(quote=p) for p in result.placeholders[:3]]))
+        score -= 15
+    n_ok = sum(1 for b in result.blocks if b.parses)
+    strengths = []
+    if n_ok:
+        strengths.append(f"{n_ok} code block(s) parse cleanly.")
+    if not result.placeholders:
+        strengths.append("No TODO/placeholder stubs.")
+    return ParameterScore(
+        parameter="code_validity", display_name="Code validity (static)",
+        score=_clamp(score),
+        verdict=("Code parses and covers the requested surface (static checks only — not executed)."
+                 if not findings else "Static analysis found problems (code was not executed)."),
+        strengths=strengths, findings=findings, rubric_source=src)
+
+
+_STOPWORDS = frozenset(
+    "the a an of to in on at for with and or is are was were did do does what who "
+    "when where why how which whom whose it its they their this that these those "
+    "there here be been has have had will would can could should you your".split())
+
+
+def _distinctive_words(part: str, other_parts: list[str]) -> set[str]:
+    """Content words unique to this question part (5-char prefixes so
+    'funded' matches 'funding')."""
+    shared = {w[:5] for other in other_parts for w in words(other)}
+    return {w[:5] for w in words(part) if w not in _STOPWORDS and len(w) >= 4 and w[:5] not in shared}
+
+
+def question_coverage(ctx: EvalContext) -> ParameterScore:
+    """Multi-part questions: is every part addressed?
+
+    PROXY, honestly bounded: bi-encoders measure topicality, not answerhood,
+    so a part is only flagged when its topic is missing — low embedding
+    similarity to every output sentence, or its distinctive keywords absent.
+    A question that is mentioned but left unanswered can slip through;
+    judge mode closes that gap.
+    """
+    src = "completeness adaptation of IFEval / RAGAS (topical proxy)"
+    parts = [s for s in split_sentences(ctx.instruction) if s.strip().endswith("?")]
+    if len(parts) < 2:
+        return ParameterScore(
+            parameter="question_coverage", display_name="Question coverage (proxy)",
+            score=None, verdict="Single-part question; coverage check not applicable.",
+            skipped_reason="not multi-part", is_proxy=True, rubric_source=src)
+
+    from ..models import get_embedder
+    embedder = get_embedder()
+    if embedder is None:
+        return ParameterScore(
+            parameter="question_coverage", display_name="Question coverage (proxy)",
+            score=None, verdict="Embedding model unavailable.",
+            skipped_reason="embedding model not loaded", is_proxy=True, rubric_source=src)
+
+    out_sents = split_sentences(ctx.output) or [ctx.output]
+    qv = embedder.encode(parts, normalize_embeddings=True)
+    ov = embedder.encode(out_sents, normalize_embeddings=True)
+    low_out_prefixes = {w[:5] for w in words(ctx.output)}
+
+    findings: list[Finding] = []
+    hit = 0
+    for i, part in enumerate(parts):
+        best_sim = float(max(qv[i] @ v for v in ov))
+        distinctive = _distinctive_words(part, parts[:i] + parts[i + 1:])
+        keywords_present = (not distinctive) or any(d in low_out_prefixes for d in distinctive)
+        if best_sim < 0.45 or (best_sim < 0.68 and not keywords_present):
+            findings.append(Finding(
+                failure_mode=FailureMode.INCOMPLETE_COVERAGE,
+                detail=(f"This part of the question appears unaddressed — its key terms never appear in the "
+                        f"output (best sentence similarity {best_sim:.2f})."),
+                evidence=[Evidence(quote=part, source="prompt")],
+                data={"similarity": round(best_sim, 3)}))
+        else:
+            hit += 1
+    return ParameterScore(
+        parameter="question_coverage", display_name="Question coverage (proxy)",
+        score=_clamp(100 * hit / len(parts)),
+        verdict=(f"{hit} of {len(parts)} question parts topically addressed. "
+                 "Topical check only — a part mentioned but not truly answered can pass."),
+        strengths=[f"All {len(parts)} question parts are addressed in the output."] if not findings else [],
+        findings=findings, is_proxy=True, rubric_source=src)
+
+
+def meaning_preservation(ctx: EvalContext) -> ParameterScore:
+    src = "embedding similarity (RAGAS-style), rewriting adaptation"
+    if not ctx.source:
+        return ParameterScore(
+            parameter="meaning_preservation", display_name="Meaning preservation",
+            score=None, verdict="Original text not found in the prompt.",
+            skipped_reason="no source text", rubric_source=src)
+    from ..metrics.relevance import similarity
+    sim = similarity(ctx.source, ctx.output)
+    if sim is None:
+        return ParameterScore(
+            parameter="meaning_preservation", display_name="Meaning preservation",
+            score=None, verdict="Embedding model unavailable.",
+            skipped_reason="embedding model not loaded", rubric_source=src)
+    findings: list[Finding] = []
+    if sim < 0.6:
+        findings.append(Finding(
+            failure_mode=FailureMode.UNFAITHFUL_CONTENT,
+            detail=f"The rewrite drifts from the original's meaning (similarity {sim:.2f}).",
+            data={"similarity": sim}))
+    return ParameterScore(
+        parameter="meaning_preservation", display_name="Meaning preservation",
+        score=_band(sim, 0.45, 0.9),
+        verdict=f"Semantic similarity to the original: {sim:.2f}.",
+        strengths=[f"Rewrite preserves the original's meaning (similarity {sim:.2f})."] if not findings else [],
+        findings=findings, rubric_source=src)
+
+
+def extraction_grounding(ctx: EvalContext) -> ParameterScore:
+    """No-invention check: extracted values must appear in the source."""
+    src = "no-invention adaptation of faithfulness (RESEARCH.md 1.2/1.9)"
+    if not ctx.source:
+        return ParameterScore(
+            parameter="extraction_grounding", display_name="Extracted values appear in source",
+            score=None, verdict="No source text found to validate extractions against.",
+            skipped_reason="no source text", rubric_source=src)
+    import json
+    import re as _re
+    values: list[str] = []
+    try:
+        fence = _re.search(r"```(?:json)?\s*(.+?)```", ctx.output, flags=_re.S)
+        data = json.loads((fence.group(1) if fence else ctx.output).strip())
+
+        def walk(node):
+            if isinstance(node, dict):
+                for v in node.values():
+                    walk(v)
+            elif isinstance(node, list):
+                for v in node:
+                    walk(v)
+            elif isinstance(node, str) and 2 <= len(node) <= 80:
+                values.append(node)
+        walk(data)
+    except (json.JSONDecodeError, AttributeError):
+        values = [_re.sub(r"^[-*\d.)\s]+", "", ln).strip()
+                  for ln in ctx.output.splitlines() if ln.strip()]
+        values = [v for v in values if 2 <= len(v) <= 80]
+    if not values:
+        return ParameterScore(
+            parameter="extraction_grounding", display_name="Extracted values appear in source",
+            score=None, verdict="Could not isolate extracted values in the output.",
+            skipped_reason="no parseable values", rubric_source=src)
+    low_src = ctx.source.lower()
+    invented = [v for v in values if v.lower() not in low_src]
+    findings = [Finding(
+        failure_mode=FailureMode.INVENTED_FIELDS,
+        detail="Extracted value does not appear in the source text.",
+        evidence=[Evidence(quote=v)]) for v in invented[:5]]
+    ok = len(values) - len(invented)
+    return ParameterScore(
+        parameter="extraction_grounding", display_name="Extracted values appear in source",
+        score=_clamp(100 * ok / len(values)),
+        verdict=f"{ok} of {len(values)} extracted values found verbatim in the source.",
+        strengths=[f"All {len(values)} extracted values trace to the source."] if not invented else [],
+        findings=findings, rubric_source=src)
+
+
+def reasoning_structure(ctx: EvalContext) -> ParameterScore:
+    src = "CoT structure (Wei et al. 2022); derivation-shape heuristics"
+    out = ctx.output
+    findings: list[Finding] = []
+    has_steps = bool(__import__("re").search(
+        r"(?im)^(?:step\s*\d|first|then|next|finally|\d+[.)])[,:\s]", out))
+    sents = split_sentences(out)
+    has_answer = bool(__import__("re").search(
+        r"(?i)\b(?:therefore|so|thus|the answer is|equals?|=\s*\S|in total|final answer)\b", out))
+    score = 100.0
+    if not has_steps and len(sents) <= 2:
+        findings.append(Finding(
+            failure_mode=FailureMode.WEAK_REASONING,
+            detail="The output gives a conclusion with no visible working — nothing to check the answer against.",
+            evidence=[Evidence(quote=snippet(out, 200))]))
+        score -= 45
+    if not has_answer:
+        findings.append(Finding(
+            failure_mode=FailureMode.WEAK_REASONING,
+            detail="No clearly stated final answer found.",
+        ))
+        score -= 30
+    strengths = []
+    if has_steps:
+        strengths.append("Shows step-by-step working.")
+    if has_answer:
+        strengths.append("States a final answer explicitly.")
+    return ParameterScore(
+        parameter="reasoning_structure", display_name="Reasoning structure",
+        score=_clamp(score),
+        verdict=("Derivation shows its work and lands on an answer." if not findings
+                 else "The derivation's shape has gaps (this checks structure, not mathematical truth)."),
+        strengths=strengths, findings=findings, is_proxy=True, rubric_source=src)

rubriq/scoring/taxonomy.py

@@ -0,0 +1,82 @@
+"""Task type -> parameter set with default weights (RESEARCH.md Part 3).
+
+Weights are relative within a task and user-overridable via
+EvalOptions.weights (advanced section). Parameters that return score=None
+drop out of the weighted overall automatically.
+"""
+
+from __future__ import annotations
+
+from typing import Callable
+
+from . import parameters as p
+from .parameters import EvalContext
+from ..schemas import ParameterScore, TaskType
+
+Evaluator = Callable[[EvalContext], ParameterScore]
+
+# (evaluator, default weight)
+TAXONOMY: dict[TaskType, list[tuple[Evaluator, float]]] = {
+    TaskType.SUMMARIZATION: [
+        (p.faithfulness_param, 3.0),
+        (p.answer_relevance, 2.0),
+        (p.instruction_following, 2.0),
+        (p.conciseness, 1.5),
+        (p.coherence_proxy, 1.0),
+        (p.readability_audience, 1.0),
+    ],
+    TaskType.QA_GROUNDED: [
+        (p.faithfulness_param, 3.0),
+        (p.answer_relevance, 2.5),
+        (p.question_coverage, 2.0),
+        (p.instruction_following, 1.5),
+        (p.conciseness, 1.0),
+    ],
+    TaskType.QA_OPEN: [
+        (p.hallucination_signals, 2.5),
+        (p.factual_accuracy_placeholder, 0.0),
+        (p.answer_relevance, 2.5),
+        (p.question_coverage, 2.0),
+        (p.instruction_following, 1.5),
+        (p.coherence_proxy, 1.0),
+        (p.readability_audience, 1.0),
+    ],
+    TaskType.CODE_GENERATION: [
+        (p.code_validity, 3.0),
+        (p.instruction_following, 2.0),
+        (p.answer_relevance, 1.0),
+        (p.conciseness, 0.5),
+    ],
+    TaskType.CREATIVE_WRITING: [
+        (p.creative_proxies, 2.5),
+        (p.instruction_following, 2.5),
+        (p.coherence_proxy, 1.0),
+        (p.readability_audience, 1.0),
+    ],
+    TaskType.EXTRACTION: [
+        (p.extraction_grounding, 3.0),
+        (p.instruction_following, 3.0),
+        (p.answer_relevance, 1.0),
+    ],
+    TaskType.REASONING: [
+        (p.reasoning_structure, 2.5),
+        (p.answer_relevance, 1.5),
+        (p.instruction_following, 2.0),
+        (p.hallucination_signals, 1.0),
+        (p.conciseness, 0.5),
+    ],
+    TaskType.REWRITING: [
+        (p.meaning_preservation, 3.0),
+        (p.instruction_following, 2.5),
+        (p.readability_audience, 1.5),
+        (p.creative_proxies, 1.0),
+    ],
+    TaskType.GENERAL: [
+        (p.instruction_following, 2.5),
+        (p.answer_relevance, 2.0),
+        (p.hallucination_signals, 1.5),
+        (p.coherence_proxy, 1.0),
+        (p.readability_audience, 1.0),
+        (p.conciseness, 1.0),
+    ],
+}

tests/samples.py

@@ -0,0 +1,122 @@
+"""Curated prompt/output pairs with known, intended failure modes.
+Each pair documents what a sensible evaluator MUST notice."""
+
+SOURCE_ARTICLE = """\
+The Larsen Glacier expedition of 1987 was led by Norwegian climatologist
+Dr. Ingrid Solheim. Over four months, the twelve-person team drilled ice
+cores to a depth of 800 meters, recovering samples dating back roughly
+12,000 years. The expedition's findings, published in 1989, showed that
+regional temperatures had risen 1.4 degrees Celsius over the preceding
+century. Funding came primarily from the Norwegian Polar Institute, with
+additional support from the University of Bergen. Two team members were
+evacuated in the third month after a crevasse accident, but there were no
+fatalities. The ice core archive remains in Tromsø and is still used by
+researchers today."""
+
+# --- summarization ---------------------------------------------------------
+
+SUMM_PROMPT = (
+    "Summarize the following text in at most 60 words.\n\n" + SOURCE_ARTICLE
+)
+
+SUMM_GOOD = (
+    "In 1987, Dr. Ingrid Solheim led a twelve-person Norwegian expedition that "
+    "drilled 800-meter ice cores on the Larsen Glacier, capturing 12,000 years "
+    "of climate data. Published in 1989, the findings showed regional warming "
+    "of 1.4 degrees Celsius over a century. Despite a crevasse accident, "
+    "no one died."
+)  # 51 words, faithful
+
+SUMM_BAD = (
+    "The famous 1987 Larsen Glacier expedition, led by the celebrated Dr. Ingrid "
+    "Solheim, made groundbreaking discoveries that changed climate science forever. "
+    "The team of fifteen brave scientists drilled to an unprecedented 1,200 meters, "
+    "and tragically three researchers lost their lives in an avalanche. Their "
+    "findings proved that global temperatures rose 3 degrees. It is important to "
+    "note that this expedition remains a testament to human perseverance and "
+    "stands as a pivotal moment in the history of polar research, inspiring "
+    "generations of scientists around the world to pursue climate research."
+)  # ~85 words: over limit, wrong team size, wrong depth, invented deaths, wrong figure, filler
+
+# --- code generation --------------------------------------------------------
+
+CODE_PROMPT = (
+    "Write a Python function called parse_log_line that takes a log line like "
+    "'2024-01-03 ERROR disk full' and returns a dict with keys date, level, "
+    "and message."
+)
+
+CODE_GOOD = '''Here's the function:
+
+```python
+def parse_log_line(line: str) -> dict:
+    date, level, message = line.split(" ", 2)
+    return {"date": date, "level": level, "message": message}
+```
+
+It splits on the first two spaces so messages may contain spaces.'''
+
+CODE_BAD = '''```python
+def parse_line(line):
+    # TODO: implement parsing logic
+    parts = line.split(" "
+    return {"date": parts[0]}
+```'''  # wrong name, syntax error, TODO stub
+
+# --- extraction -------------------------------------------------------------
+
+EXTRACT_PROMPT = (
+    "Extract all person names and institutions from the following text as JSON "
+    'with keys "people" and "institutions".\n\n' + SOURCE_ARTICLE
+)
+
+EXTRACT_GOOD = """{"people": ["Dr. Ingrid Solheim"], "institutions": ["Norwegian Polar Institute", "University of Bergen"]}"""
+
+EXTRACT_BAD = """{"people": ["Dr. Ingrid Solheim", "Dr. Erik Larsen"], "institutions": ["Norwegian Polar Institute", "Oslo Climate Center"]}"""
+# Erik Larsen and Oslo Climate Center are invented
+
+# --- open-world QA ----------------------------------------------------------
+
+QA_OPEN_PROMPT = "What are the health benefits of intermittent fasting?"
+
+QA_OPEN_RISKY = (
+    "Intermittent fasting has remarkable health benefits. Studies show that it "
+    "extends lifespan by 23.5% and reduces cancer risk by 41%. According to "
+    "Dr. James Hoffman of Harvard Medical School, \"fasting is the single most "
+    "powerful intervention in modern medicine\" (Hoffman et al., 2019). Research "
+    "published at https://fasting-institute.org/study confirms these findings."
+)
+
+# --- creative writing -------------------------------------------------------
+
+CREATIVE_PROMPT = "Write a short story opening about a sailor returning home after ten years at sea."
+
+CREATIVE_CLICHED = (
+    "At the end of the day, the old sailor knew that home was where the heart is. "
+    "Little did he know what awaited him. His heart skipped a beat as the ship "
+    "pulled into the bustling harbor, a whirlwind of emotions washing over him. "
+    "In the blink of an eye, ten long years had passed, and time stood still as "
+    "he gazed upon the picturesque village nestled in the hills, a hidden gem "
+    "against all odds. It was a testament to his perseverance."
+)
+
+# --- multi-part grounded QA -------------------------------------------------
+
+QA_MULTI_PROMPT = (
+    "Based on the following text, answer these questions. Who led the expedition? "
+    "How deep did they drill? Who funded it?\n\n"
+    + SOURCE_ARTICLE
+)
+
+QA_MULTI_PARTIAL = (
+    "The expedition was led by Dr. Ingrid Solheim, a Norwegian climatologist. "
+    "The team drilled ice cores to a depth of 800 meters over four months."
+)  # funding question ignored — its key terms never appear in the answer
+
+# Documented limitation of the topical coverage proxy: a question whose TOPIC
+# appears in the answer but is not actually answered will pass. Kept as a test
+# so the limitation stays visible.
+QA_MULTI_TOPIC_MENTIONED = (
+    "Based on the following text, answer these questions. Who led the expedition? "
+    "What happened to the ice cores afterwards?\n\n" + SOURCE_ARTICLE
+)

tests/test_classifier.py

@@ -0,0 +1,45 @@
+"""Task classification on the sample pairs (rules; embeddings when cached)."""
+
+import samples
+from rubriq.classify import classify, split_instruction_and_source
+from rubriq.schemas import TaskType
+
+
+def test_summarization_detected():
+    c = classify(samples.SUMM_PROMPT, samples.SUMM_GOOD)
+    assert c.task_type == TaskType.SUMMARIZATION
+    assert c.has_source_text
+
+
+def test_code_detected():
+    c = classify(samples.CODE_PROMPT, samples.CODE_GOOD)
+    assert c.task_type == TaskType.CODE_GENERATION
+
+
+def test_extraction_detected():
+    c = classify(samples.EXTRACT_PROMPT, samples.EXTRACT_GOOD)
+    assert c.task_type == TaskType.EXTRACTION
+    assert c.has_source_text
+
+
+def test_creative_detected():
+    c = classify(samples.CREATIVE_PROMPT, samples.CREATIVE_CLICHED)
+    assert c.task_type == TaskType.CREATIVE_WRITING
+
+
+def test_open_qa_detected():
+    c = classify(samples.QA_OPEN_PROMPT, samples.QA_OPEN_RISKY)
+    assert c.task_type == TaskType.QA_OPEN
+    assert not c.has_source_text
+
+
+def test_grounded_qa_upgrade():
+    c = classify(samples.QA_MULTI_PROMPT, samples.QA_MULTI_PARTIAL)
+    assert c.task_type == TaskType.QA_GROUNDED
+    assert c.has_source_text
+
+
+def test_source_split_keeps_instruction():
+    instruction, source = split_instruction_and_source(samples.SUMM_PROMPT)
+    assert "at most 60 words" in instruction
+    assert source is not None and "Solheim" in source

tests/test_constraints.py

@@ -0,0 +1,70 @@
+"""Deterministic constraint extraction + checking. No models needed."""
+
+from rubriq.constraints import extract_constraints, run_checks
+
+
+def kinds(instruction):
+    return {c.kind for c in extract_constraints(instruction)}
+
+
+class TestExtraction:
+    def test_max_words(self):
+        assert "max_word" in kinds("Summarize this in at most 150 words.")
+        assert "max_word" in kinds("Keep it under 50 words please")
+        assert "max_word" in kinds("no more than 100 words")
+
+    def test_min_and_exact_and_approx(self):
+        assert "min_word" in kinds("Write at least 300 words about dogs.")
+        assert "exact_sentence" in kinds("Answer in exactly two sentences.")
+        assert "approx_word" in kinds("Write about 200 words on this topic.")
+
+    def test_bullets_and_format(self):
+        assert "max_bullet" in kinds("Give me at most 5 bullet points.")
+        assert "format_json" in kinds("Return the result as JSON.")
+        assert "format_json" in kinds("Output valid JSON only")
+        assert "format_table" in kinds("Present this as a markdown table.")
+        assert "single_paragraph" in kinds("Explain it in a single paragraph.")
+
+    def test_include_exclude(self):
+        ks = extract_constraints('Include the word "sustainability" and don\'t use the word "synergy".')
+        got = {(c.kind, c.params.get("term")) for c in ks}
+        assert ("must_include", "sustainability") in got
+        assert ("must_exclude", "synergy") in got
+
+    def test_no_false_positives_on_plain_prompt(self):
+        assert kinds("Tell me about the history of Rome.") == set()
+
+
+class TestChecking:
+    def test_word_limit_violation_detected(self):
+        results = run_checks("Summarize in at most 10 words.", "This output has quite a few more than ten words in it, clearly.")
+        r = next(r for r in results if r.constraint.kind == "max_word")
+        assert not r.passed
+
+    def test_word_limit_pass(self):
+        results = run_checks("Summarize in at most 10 words.", "Seven words are in this sentence, exactly.")
+        r = next(r for r in results if r.constraint.kind == "max_word")
+        assert r.passed
+
+    def test_json_validity(self):
+        ok = run_checks("Return as JSON.", '{"a": 1}')
+        assert ok[0].passed
+        bad = run_checks("Return as JSON.", '{"a": 1,,}')
+        assert not bad[0].passed
+        assert "position" in bad[0].detail
+
+    def test_json_in_fence_ok(self):
+        ok = run_checks("Return as JSON.", '```json\n{"a": 1}\n```')
+        assert ok[0].passed
+
+    def test_must_exclude_quotes_evidence(self):
+        results = run_checks('Avoid using the word "leverage".', "We can leverage this approach.")
+        r = next(r for r in results if r.constraint.kind == "must_exclude")
+        assert not r.passed
+        assert r.evidence and "leverage" in r.evidence[0].quote.lower()
+
+    def test_code_fence_excluded_from_word_count(self):
+        prompt = "Explain in at most 20 words."
+        output = "Short answer here, well within limits.\n```python\n" + "x = 1\n" * 200 + "```"
+        r = next(r for r in run_checks(prompt, output) if r.constraint.kind == "max_word")
+        assert r.passed

tests/test_prompts.py

@@ -0,0 +1,88 @@
+"""The flagship contract: improvement prompts must name the diagnosed
+problem, quote evidence, be paste-ready, and use the named model family's
+mechanism. Generic advice is a test failure by design."""
+
+import samples
+from rubriq.schemas import EvalRequest, FailureMode, ModelFamily
+from rubriq.scoring import LocalScorer
+
+scorer = LocalScorer()
+
+GENERIC_PHRASES = [
+    "be more specific", "add more detail", "improve your prompt",
+    "be clearer", "provide more context",
+]
+
+
+def run(prompt, output, model=""):
+    return scorer.evaluate(EvalRequest(prompt=prompt, output=output, model_name=model))
+
+
+def test_bad_summary_generates_prompts():
+    report = run(samples.SUMM_PROMPT, samples.SUMM_BAD, model="gpt-4o")
+    assert report.improvement_prompts, "a failing output must generate repairs"
+    modes = {p.failure_mode for p in report.improvement_prompts}
+    assert FailureMode.UNFAITHFUL_CONTENT in modes
+    assert FailureMode.MISSED_CONSTRAINT in modes or FailureMode.VERBOSITY in modes
+
+
+def test_no_generic_advice():
+    for pair in [(samples.SUMM_PROMPT, samples.SUMM_BAD), (samples.CODE_PROMPT, samples.CODE_BAD)]:
+        report = run(*pair, model="gpt-4o")
+        for ip in report.improvement_prompts:
+            low = (ip.diagnosis + " " + ip.prompt_text).lower()
+            for phrase in GENERIC_PHRASES:
+                assert phrase not in low, f"generic advice '{phrase}' in {ip.title}"
+
+
+def test_diagnosis_names_the_evidence():
+    report = run(samples.SUMM_PROMPT, samples.SUMM_BAD, model="gpt-4o")
+    unfaithful = next(p for p in report.improvement_prompts
+                      if p.failure_mode == FailureMode.UNFAITHFUL_CONTENT)
+    # The repair prompt must carry the actual invented claims back to the model.
+    assert '"' in unfaithful.prompt_text
+    assert "previous attempt" in unfaithful.prompt_text.lower()
+
+
+def test_prompt_contains_original_instruction():
+    report = run(samples.SUMM_PROMPT, samples.SUMM_BAD, model="gpt-4o")
+    for ip in report.improvement_prompts:
+        assert "60 words" in ip.prompt_text or "Summarize" in ip.prompt_text, \
+            "repair prompts must be complete replacements, not advice"
+
+
+def test_claude_gets_xml_mechanism():
+    report = run(samples.SUMM_PROMPT, samples.SUMM_BAD, model="claude sonnet")
+    unfaithful = next(p for p in report.improvement_prompts
+                      if p.failure_mode == FailureMode.UNFAITHFUL_CONTENT)
+    assert unfaithful.model_family == ModelFamily.CLAUDE
+    assert "<quotes>" in unfaithful.prompt_text or "<source>" in unfaithful.prompt_text
+
+
+def test_gpt_gets_delimiters_and_tail_emphasis():
+    report = run(samples.SUMM_PROMPT, samples.SUMM_BAD, model="gpt-4o")
+    unfaithful = next(p for p in report.improvement_prompts
+                      if p.failure_mode == FailureMode.UNFAITHFUL_CONTENT)
+    assert unfaithful.model_family == ModelFamily.GPT
+    assert "IMPORTANT" in unfaithful.prompt_text or "###" in unfaithful.prompt_text
+
+
+def test_gemini_gets_fewshot_for_format():
+    report = run(samples.SUMM_PROMPT, samples.SUMM_BAD, model="gemini-1.5-pro")
+    constraint = next((p for p in report.improvement_prompts
+                       if p.failure_mode in (FailureMode.MISSED_CONSTRAINT, FailureMode.FORMAT_VIOLATION)), None)
+    if constraint:
+        assert "example" in constraint.prompt_text.lower()
+
+
+def test_code_repair_names_defects():
+    report = run(samples.CODE_PROMPT, samples.CODE_BAD)
+    code = next(p for p in report.improvement_prompts if p.failure_mode == FailureMode.INVALID_CODE)
+    assert "parse_log_line" in code.prompt_text  # the missing requested name travels into the repair
+    assert "fix each" in code.prompt_text.lower()
+
+
+def test_technique_sources_cited():
+    report = run(samples.SUMM_PROMPT, samples.SUMM_BAD, model="gpt-4o")
+    for ip in report.improvement_prompts:
+        assert ip.technique_source.strip(), "every template must cite its source"

tests/test_scoring.py

@@ -0,0 +1,135 @@
+"""End-to-end: LocalScorer must behave sensibly on the curated pairs.
+"Sensibly" is concrete: good outputs beat bad outputs, known flaws are
+found and quoted, and improvement prompts obey the no-generic-advice
+contract. Requires the two local models (cached by warmup)."""
+
+import pytest
+
+import samples
+from rubriq.schemas import EvalRequest, FailureMode
+from rubriq.scoring import LocalScorer
+
+scorer = LocalScorer()
+
+
+def run(prompt, output, model=""):
+    return scorer.evaluate(EvalRequest(prompt=prompt, output=output, model_name=model))
+
+
+@pytest.fixture(scope="module")
+def summ_good():
+    return run(samples.SUMM_PROMPT, samples.SUMM_GOOD)
+
+
+@pytest.fixture(scope="module")
+def summ_bad():
+    return run(samples.SUMM_PROMPT, samples.SUMM_BAD, model="gpt-4o")
+
+
+class TestSummarization:
+    def test_good_beats_bad_overall(self, summ_good, summ_bad):
+        assert summ_good.overall.score > summ_bad.overall.score + 10
+
+    def test_bad_fails_word_limit(self, summ_bad):
+        instr = next(p for p in summ_bad.parameters if p.parameter == "instruction_following")
+        assert any(f.failure_mode in (FailureMode.MISSED_CONSTRAINT, FailureMode.FORMAT_VIOLATION)
+                   for f in instr.findings)
+
+    def test_good_passes_word_limit(self, summ_good):
+        instr = next(p for p in summ_good.parameters if p.parameter == "instruction_following")
+        assert instr.score == 100.0
+
+    def test_bad_flagged_unfaithful(self, summ_bad):
+        faith = next(p for p in summ_bad.parameters if p.parameter == "faithfulness")
+        assert faith.score is not None
+        assert any(f.failure_mode == FailureMode.UNFAITHFUL_CONTENT for f in faith.findings)
+
+    def test_good_faithfulness_beats_bad(self, summ_good, summ_bad):
+        fg = next(p for p in summ_good.parameters if p.parameter == "faithfulness")
+        fb = next(p for p in summ_bad.parameters if p.parameter == "faithfulness")
+        assert fg.score > fb.score
+
+    def test_findings_quote_the_output(self, summ_bad):
+        faith = next(p for p in summ_bad.parameters if p.parameter == "faithfulness")
+        quoted = [e.quote for f in faith.findings for e in f.evidence if e.source == "output"]
+        assert quoted, "unfaithfulness findings must quote the offending sentences"
+
+
+class TestCode:
+    def test_bad_code_flagged(self):
+        report = run(samples.CODE_PROMPT, samples.CODE_BAD)
+        code = next(p for p in report.parameters if p.parameter == "code_validity")
+        details = " ".join(f.detail for f in code.findings)
+        assert "parse" in details.lower()
+        assert "parse_log_line" in details  # requested name missing
+
+    def test_good_code_clean(self):
+        report = run(samples.CODE_PROMPT, samples.CODE_GOOD)
+        code = next(p for p in report.parameters if p.parameter == "code_validity")
+        assert code.score >= 80
+
+    def test_honesty_note_static_only(self):
+        report = run(samples.CODE_PROMPT, samples.CODE_GOOD)
+        assert any("never executed" in n for n in report.honesty_notes)
+
+
+class TestExtraction:
+    def test_invented_values_caught(self):
+        report = run(samples.EXTRACT_PROMPT, samples.EXTRACT_BAD)
+        g = next(p for p in report.parameters if p.parameter == "extraction_grounding")
+        invented = [e.quote for f in g.findings for e in f.evidence]
+        assert any("Erik Larsen" in q for q in invented)
+        assert any("Oslo Climate Center" in q for q in invented)
+
+    def test_clean_extraction_scores_full(self):
+        report = run(samples.EXTRACT_PROMPT, samples.EXTRACT_GOOD)
+        g = next(p for p in report.parameters if p.parameter == "extraction_grounding")
+        assert g.score == 100.0
+
+
+class TestOpenQA:
+    def test_risk_signals_found(self):
+        report = run(samples.QA_OPEN_PROMPT, samples.QA_OPEN_RISKY)
+        risk = next(p for p in report.parameters if p.parameter == "hallucination_risk")
+        kinds = {f.data.get("kind") for f in risk.findings}
+        assert "citation_pattern" in kinds
+        assert "precise_statistic" in kinds
+        assert "url" in kinds
+
+    def test_factual_accuracy_declared_out_of_scope(self):
+        report = run(samples.QA_OPEN_PROMPT, samples.QA_OPEN_RISKY)
+        fact = next(p for p in report.parameters if p.parameter == "factual_accuracy")
+        assert fact.score is None and fact.requires_judge
+
+    def test_carveout_honesty_note_present(self):
+        report = run(samples.QA_OPEN_PROMPT, samples.QA_OPEN_RISKY)
+        assert any("judge" in n.lower() for n in report.honesty_notes)
+
+
+class TestCreative:
+    def test_cliches_flagged_as_proxy(self):
+        report = run(samples.CREATIVE_PROMPT, samples.CREATIVE_CLICHED)
+        style = next(p for p in report.parameters if p.parameter == "creative_proxies")
+        assert style.is_proxy
+        assert any(f.failure_mode == FailureMode.LOW_DIVERSITY for f in style.findings)
+        quoted = " ".join(e.quote for f in style.findings for e in f.evidence)
+        assert "at the end of the day" in quoted or "heart skipped a beat" in quoted
+
+
+class TestCoverage:
+    def test_unanswered_part_found(self):
+        report = run(samples.QA_MULTI_PROMPT, samples.QA_MULTI_PARTIAL)
+        cov = next(p for p in report.parameters if p.parameter == "question_coverage")
+        assert cov.score is not None and cov.score < 100
+        assert cov.is_proxy
+        quoted = " ".join(e.quote for f in cov.findings for e in f.evidence)
+        assert "funded" in quoted.lower()  # the ignored funding question
+
+    def test_known_limitation_topic_mentioned_passes(self):
+        """Documents the proxy's honest limit: a question whose topic appears
+        in the answer (ice cores) passes the topical check even though it was
+        not truly answered. Judge mode is the fix, not a stricter threshold."""
+        report = run(samples.QA_MULTI_TOPIC_MENTIONED, samples.QA_MULTI_PARTIAL)
+        cov = next(p for p in report.parameters if p.parameter == "question_coverage")
+        assert cov.score == 100.0
+        assert "topical" in cov.verdict.lower()