docs/EVALUATION.md

Evaluation guide

How Charles (or any reviewer) can convince themselves the project works, and how to interpret the numbers once ML components are swapped in.

1. The 60-second smoke test

uv sync --extra dev
uv run pytest -m "not ml"
uv run hcc-meat-eval regression
uv run hcc-meat-eval faithfulness

Expected: every unit test green; regression reports 10/10 PASS (or it lists the exact fixture names and what was missing/forbidden); faithfulness prints 0 fabrications.

If any of those fails on a fresh checkout, fix before anything else.

2. Interactive UI walkthrough (5 minutes)

uv run hcc-meat serve &
(cd ui && npm install && npm run dev)

Open http://localhost:5173 and click through the ten gallery items. For each, verify:

Case	What to look for
Straightforward DM2 with chronic complications	HCC-18 appears first, LOW risk, all four MEAT cells filled
Family history of malignancy	No HCCs in the list (empty accepted set)
Ruled-out diagnosis	No HCCs; "denies diabetes" does not code
Historical chronic DM	HCC-19 appears with reasonable MEAT coverage
DM hierarchy trumping	HCC-18 appears non-trumped; HCC-19 appears trumped-by HCC-18 (dimmed in the list)
CHF insufficient MEAT	HCC-85 appears but with MEDIUM risk; drill-in shows "Missing MEAT categories: Monitor, Evaluate, Treat"
Multi-specialty	At least HCC-85, HCC-137, HCC-18 captured
Short note	HCC-111 captured despite absence of headers
Dialysis trumps CKD	HCC-134 appears non-trumped; HCC-136 dimmed, labeled "trumped by HCC-134"
Synonym robustness	DM2 and CHF surface forms resolve to HCC-19 and HCC-85

Every verified MEAT quote in the drill-in must visibly highlight in the note preview on the left (color-coded by M/E/A/T).

3. Metrics Charles should care about

Priority matches the brief's success criteria.

S1 — Faithfulness (non-negotiable)

• Substring rate: 100%. Any deviation is a bug.
• NLI entailment rate: measured once the NLI model is wired. Target ≥ 0.8 on the held-out MEAT dataset.
• Verifier drop count: reported per request in CodingResult.verifier_drops and shown in the UI footer. Should stay near 0 once the LLM extractor is in use.

S2 — Latency (free CPU tier target < 8s per note)

• Measured via CodingResult.latency_ms. The rule-based baseline runs in < 50 ms on any note. The LLM extractor is the bottleneck; plan to quantize to Q4 GGUF and cache gallery outputs server-side.

S3 — Subgroup slices (UI surface)

• Not yet in the UI. Planned tabs: by age band, note length bucket, specialty, common-vs-rare HCC. eval_harness/cli.py subgroups will emit the JSON that feeds the tab.

S5 — Coding quality (aspirational, held-out)

• Macro-F1 on HCC code suggestion.
• Per-HCC precision / recall / F1 bar chart.
• Mean RAF captured vs ground-truth RAF.
• Reported via eval_harness/cli.py once a labeled Synthea held-out set exists.

Calibration

• Reliability diagram (10 bins).
• Brier score, ECE.
• Only meaningful once the calibrator is fit, not while it's an identity pass-through.

Over-coding audit (built-in because the brief asks for it)

• For each candidate the pipeline marks LOW risk, does the held-out label confirm MEAT actually supports it? Reports audit_false_positive_rate.

4. Where the numbers come from

eval_harness/cli.py subcommands:

Subcommand	What it does	Status
regression	Run the fixture suite, emit pass/fail report	✅ Implemented
faithfulness	Assert every MEAT quote is a byte-exact substring of the source	✅ Implemented
subgroups	Sliced metrics (age, length, specialty, frequency)	Stub
calibration	Reliability diagram + Brier + ECE	Stub

Run any subcommand with --help for options.

5. What "good enough to ship v1" looks like

From AGENT_BRIEF.md §3 in priority order:

• S1 faithfulness — 100% substring, ≥ 0.8 NLI entailment.
• S2 latency — p95 < 8s on free CPU tier with the gallery precomputed.
• S3 subgroups — rendered in the UI, not hidden in a notebook.
• S4 30-second recruiter skim — clear clinical value without ML jargon.
• S5 macro-F1 > 0.55 on held-out HCC suggestion, OR an honest report of a lower number with a clear explanation in the model card.

If we undershoot S5, ship with the honest number rather than extending the timeline — the portfolio signal is auditable + calibrated, not SOTA F1.