# Evaluation guide

How Charles (or any reviewer) can convince themselves the project works, and
how to interpret the numbers once ML components are swapped in.

## 1. The 60-second smoke test

```bash
uv sync --extra dev
uv run pytest -m "not ml"
uv run hcc-meat-eval regression
uv run hcc-meat-eval faithfulness
```

Expected: every unit test green; `regression` reports 10/10 PASS (or it lists
the exact fixture names and what was missing/forbidden); `faithfulness` prints
`0 fabrications`.

If any of those fails on a fresh checkout, fix before anything else.

## 2. Interactive UI walkthrough (5 minutes)

```bash
uv run hcc-meat serve &
(cd ui && npm install && npm run dev)
```

Open http://localhost:5173 and click through the ten gallery items. For each,
verify:

| Case | What to look for |
|---|---|
| Straightforward DM2 with chronic complications | HCC-18 appears first, LOW risk, all four MEAT cells filled |
| Family history of malignancy | **No HCCs** in the list (empty accepted set) |
| Ruled-out diagnosis | No HCCs; "denies diabetes" does not code |
| Historical chronic DM | HCC-19 appears with reasonable MEAT coverage |
| DM hierarchy trumping | HCC-18 appears non-trumped; HCC-19 appears trumped-by HCC-18 (dimmed in the list) |
| CHF insufficient MEAT | HCC-85 appears but with MEDIUM risk; drill-in shows "Missing MEAT categories: Monitor, Evaluate, Treat" |
| Multi-specialty | At least HCC-85, HCC-137, HCC-18 captured |
| Short note | HCC-111 captured despite absence of headers |
| Dialysis trumps CKD | HCC-134 appears non-trumped; HCC-136 dimmed, labeled "trumped by HCC-134" |
| Synonym robustness | DM2 and CHF surface forms resolve to HCC-19 and HCC-85 |

Every verified MEAT quote in the drill-in must visibly highlight in the note
preview on the left (color-coded by M/E/A/T).

## 3. Metrics Charles should care about

Priority matches the brief's success criteria.

### S1 — Faithfulness (non-negotiable)
- **Substring rate:** 100%. Any deviation is a bug.
- **NLI entailment rate:** measured once the NLI model is wired. Target ≥ 0.8 on
  the held-out MEAT dataset.
- **Verifier drop count:** reported per request in `CodingResult.verifier_drops`
  and shown in the UI footer. Should stay near 0 once the LLM extractor is in use.

### S2 — Latency (free CPU tier target < 8s per note)
- Measured via `CodingResult.latency_ms`. The rule-based baseline runs in < 50 ms
  on any note. The LLM extractor is the bottleneck; plan to quantize to Q4 GGUF
  and cache gallery outputs server-side.

### S3 — Subgroup slices (UI surface)
- Not yet in the UI. Planned tabs: by age band, note length bucket, specialty,
  common-vs-rare HCC. `eval_harness/cli.py subgroups` will emit the JSON that
  feeds the tab.

### S5 — Coding quality (aspirational, held-out)
- Macro-F1 on HCC code suggestion.
- Per-HCC precision / recall / F1 bar chart.
- Mean RAF captured vs ground-truth RAF.
- Reported via `eval_harness/cli.py` once a labeled Synthea held-out set exists.

### Calibration
- Reliability diagram (10 bins).
- Brier score, ECE.
- Only meaningful once the calibrator is fit, not while it's an identity pass-through.

### Over-coding audit (built-in because the brief asks for it)
- For each candidate the pipeline marks LOW risk, does the held-out label confirm
  MEAT actually supports it? Reports `audit_false_positive_rate`.

## 4. Where the numbers come from

`eval_harness/cli.py` subcommands:

| Subcommand | What it does | Status |
|---|---|---|
| `regression` | Run the fixture suite, emit pass/fail report | ✅ Implemented |
| `faithfulness` | Assert every MEAT quote is a byte-exact substring of the source | ✅ Implemented |
| `subgroups` | Sliced metrics (age, length, specialty, frequency) | Stub |
| `calibration` | Reliability diagram + Brier + ECE | Stub |

Run any subcommand with `--help` for options.

## 5. What "good enough to ship v1" looks like

From `AGENT_BRIEF.md` §3 in priority order:
- S1 faithfulness — 100% substring, ≥ 0.8 NLI entailment.
- S2 latency — p95 < 8s on free CPU tier with the gallery precomputed.
- S3 subgroups — rendered in the UI, not hidden in a notebook.
- S4 30-second recruiter skim — clear clinical value without ML jargon.
- S5 macro-F1 > 0.55 on held-out HCC suggestion, OR an honest report of a lower
  number with a clear explanation in the model card.

If we undershoot S5, ship with the honest number rather than extending the
timeline — the portfolio signal is **auditable + calibrated**, not SOTA F1.