Model v0.3 results: 100% on held-out data, 36/36 report parts in a real Mac audit

README.md

@@ -197,7 +197,7 @@ Devseis builds the bundle with `python3 vulndb/build_bundle.py` (needs internet;
 | Collectors (Windows, Linux, macOS) | **0.2** (catalog 0.3) | macOS tested on a real Mac; Linux tested on Ubuntu 24.04 and Fedora 42 containers; Windows parse-checked only (needs a real Windows test). Known vulnerabilities: inventory only, matching comes with the offline vulnerability bundle |
 | Vulnerability matcher and bundle | 0.1 | Tested: this Mac (Chrome 154 → 58 critical/high, fixed in 155), Ubuntu 24.04 (OpenSSL), Rocky 9 (25 of 25 packages agree with dnf's own security list), Fedora reported as not covered |
 | Phone check (web) | **0.3** | Tested in a phone-sized window as iPhone (Safari 26) and Android (Chrome 141); needs testing on real phones |
-| Fine-tuned model | **v0.1 published**, v0.3 training | [Devseis/endpoint-auditor-0.5b](https://huggingface.co/Devseis/endpoint-auditor-0.5b) (+ q0f16 / q4f16_1 / q4f32_1 WebLLM builds), tag `v0.1`: 100% valid and grounded on held-out data vs 0% for the base model; used by the app and the phone check. v0.3 (phones, computed summary scores) is training and replaces it when evaluated |
+| Fine-tuned model | **v0.3 in use** (v0.1 kept) | [Devseis/endpoint-auditor-0.5b](https://huggingface.co/Devseis/endpoint-auditor-0.5b) (+ q0f16 / q4f16_1 / q4f32_1 WebLLM builds), tags `v0.1` and `v0.3`. v0.3 on 60 held-out examples (desktop and phones): 100% valid, 100% grounded, 100% exact fields; base model 0%. Used by the desktop apps and the phone check; in a real audit of a Mac it writes 36 of 36 report parts (v0.1: 22 of 36) |
 
 ## Test status (collector version 0.1.0)
 

docs/MODEL_LLM.md

@@ -65,6 +65,7 @@ wrong.
 | Base model | Qwen2.5-0.5B-Instruct (Apache-2.0), quantised q4f16 / q4f32 for WebLLM |
 | Base model without fine-tuning | 0% valid JSON, 0% grounded on 30 held-out examples: it wraps answers in Markdown, invents its own format and invents facts |
 | Fine-tuned v0.1 pilot | **100% valid JSON, 100% grounded, 87% exact fields** on the same 30 examples (findings and AI classification 100%; summaries grounded but score / top gaps not exact) |
+| **Fine-tuned v0.3 (in use)** | **100% valid JSON, 100% grounded, 100% exact fields** on 60 held-out v0.3 examples (43 findings, 8 summaries, 9 AI classifications, desktop and phones); base model 0% on the same examples. Final validation loss 0.0109 |
 | Fine-tuning | LoRA on CPU (deliberately possible on an ordinary Mac). **In use now: run 1 on dataset v0.1** (pilot); next run on v0.3. See [Fine-tuning versions](#fine-tuning-versions) |
 | Training data | `Devseis/endpoint-auditor-synthetic` (CC BY 4.0): synthetic devices, real CVEs from the vulnerability bundle, wording identical to the collectors and the phone page |
 | Evaluation | `training/evaluate.py`: base vs fine-tuned on held-out data (JSON validity, grounding, status / reference accuracy) |
@@ -103,7 +104,7 @@ date it was built with, so a dataset version stays reproducible.
 |---|---|---|---|---|
 | Base model (no fine-tuning) | — | — | — | Used by the app and phone page today; 0 of 30 answers grounded, so the built-in writer writes the report |
 | **auditor-0.5b v0.1 (pilot)** | **v0.1** (tag `v0.1`) | 988 of 29,000 (balanced sample; 12 longer than 2,048 tokens skipped) | Windows, Linux, macOS; 58 / 51 / 51 checks | **Trained** 2026-10-08 (124 steps, final validation loss 0.0173). Evaluated: 100% valid, 100% grounded, 87% exact fields vs 0% for the base model. Published as `Devseis/endpoint-auditor-0.5b` tag `v0.1` |
-| auditor-0.5b v0.3 | v0.3 (main) | 3,000 balanced, including phones (out of 39,500; none over 3,072 tokens) | Windows, Linux, macOS, iPhone, Android; 74 checks; real CVEs; evidence labels; computed summary scores | **Training** since 2026-10-08 21:20 (about 375 steps, about 17 hours). First version to ship |
+| **auditor-0.5b v0.3** | v0.3 (tag `v0.3`) | 3,000 balanced, including phones (out of 39,500; none over 3,072 tokens) | Windows, Linux, macOS, iPhone, Android; 74 checks; real CVEs; evidence labels; computed summary scores | **In use** by the desktop apps and the phone check since 2026-10-10. Trained 375 steps (final validation loss 0.0109). Held-out: 100% valid, 100% grounded, 100% exact fields (60 examples). WebLLM on the GPU: q0f16 8/8 and q4f16_1 8/8 grounded on phone cases. Published with tag `v0.3` in all four model repositories |
 | auditor-0.5b v0.4+ | v0.3 + expert-reviewed library + gold findings | more, on a GPU if available | same, better wording and priorities | Later |
 
 Dataset v0.2 is not trained on: it was superseded by v0.3 before a run started.
@@ -140,12 +141,17 @@ Each published model version names the dataset version it was trained on, so res
 
 | Build | Repository | Size | Used by | Measured (held-out prompts, WebLLM on an Intel Mac GPU) |
 |---|---|---|---|---|
-| q0f16 (unquantised) | [Devseis/endpoint-auditor-0.5b-q0f16-MLC](https://huggingface.co/Devseis/endpoint-auditor-0.5b-q0f16-MLC) | 1.0 GB | desktop app, when the GPU supports 16-bit maths | v0.1: 6 of 6 grounded, 23–78 s per answer |
-| q4f16_1 (4-bit) | [Devseis/endpoint-auditor-0.5b-q4f16_1-MLC](https://huggingface.co/Devseis/endpoint-auditor-0.5b-q4f16_1-MLC) | 290 MB | phone check | v0.1: 5 of 6 grounded with the answer schema (0 of 6 without); full phone flow: 11 of 16 report parts written by the model (v0.1 never saw phone data) |
+| q0f16 (unquantised) | [Devseis/endpoint-auditor-0.5b-q0f16-MLC](https://huggingface.co/Devseis/endpoint-auditor-0.5b-q0f16-MLC) | 1.0 GB | desktop app, when the GPU supports 16-bit maths | v0.1: 6 of 6 grounded (desktop cases), 23–78 s per answer · **v0.3: 8 of 8 grounded (phone cases), 16–58 s** |
+| q4f16_1 (4-bit) | [Devseis/endpoint-auditor-0.5b-q4f16_1-MLC](https://huggingface.co/Devseis/endpoint-auditor-0.5b-q4f16_1-MLC) | 290 MB | phone check | v0.1: 5 of 6 grounded with the answer schema (0 of 6 without); full phone flow 11 of 16 parts by the model · **v0.3: 8 of 8 grounded on phone cases** |
 | q4f32_1 (4-bit) | [Devseis/endpoint-auditor-0.5b-q4f32_1-MLC](https://huggingface.co/Devseis/endpoint-auditor-0.5b-q4f32_1-MLC) | 290 MB | GPUs without 16-bit maths | layout and format verified |
 
-Each repository is tagged per model version (`v0.1`, `v0.3`); the app picks the version with `MODEL_VERSION` in
-`app/renderer/model-core.js` and falls back to the base model if the download fails.
+**Real audit with the packaged Mac app (same Mac, same audit, non-elevated):** v0.1 wrote 22 of 36 report parts (the
+built-in writer filled 14: checks v0.1 never saw); **v0.3 wrote 36 of 36**. The app, built when v0.1 was current,
+switched to v0.3 by itself through `app/model-version.json`.
+
+Each repository is tagged per model version (`v0.1`, `v0.3`). The desktop apps and the phone check read the version to
+use from `app/model-version.json` on the Space (now `v0.3`), so installed apps switch versions without reinstalling;
+`MODEL_VERSION` in `app/renderer/model-core.js` is the offline fallback, and the base model is used if the download fails.
 
 Lessons that shaped this: