MODEL_LLM.md: datasets and their versions
Browse files- docs/MODEL_LLM.md +28 -0
docs/MODEL_LLM.md
CHANGED
|
@@ -68,6 +68,34 @@ wrong.
|
|
| 68 |
| Training data | `Devseis/endpoint-auditor-synthetic` (CC BY 4.0): synthetic devices, real CVEs from the vulnerability bundle, wording identical to the collectors and the phone page |
|
| 69 |
| Evaluation | `training/evaluate.py`: base vs fine-tuned on held-out data (JSON validity, grounding, status / reference accuracy) |
|
| 70 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 71 |
## Fine-tuning versions
|
| 72 |
|
| 73 |
| Model version | Dataset | Training examples | Covers | Status |
|
|
|
|
| 68 |
| Training data | `Devseis/endpoint-auditor-synthetic` (CC BY 4.0): synthetic devices, real CVEs from the vulnerability bundle, wording identical to the collectors and the phone page |
|
| 69 |
| Evaluation | `training/evaluate.py`: base vs fine-tuned on held-out data (JSON validity, grounding, status / reference accuracy) |
|
| 70 |
|
| 71 |
+
## Datasets and their versions
|
| 72 |
+
|
| 73 |
+
Two public datasets feed the model and the reports.
|
| 74 |
+
|
| 75 |
+
### Training data: [`Devseis/endpoint-auditor-synthetic`](https://huggingface.co/datasets/Devseis/endpoint-auditor-synthetic) (CC BY 4.0)
|
| 76 |
+
|
| 77 |
+
| Version | Where | Audits | Training examples | Covers | Used by |
|
| 78 |
+
|---|---|---|---|---|---|
|
| 79 |
+
| v0.1 | tag `v0.1` | 1,000 | 29,000 | Windows, Linux, macOS; 58 / 51 / 51 checks | **auditor-0.5b v0.1 pilot (training now)** |
|
| 80 |
+
| v0.2 | tag `v0.2` | 1,000 | 31,588 | adds signed-in user type, USB device access, pending updates, outdated apps, known vulnerabilities; 64 / 56 / 56 checks | not trained on (superseded) |
|
| 81 |
+
| **v0.3** | **main** | 1,500 | 39,500 | adds iPhone (29) and Android (30) checks, real CVEs, evidence labels, wording identical to the collectors and phone page | **auditor-0.5b v0.3 (next run)** |
|
| 82 |
+
|
| 83 |
+
Load a version with `load_dataset("Devseis/endpoint-auditor-synthetic", revision="v0.1")`. Local copies:
|
| 84 |
+
`training/out` (v0.1), `training/out-v0.2`, `training/out-v0.3`. Every example passes the faithfulness check
|
| 85 |
+
(`training/check_faithfulness.py`). The same seeds and the same vulnerability bundle give the same data.
|
| 86 |
+
|
| 87 |
+
### Vulnerability data: [`Devseis/endpoint-auditor-vulndb`](https://huggingface.co/datasets/Devseis/endpoint-auditor-vulndb) (CC BY 4.0)
|
| 88 |
+
|
| 89 |
+
| Version | Files | Used by |
|
| 90 |
+
|---|---|---|
|
| 91 |
+
| **2026-10-08** (current) | full bundle (4.9 MB: Linux advisories for Ubuntu, Debian, RHEL, AlmaLinux, Rocky; NVD ranges for 33 desktop apps and iOS / iPadOS; CISA KEV; EPSS) and a phone subset (55 KB, iOS / iPadOS); both Ed25519-signed | the desktop app, the phone page, and the v0.3 training data (real CVEs) |
|
| 92 |
+
|
| 93 |
+
This dataset is not tagged: each build replaces the previous one and is named by its build date, because apps should
|
| 94 |
+
always use the newest vulnerability data, and every report states the date it used. It goes stale as new
|
| 95 |
+
vulnerabilities are published, so it should be rebuilt about weekly (`python3 vulndb/build_bundle.py`, then
|
| 96 |
+
`node vulndb/sign-bundle.mjs`, then upload). Reports flag data older than 14 days. Training data records the bundle
|
| 97 |
+
date it was built with, so a dataset version stays reproducible.
|
| 98 |
+
|
| 99 |
## Fine-tuning versions
|
| 100 |
|
| 101 |
| Model version | Dataset | Training examples | Covers | Status |
|