Update root adapter

README.md

@@ -22,6 +22,25 @@ tags:
 base_model: fastino/GLiNER2.5-Decide
 metrics:
   - accuracy
+model-index:
+  - name: haruspex
+    results:
+      - task:
+          type: text-classification
+          name: Security decision (boolean / choice / score)
+        dataset:
+          type: custom
+          name: security pack test split (188 rows)
+        metrics:
+          - type: accuracy
+            value: 0.8777
+            name: accuracy
+          - type: ece
+            value: 0.0353
+            name: ece
+          - type: brier
+            value: 0.0845
+            name: brier
 ---
 
 # TyrusRC/haruspex
@@ -36,8 +55,9 @@ Haruspex **reuses** the published decision encoder
 specializes it on security decisions. It does not train a decision model from scratch.
 
 - Base: `fastino/GLiNER2.5-Decide`.
-- Added: a typed decision contract, security specialization data, and a
-  split-conformal **selective-calibration** layer (accept/abstain against a risk budget).
+- Added: a typed decision contract, security specialization data, a certified
+  **selective-calibration** layer (accept/abstain against a risk budget), conformal
+  **prediction sets**, and a **novelty gate** that abstains on unfamiliar input.
 - Serves the TypeSafe `POST /v1/systemone` and native `/v1/evaluate` contracts.
 
 ## Intended use (security, both sides)
@@ -45,8 +65,10 @@ specializes it on security decisions. It does not train a decision model from sc
 - **Offensive (pentest / red-team):** is this template/probe relevant to the target?
   is this scanner hit a real finding? finding severity; verdict discipline
   (vulnerable / not-vulnerable / inconclusive); attack-path next step.
-- **Defensive (SOC / blue-team):** alert triage (true vs false positive), alert
-  severity/priority, detection-rule applicability, incident routing, CVE applicability.
+- **Defensive (SOC / blue-team):** alert and incident triage (true / benign / false
+  positive), alert severity/priority, detection-rule applicability, incident routing, CVE
+  applicability, ATT&CK tactic of Windows/Sysmon events. Raw Defender, Sentinel, Elastic
+  ECS and Windows event JSON is normalised before the model reads it.
 
 It is a classifier for bounded decisions — not a chatbot, not an exploit generator.
 
@@ -69,18 +91,34 @@ haruspex-serve --model TyrusRC/haruspex --port 3000      # POST /v1/systemone
 
 ## Calibration
 
-Fitted calibrator: accept threshold `0.999998`, target risk `0.2`, coverage `1.0` (n=129).
+Fitted calibrator: accept threshold `0.8`, target risk `0.2` at delta `0.1`, coverage on the calibration split `0.849057` (n=106), temperature `6.297`.
 
-A split-conformal selective layer gates each verdict: it accepts when the top-1
-confidence clears a threshold fitted to hold the error AMONG accepted items at a target
-risk, and abstains otherwise. Fit it with `python -m haruspex.calibrate`; evaluate
-coverage/risk with `python -m haruspex.eval`.
+Each verdict carries the full probability distribution over its options (temperature-
+scaled). A Learn-then-Test selective layer accepts a verdict only when its confidence
+clears a threshold certified on a held-out calibration split, so that with probability
+1 - delta the error among accepted verdicts stays within the target risk; otherwise it
+abstains with `"route": "escalate"`. A conformal prediction set lists the options the
+true answer is among, and a novelty gate abstains on inputs unlike the training data.
+Fit with `python -m haruspex.calibrate` and `python -m haruspex.ood`; evaluate with
+`python -m haruspex.eval`.
+
+## Evaluation
+
+Held-out test sets, never used for training, model selection or calibration:
+
+- **security pack test split (188 rows)**: accuracy 0.8777, ECE 0.0353, Brier 0.0845; with the calibrator it answers 0.8382 of rows at accuracy 0.931 and abstains on the rest.
+
+Comparisons against general LLMs are in the repo's `docs/benchmark.md`.
 
 ## Limitations
 
 - A specialist for the decision types above; novel free-form judgments belong to a
   general LLM.
-- Calibrated on the training mix — recalibrate for a very different domain.
-- Prediction SETS (not just abstention) await a GLiNER2 fix for per-label scores.
+- Calibrated on this domain's held-out split; recalibrate before relying on the
+  guarantee on your own data.
+- Prediction-set coverage is marginal over the calibration mix, not per input.
+- A calibrator fitted on a mix of domains holds its risk target on average over that
+  mix, not on each domain: check each test set's accepted accuracy under Evaluation,
+  and route a domain with its own adapter to that adapter.
 
 Apache-2.0. Repo + recipe: https://github.com/TyrusRC/haruspex

calibrator.json

@@ -1,7 +1,12 @@
 {
-  "threshold": 0.999998,
+  "threshold": 0.8,
   "target_risk": 0.2,
-  "achieved_risk": 0.007752,
-  "coverage": 1.0,
-  "n": 129
+  "achieved_risk": 0.088889,
+  "coverage": 0.849057,
+  "n": 106,
+  "delta": 0.1,
+  "temperature": 6.296538261026657,
+  "temperature_scope": "all",
+  "set_q": 0.898328,
+  "set_alpha": 0.1
 }

eval.json

@@ -1,7 +1,34 @@
 {
-  "accuracy": 0.8718,
-  "ece": 0.1076,
-  "brier": 0.1071,
-  "test_n": 156,
-  "dataset": "web held-out"
+ "n": 188,
+ "accuracy": 0.8777,
+ "accuracy_ci95": [
+  0.8231,
+  0.9171
+ ],
+ "brier": 0.0845,
+ "ece": 0.0353,
+ "ece_adaptive": 0.0512,
+ "p50_ms": 37.2,
+ "by_source": {
+  "cve_lookup": {
+   "n": 13,
+   "accuracy": 1.0
+  },
+  "model": {
+   "n": 173,
+   "accuracy": 0.8786
+  },
+  "model_declined": {
+   "n": 2,
+   "accuracy": 0.0
+  }
+ },
+ "score_mae": 0.3333,
+ "selective": {
+  "coverage": 0.8382,
+  "accepted_accuracy": 0.931,
+  "accepted_risk": 0.069
+ },
+ "test_n": 188,
+ "dataset": "security pack test split"
 }

final/adapter_config.json

@@ -36,21 +36,21 @@
   "revision": null,
   "target_modules": [
     "project_start.3",
-    "out_project.3",
-    "classifier.0",
-    "project_end.0",
-    "classifier.2",
+    "query_proj",
     "count_pred.0",
+    "projector.2",
+    "classifier.2",
+    "project_end.0",
     "count_pred.2",
-    "dense",
-    "projector.0",
-    "query_proj",
     "project_start.0",
-    "value_proj",
+    "classifier.0",
     "out_project.0",
-    "projector.2",
+    "value_proj",
+    "dense",
     "key_proj",
-    "project_end.3"
+    "out_project.3",
+    "project_end.3",
+    "projector.0"
   ],
   "target_parameters": null,
   "task_type": null,

final/adapter_model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:e83d73d06ab1fb36313b060a025900e54896bae406d3c072c25edcf28029a471
+oid sha256:6d0947bb4d17a7d949082514600510ca5e96b4c3572283048b1fbfd033fb3643
 size 31766440

ood.json

@@ -0,0 +1 @@
+{"alpha": 0.01, "dim": 2048, "cal_scores": [0.379686, 0.415911, 0.419723, 0.433176, 0.438229, 0.44846, 0.450452, 0.453237, 0.454173, 0.457001, 0.45802, 0.46607, 0.469339, 0.470751, 0.477218, 0.482065, 0.493416, 0.497699, 0.499437, 0.499709, 0.50009, 0.503256, 0.504262, 0.504686, 0.50469, 0.507481, 0.510609, 0.51405, 0.515571, 0.516528, 0.523065, 0.526748, 0.530391, 0.532597, 0.533156, 0.533582, 0.537567, 0.542701, 0.544381, 0.546583, 0.547582, 0.548526, 0.551505, 0.552766, 0.553284, 0.556115, 0.557339, 0.558132, 0.566669, 0.569751, 0.572319, 0.574581, 0.577278, 0.578974, 0.58284, 0.583429, 0.585577, 0.58988, 0.592589, 0.593117, 0.594803, 0.598521, 0.600373, 0.608617, 0.616313, 0.616559, 0.618441, 0.623389, 0.624405, 0.624429, 0.625114, 0.633323, 0.640539, 0.640734, 0.640944, 0.641478, 0.641721, 0.642206, 0.643017, 0.643252, 0.64503, 0.645254, 0.650759, 0.652288, 0.66018, 0.662207, 0.665508, 0.665862, 0.675725, 0.676999, 0.678275, 0.68905, 0.694161, 0.698891, 0.705238, 0.710807, 0.717208, 0.728479, 0.729983, 0.730946, 0.731003, 0.746029, 0.74889, 0.763978, 0.777754, 0.802531, 0.805824, 0.806336, 0.819453, 0.835415, 0.861898, 0.870694, 0.927479]}

ood_train.npy

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1ad1eab461aae248402f4a1c53bf4baa97c991d30806fedd585c139cc57ddea7
+size 1794176