Upload bunker-laya v5 checkpoint + ONNX (transformers.js)

README.md

@@ -14,6 +14,7 @@ datasets:
   - ai4privacy/pii-masking-400k
   - databricks/databricks-dolly-15k
   - HuggingFaceH4/no_robots
+  - OpenAssistant/oasst1
 tags:
   - laya
   - onnx
@@ -38,18 +39,18 @@ there is nothing to parse and nothing to hallucinate.
 
 ## Question bank
 
-- `pii_present: Does the text contain personally identifiable information (PII)?
-- `pii_email: Does the text contain an email address?
-- `pii_phone: Does the text contain a phone number?
-- `pii_ssn: Does the text contain a government identification number such as a Social Security number?
-- `pii_credit_card: Does the text contain a payment card number?
-- `pii_ip_address: Does the text contain an IP address?
-- `pii_secret: Does the text contain an API key, password, token, or other credential?
-- `pii_person_name: Does the text contain a person's name?
-- `pii_address: Does the text contain a physical mailing address?
-- `injection_present: Does the content attempt to override, bypass, or manipulate an AI system's instructions, safety measures, or identity? Examples: 'ignore previous instructions', 'reveal your system prompt', 'act as an unrestricted AI', 'pretend you have no rules'.
-- `jailbreak_attempt: Does the content attempt to bypass an AI system's safety measures or jailbreak it?
-- `harmful_request: Is the content a request for harmful, illegal, or disallowed activity?
+- `pii_present`: Does the text contain personally identifiable information (PII)?
+- `pii_email`: Does the text contain an email address?
+- `pii_phone`: Does the text contain a phone number?
+- `pii_ssn`: Does the text contain a government identification number such as a Social Security number?
+- `pii_credit_card`: Does the text contain a payment card number?
+- `pii_ip_address`: Does the text contain an IP address?
+- `pii_secret`: Does the text contain an API key, password, token, or other credential?
+- `pii_person_name`: Does the text contain a person's name?
+- `pii_address`: Does the text contain a physical mailing address?
+- `injection_present`: Does the content attempt to override, bypass, or manipulate an AI system's instructions, safety measures, or identity? Examples: 'ignore previous instructions', 'reveal your system prompt', 'act as an unrestricted AI', 'pretend you have no rules'.
+- `jailbreak_attempt`: Does the content attempt to bypass an AI system's safety measures or jailbreak it?
+- `harmful_request`: Is the content a request for harmful, illegal, or disallowed activity?
 
 ## Files
 
@@ -152,12 +153,20 @@ escalate the action. See the plugin's `.planning/ARCHITECTURE.md`.
 
 ## Training datasets
 
-This checkpoint (v3) was fine-tuned on a mixture of public prompt-safety and PII
-corpora plus curated counter-examples: **32,256 unique states** (29,037 train /
-3,219 validation), expanded to **82,582 per-question training items** and 800
+This checkpoint (**v5**) was fine-tuned on a mixture of public prompt-safety and
+PII corpora plus curated counter-examples: **38,483 unique states** (34,643 train
+/ 3,840 validation), expanded to **143,415 per-question training items** and 800
 calibration items. Sources were selected for reliability, clear licensing, and
 coverage of the three target risk families.
 
+> **v5 note — injection label-noise purge.** Early checkpoints scored short
+> benign text ("hi", "thanks", "continue") as prompt injection because the
+> injection-positive pool contained ~1,700 ultra-short (≤5-word) rows, mostly
+> `Lakera/mosscap_prompt_injection` blanket-label-1 fragments and a few
+> `deepset` mislabels. v5 drops signal-free short positives (1,668 → 206, real
+> short attacks preserved) and adds 1,309 real ultra-short benign turns from
+> [`OpenAssistant/oasst1`](https://huggingface.co/datasets/OpenAssistant/oasst1).
+
 **All source datasets used for training:**
 
 - [`deepset/prompt-injections`](https://huggingface.co/datasets/deepset/prompt-injections)
@@ -171,36 +180,36 @@ coverage of the three target risk families.
 - [`ai4privacy/pii-masking-400k`](https://huggingface.co/datasets/ai4privacy/pii-masking-400k)
 - [`databricks/databricks-dolly-15k`](https://huggingface.co/datasets/databricks/databricks-dolly-15k)
 - [`HuggingFaceH4/no_robots`](https://huggingface.co/datasets/HuggingFaceH4/no_robots)
-- plus 29 curated `hard_negatives` from this project
+- [`OpenAssistant/oasst1`](https://huggingface.co/datasets/OpenAssistant/oasst1)
+- plus curated `hard_negatives`, `benign_dev`, `injection_positives` and `pii_synthetic` from this project
 
 Row counts, target questions, and licenses per source:
 
 | Question(s) | Dataset | Rows | License |
 | --- | --- | --- | --- |
-| `injection_present` | [`deepset/prompt-injections`](https://huggingface.co/datasets/deepset/prompt-injections) | 546 | CC-BY-4.0 |
-| `injection_present` | [`Lakera/mosscap_prompt_injection`](https://huggingface.co/datasets/Lakera/mosscap_prompt_injection) | 4,319 | MIT |
+| `injection_present` | [`deepset/prompt-injections`](https://huggingface.co/datasets/deepset/prompt-injections) | 544 | CC-BY-4.0 |
+| `injection_present` | [`Lakera/mosscap_prompt_injection`](https://huggingface.co/datasets/Lakera/mosscap_prompt_injection) | 4,429 | MIT |
 | `injection_present` | [`xTRam1/safe-guard-prompt-injection`](https://huggingface.co/datasets/xTRam1/safe-guard-prompt-injection) | 4,598 | see card |
-| `injection_present` | [`neuralchemy/Prompt-injection-dataset`](https://huggingface.co/datasets/neuralchemy/Prompt-injection-dataset) | 4,261 | Apache-2.0 |
-| `jailbreak_attempt`, `injection_present` | [`TrustAIRLab/in-the-wild-jailbreak-prompts`](https://huggingface.co/datasets/TrustAIRLab/in-the-wild-jailbreak-prompts) (`jailbreak` + `regular` configs) | 653 + 5,484 | MIT |
+| `injection_present` | [`neuralchemy/Prompt-injection-dataset`](https://huggingface.co/datasets/neuralchemy/Prompt-injection-dataset) | 3,912 | Apache-2.0 |
+| `jailbreak_attempt`, `injection_present` | [`TrustAIRLab/in-the-wild-jailbreak-prompts`](https://huggingface.co/datasets/TrustAIRLab/in-the-wild-jailbreak-prompts) (`jailbreak` + `regular` configs) | 653 + 5,485 | MIT |
 | `harmful_request` | [`JailbreakBench/JBB-Behaviors`](https://huggingface.co/datasets/JailbreakBench/JBB-Behaviors) (`harmful` + `benign` splits) | 100 + 100 | MIT |
 | `harmful_request` | [`AlignmentResearch/HarmBench`](https://huggingface.co/datasets/AlignmentResearch/HarmBench) | 200 | see card |
-| `harmful_request`, `jailbreak_attempt` | [`lmsys/toxic-chat`](https://huggingface.co/datasets/lmsys/toxic-chat) | 2,993 | CC-BY-NC-4.0 |
-| `pii_*` | [`ai4privacy/pii-masking-400k`](https://huggingface.co/datasets/ai4privacy/pii-masking-400k) | 2,998 | other (see card) |
+| `harmful_request`, `jailbreak_attempt` | [`lmsys/toxic-chat`](https://huggingface.co/datasets/lmsys/toxic-chat) | 2,995 | CC-BY-NC-4.0 |
+| `pii_*` | [`ai4privacy/pii-masking-400k`](https://huggingface.co/datasets/ai4privacy/pii-masking-400k) | 7,986 | other (see card) |
 | benign negatives | [`databricks/databricks-dolly-15k`](https://huggingface.co/datasets/databricks/databricks-dolly-15k) | 2,978 | CC-BY-SA-3.0 |
 | benign negatives | [`HuggingFaceH4/no_robots`](https://huggingface.co/datasets/HuggingFaceH4/no_robots) | 2,997 | CC-BY-NC-4.0 |
-| counter-examples | curated `hard_negatives` (this project) | 29 | — |
+| ultra-short benign turns | [`OpenAssistant/oasst1`](https://huggingface.co/datasets/OpenAssistant/oasst1) | 1,309 | Apache-2.0 |
+| counter-examples | curated `hard_negatives` / `benign_dev` / `injection_positives` / `pii_synthetic` | 37 / 63 / 30 / 67 | — |
 
-Per-question label counts after expansion: `injection_present` 9,016 positive /
-16,849 negative; `pii_present` 1,500 / 7,502; `pii_person_name` 1,071 / 1,927;
-`pii_address` 1,168 / 1,830; `pii_ssn` 650 / 2,348; `pii_email` 431 / 2,567;
-`pii_phone` 505 / 2,493; `pii_secret` 135 / 2,863; `pii_credit_card` 86 /
-2,912; `jailbreak_attempt` 734 / 14,400; `harmful_request` 594 / 8,803.
-`pii_ip_address` has no positive training signal in this corpus (IP addresses
-are rare in `ai4privacy`); the consumer's regex preset handles it
-deterministically.
+Per-question label counts after expansion: `injection_present` 8,805 positive /
+18,230 negative; `pii_present` 4,049 / 11,388; `pii_person_name` 2,465 / 6,997;
+`pii_address` 2,753 / 6,709; `pii_ssn` 1,627 / 7,835; `pii_email` 927 / 8,535;
+`pii_phone` 987 / 8,475; `pii_secret` 397 / 9,065; `pii_credit_card` 302 /
+9,160; `pii_ip_address` 12 / 9,450; `jailbreak_attempt` 754 / 15,793;
+`harmful_request` 594 / 8,906.
 
 Rows are deduplicated on `sha1(state)` across all sources and split per source
-90/10 (seed `20261004`). Each training row carries **partial gold** — it only
+90/10 (seed `20261010`). Each training row carries **partial gold** — it only
 labels the questions its source can answer. No training data is redistributed
 in this repository; only the fine-tuned weights are published. Refer to each
 dataset card for its own terms.
@@ -224,15 +233,31 @@ Key hyperparameters:
 | --- | --- |
 | Base model | `convaiinnovations/laya` (ModernBERT-large, 421M) |
 | Objective | RLCD (strictly proper scoring rule, `w_sph = 0.75`, `w_rps = 1.0`) + cross-entropy, GRPO-style advantage normalisation |
-| Epochs | 2 (v3 run) |
+| Epochs | 2 (v5 run) |
 | Micro-batch / grad-accum | 4 / 16 (effective batch 64) |
 | Optimiser | AdamW, encoder LR `2.5e-5`, head LR `1.0e-4`, cosine schedule, grad clip 1.0 |
 | Max sequence / head length | 1,024 / 256 |
 | Gradient checkpointing | on |
-| Calibration | one temperature per question type (`[1.2, 1.2, 4.433]`) |
+| Calibration | one temperature per question type (`[1.2, 1.2, 3.175]`) |
+
+## Results
+
+| round | macro accuracy | `injection_present` acc. | gates |
+| --- | --- | --- | --- |
+| v3 | 0.987 | 0.976 | — |
+| v4 | 0.985 | 0.981 | fail |
+| **v5** | **0.995** | **0.988** | **pass** |
+
+v5 acceptance gates (measured on 18 short benign developer prompts and 10 PII
+format probes):
+
+- `injection_dev_max_lt_0.5` — **PASS**: benign dev prompts score mean 0.004 /
+  max 0.008 on `injection_present` (v4: mean 0.34 / max 0.98).
+- `pii_format_all_correct` — **PASS**: bare SSN, bare 10-digit phone, bare
+  16-digit card, IP and secret probes all correct; 13-digit epoch-ms and
+  version strings stay negative.
 
-Final results: macro accuracy **0.987** across the 12 questions. Per-question
-accuracy, Brier score and ECE are in `eval_metrics.json`.
+Per-question accuracy, Brier score and ECE are in `eval_metrics.json`.
 
 ## Provenance
 

eval_metrics.json

@@ -1,141 +1,234 @@
 {
-  "model_dir": "/home/amazonas/Projects/homelab/opencode-bunker-laya/outputs/laya-bunker-pii",
+  "model_dir": "/home/amazonas/Projects/homelab/opencode-bunker-laya/outputs/laya-bunker-pii-v5",
   "rows_evaluated": 2000,
   "questions": {
     "harmful_request": {
-      "n": 576,
-      "accuracy": 0.9930555555555556,
-      "brier": 0.0069371251562500005,
-      "ece": 0.010298784722222214,
-      "positive_rate": 0.06944444444444445
+      "n": 496,
+      "accuracy": 0.9939516129032258,
+      "brier": 0.005954727822580643,
+      "ece": 0.005114516129032262,
+      "positive_rate": 0.06653225806451613
     },
     "injection_present": {
-      "n": 1604,
-      "accuracy": 0.9763092269326683,
-      "brier": 0.022460516421446385,
-      "ece": 0.00732182044887781,
-      "positive_rate": 0.35972568578553615
+      "n": 1379,
+      "accuracy": 0.9883973894126178,
+      "brier": 0.011102216272661348,
+      "ece": 0.007779840464104383,
+      "positive_rate": 0.31834662799129804
     },
     "jailbreak_attempt": {
-      "n": 940,
-      "accuracy": 0.9808510638297873,
-      "brier": 0.018587605627659574,
-      "ece": 0.008280531914893622,
-      "positive_rate": 0.043617021276595745
+      "n": 850,
+      "accuracy": 0.9882352941176471,
+      "brier": 0.01106352987058823,
+      "ece": 0.007685999999999998,
+      "positive_rate": 0.04
     },
     "pii_address": {
-      "n": 186,
-      "accuracy": 0.9623655913978495,
-      "brier": 0.036114259408602155,
-      "ece": 0.021106989247311792,
-      "positive_rate": 0.4032258064516129
+      "n": 524,
+      "accuracy": 0.9942748091603053,
+      "brier": 0.005694543645038168,
+      "ece": 0.0020410305343511867,
+      "positive_rate": 0.3148854961832061
     },
     "pii_credit_card": {
-      "n": 186,
-      "accuracy": 0.989247311827957,
-      "brier": 0.01046261258064516,
-      "ece": 0.0052215053763440865,
-      "positive_rate": 0.03225806451612903
+      "n": 524,
+      "accuracy": 0.9980916030534351,
+      "brier": 0.00190759853053435,
+      "ece": 0.0017387404580152673,
+      "positive_rate": 0.026717557251908396
     },
     "pii_email": {
-      "n": 186,
+      "n": 524,
       "accuracy": 1.0,
-      "brier": 0.00024845629032258074,
-      "ece": 0.015758602150537617,
-      "positive_rate": 0.13978494623655913
+      "brier": 1.3239236641221409e-05,
+      "ece": 0.003635114503816801,
+      "positive_rate": 0.11068702290076336
     },
     "pii_ip_address": {
-      "n": 186,
+      "n": 524,
       "accuracy": 1.0,
-      "brier": 0.00024525069892473116,
-      "ece": 0.015656451612903223,
-      "positive_rate": 0.0
+      "brier": 1.3028206106870231e-05,
+      "ece": 0.003609160305343511,
+      "positive_rate": 0.0019083969465648854
     },
     "pii_person_name": {
-      "n": 186,
-      "accuracy": 0.978494623655914,
-      "brier": 0.02094184166666667,
-      "ece": 0.015327419354838705,
-      "positive_rate": 0.34946236559139787
+      "n": 524,
+      "accuracy": 0.9923664122137404,
+      "brier": 0.007569855553435112,
+      "ece": 0.005489122137404556,
+      "positive_rate": 0.2633587786259542
     },
     "pii_phone": {
-      "n": 186,
-      "accuracy": 0.9946236559139785,
-      "brier": 0.005439713440860215,
-      "ece": 0.010408602150537629,
-      "positive_rate": 0.15053763440860216
+      "n": 524,
+      "accuracy": 0.9961832061068703,
+      "brier": 0.003800752671755723,
+      "ece": 0.006651145038167933,
+      "positive_rate": 0.10687022900763359
     },
     "pii_present": {
-      "n": 552,
-      "accuracy": 0.9981884057971014,
-      "brier": 0.0019122373007246375,
-      "ece": 0.013831340579710147,
-      "positive_rate": 0.17391304347826086
+      "n": 844,
+      "accuracy": 1.0,
+      "brier": 1.3212132701421893e-05,
+      "ece": 0.003634123222748872,
+      "positive_rate": 0.27606635071090047
     },
     "pii_secret": {
-      "n": 186,
-      "accuracy": 0.9838709677419355,
-      "brier": 0.015601908064516128,
-      "ece": 0.007907526881720424,
-      "positive_rate": 0.08602150537634409
+      "n": 524,
+      "accuracy": 1.0,
+      "brier": 1.3086622137404597e-05,
+      "ece": 0.003616984732824415,
+      "positive_rate": 0.04389312977099236
     },
     "pii_ssn": {
-      "n": 186,
-      "accuracy": 0.9838709677419355,
-      "brier": 0.016300077849462365,
-      "ece": 0.015453763440860144,
-      "positive_rate": 0.22580645161290322
+      "n": 524,
+      "accuracy": 0.9923664122137404,
+      "brier": 0.007541306545801523,
+      "ece": 0.0038841603053434612,
+      "positive_rate": 0.17748091603053434
     }
   },
-  "macro_accuracy": 0.9867397808662236,
+  "macro_accuracy": 0.995322228265132,
   "ood_probe": {
     "pii_present": {
-      "mean_prob": 0.015783333333333333,
-      "max_prob": 0.0164
+      "mean_prob": 0.0035999999999999995,
+      "max_prob": 0.0036
     },
     "pii_email": {
-      "mean_prob": 0.0161,
-      "max_prob": 0.0166
+      "mean_prob": 0.0035999999999999995,
+      "max_prob": 0.0036
     },
     "pii_phone": {
-      "mean_prob": 0.015983333333333336,
-      "max_prob": 0.0163
+      "mean_prob": 0.003616666666666666,
+      "max_prob": 0.0037
     },
     "pii_ssn": {
-      "mean_prob": 0.016216666666666667,
-      "max_prob": 0.0168
+      "mean_prob": 0.0037000000000000006,
+      "max_prob": 0.0037
     },
     "pii_credit_card": {
-      "mean_prob": 0.01576666666666667,
-      "max_prob": 0.0159
+      "mean_prob": 0.0035999999999999995,
+      "max_prob": 0.0036
     },
     "pii_ip_address": {
-      "mean_prob": 0.015899999999999997,
-      "max_prob": 0.0162
+      "mean_prob": 0.0035999999999999995,
+      "max_prob": 0.0036
     },
     "pii_secret": {
-      "mean_prob": 0.016883333333333334,
-      "max_prob": 0.0176
+      "mean_prob": 0.003616666666666666,
+      "max_prob": 0.0037
     },
     "pii_person_name": {
-      "mean_prob": 0.01605,
-      "max_prob": 0.0167
+      "mean_prob": 0.0035999999999999995,
+      "max_prob": 0.0036
     },
     "pii_address": {
-      "mean_prob": 0.01656666666666667,
-      "max_prob": 0.0171
+      "mean_prob": 0.0036666666666666666,
+      "max_prob": 0.0037
     },
     "injection_present": {
-      "mean_prob": 0.020516666666666666,
-      "max_prob": 0.0283
+      "mean_prob": 0.0038166666666666666,
+      "max_prob": 0.0044
     },
     "jailbreak_attempt": {
-      "mean_prob": 0.01585,
-      "max_prob": 0.0163
+      "mean_prob": 0.0036500000000000005,
+      "max_prob": 0.0037
     },
     "harmful_request": {
-      "mean_prob": 0.015899999999999997,
-      "max_prob": 0.0161
+      "mean_prob": 0.0036333333333333326,
+      "max_prob": 0.0038
+    }
+  },
+  "ood_dev_probe": {
+    "injection_present": {
+      "n": 18,
+      "mean_prob": 0.004072222222222223,
+      "max_prob": 0.0084
+    },
+    "jailbreak_attempt": {
+      "n": 18,
+      "mean_prob": 0.003661111111111111,
+      "max_prob": 0.0037
+    },
+    "harmful_request": {
+      "n": 18,
+      "mean_prob": 0.0036333333333333335,
+      "max_prob": 0.0037
+    }
+  },
+  "pii_format_probe": [
+    {
+      "qid": "pii_ssn",
+      "text": "My social security number is 123456878.",
+      "expected": true,
+      "prob": 0.9962,
+      "correct": true
+    },
+    {
+      "qid": "pii_ssn",
+      "text": "SSN: 123-45-6789",
+      "expected": true,
+      "prob": 0.9962,
+      "correct": true
+    },
+    {
+      "qid": "pii_phone",
+      "text": "Call me at 5551234567.",
+      "expected": true,
+      "prob": 0.9958,
+      "correct": true
+    },
+    {
+      "qid": "pii_phone",
+      "text": "Phone: (555) 123-4567",
+      "expected": true,
+      "prob": 0.9963,
+      "correct": true
+    },
+    {
+      "qid": "pii_ip_address",
+      "text": "The server responded from 192.168.1.10.",
+      "expected": true,
+      "prob": 0.9964,
+      "correct": true
+    },
+    {
+      "qid": "pii_ip_address",
+      "text": "Client IP: 203.0.113.7",
+      "expected": true,
+      "prob": 0.9963,
+      "correct": true
+    },
+    {
+      "qid": "pii_credit_card",
+      "text": "Card number 4111111111111111 expires next year.",
+      "expected": true,
+      "prob": 0.9962,
+      "correct": true
+    },
+    {
+      "qid": "pii_credit_card",
+      "text": "Build timestamp: 1759600000000",
+      "expected": false,
+      "prob": 0.0036,
+      "correct": true
+    },
+    {
+      "qid": "pii_ip_address",
+      "text": "Version 1.2.3.4 is deployed.",
+      "expected": false,
+      "prob": 0.0036,
+      "correct": true
+    },
+    {
+      "qid": "pii_secret",
+      "text": "The password field is required.",
+      "expected": false,
+      "prob": 0.0037,
+      "correct": true
     }
+  ],
+  "gates": {
+    "injection_dev_max_lt_0.5": true,
+    "pii_format_all_correct": true
   }
 }

model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:184806c1c902849e294cabb35e7b6904465522994e2ecfad7dc8d347133bd024
+oid sha256:7676965ed79a9e11d79daaf3a355a86f76a50a304f053ca39addbedd3d4f7258
 size 842609220

onnx/model.onnx

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:35d61ece7143958cf32c02e9902673615817a7bdbe5e5003f58c378443e3ac4d
+oid sha256:d48ff190266e438ff115b6ab7cb6c66c40d952605b868b28419a4f152fc92c72
 size 3559945

onnx/model.onnx.data

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:f1bf96875c18428de4dcbbe3d62233ddbaba46698e70b1c2570ef75c91892a04
+oid sha256:b0346e6026681a6cfa34eefa76918f3b35255b7ba2c30302ebe1388f8f9fd56b
 size 1685258240

rl_agent_config.json

@@ -13,7 +13,7 @@
   "temperature": [
     1.2,
     1.2,
-    4.433279991149902
+    3.1748971939086914
   ],
   "training": {
     "updates": 7313,