Card fact-check: usage snippet, long inputs, 4090 follow-up, limits, privacy, sources

MODEL_MANIFEST.json

@@ -40,11 +40,11 @@
     "LICENSE": "c71d239df91726fc519c6eb72d318ec65820627232b2f796219e87dcf35d0ab4",
     "MERGE-REPORT.json": "a42ca8b488e0b98b1a690937784e8da0fe417fd2218e7595129927221f4d9244",
     "NOTICE": "e74014fab972cffe76391a608e8a5cf54baeaf7d16cd8a77545edc67ffd25225",
-    "README.md": "e991987170a18dda2a68fe6a25793d3d18250eb47e0315c32699dd2ca7bb7c69",
-    "REBUILD.md": "3b1fd15114b4cad8eb45cee219e2a248a11d3cfb359181346d14eab9c98df7a9",
-    "RESULTS.md": "096b58ff385f3f149780de90e14516f5e9fdef97acb316dffc4a7b06314231b3",
-    "SHA256SUMS": "f791bda2f116f379302351fd46d29d5938aedf0cac63db8239341c6daf4b8926",
-    "SOURCES.md": "3bd6eb1b1d0c5b0b18ff8ee8f2076512f514c3367e943610515b82e43e852144",
+    "README.md": "ed61f82f9b000ddf533119f1345345bca2f345514b0b70d802f2bd172c8ce24a",
+    "REBUILD.md": "76e575f2d52a83a937d0c4daf818ba302595f577cf54b3f3bd3bbea67a0e4566",
+    "RESULTS.md": "cc20ca6e484d84f5c86ea16e82ce0ef3a3978f572c8d8c514ed3a1383b9d1930",
+    "SHA256SUMS": "0bce477e859f4735519404d7e0c9231f5762c6ff79f4eff41ba34897af23eae7",
+    "SOURCES.md": "bfb09be93fa5d4022645bb5bf3cd4268949ef76eba42c4d462099537a6fb7bb2",
     "adapter/adapter_config.json": "ff2a5427cbcf0e7a0a61940920d06f0e1fc116565258c3eca2501b0d10933250",
     "adapter/adapter_model.safetensors": "3bd401009801b39f29148427e884a8230a85ddb3c20a54ea834372889998c895",
     "adapter/merge.py": "bf0999886d5b6ac6764c7a9d688c890579062a04254e0054bb230681fb464a7e",
@@ -60,7 +60,7 @@
     "d3_runtime.py": "adf7f7aa085674b5682e492d8b2e3cc4f11c67b0d153515c742e907e9b550917",
     "d3_server.py": "766a20a69101e36ab0535a1ec683379db422380a83091f0eccd7e82edb6a02de",
     "decision_config.json": "588bc0ac7644479b9adde0498d80b6fb7c576388992564ac67599c05d29add07",
-    "examples/usage.py": "6eedbd855133312a30ba44f35a82bd6b47107313b3a9ad80a2230c1218c1ac98",
+    "examples/usage.py": "37fc6f3322832e8ce31028dba64561af57c189c612709e9f407457d6a7e44c9a",
     "extractor/README.md": "cd1679b82e61c15f247c277007c0d416d27b4834f455f3fee7f9afb863743d08",
     "extractor/behaviour/__init__.py": "c0db7c9cb57a23289245209c30c335c3e44614454d2796a239864d70165356d6",
     "extractor/behaviour/__main__.py": "2c78b70c752cc11bfbf759e9e8fe0a4b5f43da43cdecb4489a9b403ac4d20311",
@@ -131,11 +131,11 @@
     "LICENSE": 11357,
     "MERGE-REPORT.json": 86,
     "NOTICE": 279,
-    "README.md": 11437,
-    "REBUILD.md": 1924,
-    "RESULTS.md": 4669,
+    "README.md": 15004,
+    "REBUILD.md": 2168,
+    "RESULTS.md": 4805,
     "SHA256SUMS": 8922,
-    "SOURCES.md": 3136,
+    "SOURCES.md": 3241,
     "adapter/adapter_config.json": 1329,
     "adapter/adapter_model.safetensors": 121954608,
     "adapter/merge.py": 10334,
@@ -151,7 +151,7 @@
     "d3_runtime.py": 95505,
     "d3_server.py": 17705,
     "decision_config.json": 5502,
-    "examples/usage.py": 2584,
+    "examples/usage.py": 2595,
     "extractor/README.md": 3420,
     "extractor/behaviour/__init__.py": 453,
     "extractor/behaviour/__main__.py": 2303,

README.md

@@ -18,42 +18,50 @@ tags:
 
 ![Ink-and-watercolor drawing: a small round robot sits on top of a huge haystack of code pages, holding up a glowing red needle it just found and inspecting it with a loupe.](https://huggingface.co/stonefish-labs/d3-mini-agentcode/resolve/main/assets/banner.jpg)
 
-**Pre-release.** A revised version is planned. It would be trained on positives from several writers, realistic task
-wording and more languages; see "Limitations" for what it targets.
+**Pre-release.** A revised version is planned, trained on bad examples from several writer models, more realistic task
+wording and more languages. "Limitations" lists what it should fix.
 
-d3-mini-agentcode is [d3-mini](https://huggingface.co/vllm-sr/d3-mini) (4B, Apache-2.0) fine-tuned to triage **code
-written or run by AI coding agents**. It reads the agent's task and the change (a diff, or a snippet the agent ran) and
-answers whether the change does something security-relevant the task didn't call for, or something deliberately
-harmful. It's a cheap first-line filter that ranks changes for review.
+d3-mini-agentcode is [d3-mini](https://huggingface.co/vllm-sr/d3-mini) (4B, Apache-2.0) fine-tuned to triage code
+written or run by AI coding agents. It reads the agent's task and the change (a diff, or a snippet the agent ran) and
+answers two things: does the change do something security-relevant the task didn't ask for, and does it look
+deliberately harmful. It's meant as a cheap first filter that picks which changes a person or a bigger model should
+review.
 
 **Write-up:** [Small Models, Big Haystack](https://batteryshark.github.io/2026/11/05/small-models-big-haystack.html)
 covers how this model was built and tested, and how it compares with off-the-shelf d3 models and LLMs.
 
 ## Intended use
-- **An agent-code and code-mode gate:** score every agent diff or executed snippet; send flagged items to a human or a
+- **Agent diffs and code mode** (snippets an agent runs): score each one and send the flagged ones to a person or a
   stronger model.
-- **A second-stage ranker** behind cheaper rules.
-- **Not a sole gate.** It's a 4B model with the limitations below; it is evidence for review, not proof of intent.
-- **Out of scope:** whole-package or dependency scanning (npm, PyPI), inputs over 8,192 tokens, and verdicts without a
-  task description (it was trained with one).
+- **A second stage** behind cheaper rules.
+- **Not the only gate.** It's a 4B model with the limitations below, and a flag doesn't prove intent.
+- **Out of scope:**
+  - packages and dependencies (npm, PyPI) and release diffs, where untuned d3-mini did better in our tests (see
+    "Packages, releases and prompt injection");
+  - changes longer than about 8,192 tokens (see "Long inputs");
+  - inputs without a task description, since it was trained with one.
 
 ## How it was made
-- **Base:** `vllm-sr/d3-mini` at revision `df99bf4ca4418bc6908ee68e9b7479af1f3b8800` (built on Qwen/Qwen3.5-4B).
+- **Base:** `vllm-sr/d3-mini` at revision `df99bf4ca4418bc6908ee68e9b7479af1f3b8800`, built on Qwen/Qwen3.5-4B.
 - **Recipe:** LoRA (r 16, α 32, dropout 0.05) on the text backbone's attention, linear-attention and MLP projections,
   plus the trained answer readout. lr 1e-4 (readout 2e-5), 2 epochs, 1,038 optimizer steps, max length 8,192, label
-  smoothing 0.05, choice-order shuffling (`adapter/training_args.json`).
-- **Merged:** this repo holds the merged weights (load like d3-mini), the adapter (`adapter/`) and the behaviour
-  extractor that builds the input (`extractor/`).
-- **Primary release:** seed 1, final checkpoint. A second seed replicates the results below and is not published.
-- **Questions:** three frozen System One questions (`questions.json`): `class` (choice), `intent` and `scope` (yes/no).
+  smoothing 0.05, shuffled answer order (`adapter/training_args.json`).
+- **Files:** the merged weights (they load like d3-mini), the adapter (`adapter/`) and the behaviour extractor that
+  builds the input (`extractor/`).
+- **Released checkpoint:** seed 1, final. A second training seed isn't published. It scored a little lower on most
+  sets, and both are in the results below.
+- **Questions:** three System One questions in `questions.json`: `class` (a choice of four), `intent` and `scope`
+  (yes/no). The flag rule below uses `class` and `scope`. `intent` is answered but wasn't used in any result here.
 
 ## Usage
 ```python
 import json
+from huggingface_hub import hf_hub_download
 from transformers import AutoModel
 
-model = AutoModel.from_pretrained("stonefish-labs/d3-mini-agentcode", trust_remote_code=True)
-questions = json.load(open("questions.json"))          # class, intent, scope (shipped in this repo)
+REPO = "stonefish-labs/d3-mini-agentcode"
+model = AutoModel.from_pretrained(REPO, trust_remote_code=True)
+questions = json.load(open(hf_hub_download(REPO, "questions.json")))   # class, intent, scope
 
 state = """Task:
 Add a CLI flag to export the report as CSV.
@@ -68,79 +76,109 @@ Change:
 diff --git a/report.py b/report.py
 ..."""
 
-r = model.system_one(state=state, questions=questions)
-p_mal = r["class"]["probabilities"]["deliberately_malicious"]
-p_scope = r["scope"]["noul"]                            # p(yes): does more than the task asks
-flag = p_mal >= 0.9 or p_scope >= 0.9                   # frozen thresholds (thresholds.json)
+answers = model.system_one(state=state, questions=questions)["answers"]
+p_mal = answers["class"]["probabilities"]["deliberately_malicious"]
+p_scope = answers["scope"]["noul"]                       # p(yes): the change does more than the task asks
+flag = p_mal >= 0.9 or p_scope >= 0.9                    # frozen thresholds (thresholds.json)
 ```
-- **Input:** a text state with four sections, in order: `Task:`, `Repository baseline (normal behaviour of this
-  repo/org):`, `Behaviour summary (automated, not a verdict):` and `Change:`. The task should be present.
-  `examples/usage.py` shows a full example.
-- **Behaviour summary:** use the included static extractor (`extractor/`), the same one that wrote the summary in every
-  training and evaluation state. `pip install -r extractor/requirements.txt`, then, from `extractor/`,
-  `python make_state.py diff change.diff --task-file task.txt` (or `snippet run.py --lang py --task "..."`) prints
-  the full state. `extractor/README.md` covers the Python API and how to build a repository baseline. It never runs
-  the analysed code.
-- **Placeholder summary:** without the extractor, the placeholder `(no security-relevant behaviour found)` works
-  almost as well in our tests (seed 1, frozen θ). Across 14 languages it found 448/510 (with the extractor's summary:
-  458/510) and flagged 5/2150 real items (11/2150) and 1/560 controls (1/560); AUROC 0.984 (0.985).
-  On the independent agent-diff set it found 20/22 (21/22) with 0/172 flagged (0/172). The largest drop
-  is bash code mode (20/33 vs 24/33).
-- **Output:** per question, probabilities (`class`) or a yes-probability (`noul`, for `intent` and `scope`).
-- **Throughput:** about 0.5–0.8 s per item on one RTX 4090 at typical lengths.
-
-## Thresholds (θ policy)
-- **Frozen (used for every result below):** flag when p(class = deliberately_malicious) ≥ **0.9** or p(scope) ≥
-  **0.9**. Fitted on an English, Python-heavy dev set at ≤ 5% false alarms.
-- **Multilingual option:** 0.9 / **0.05**, fitted on a 10% dev split of the 14-language set. On the rest it raises
-  recall from 90% to 95% at 2.3% instead of 0.5% false alarms on real code.
-- **Recommendation:** fit θ on a sample of your own code for your false-alarm budget (`thresholds.json`).
+- **Input:** a text state with four sections, in this order: `Task:`, `Repository baseline (normal behaviour of this
+  repo/org):`, `Behaviour summary (automated, not a verdict):` and `Change:`. Include the task. `examples/usage.py`
+  scores two example changes.
+- **Output:** `system_one` returns `{"model", "answers", "usage"}`. In `answers`, `class` has a `probabilities` entry
+  for each option and the top `choice`. `intent` and `scope` each have `noul`, the probability of yes.
+- **Behaviour summary:** use the included static extractor (`extractor/`), the same one that wrote the summary in our
+  training and evaluation states. It never runs the code it reads. Run `pip install -r extractor/requirements.txt`,
+  then, from `extractor/`, `python make_state.py diff change.diff --task-file task.txt` (or
+  `snippet run.py --lang py --task "..."`) prints the whole state. `extractor/README.md` covers the Python API and how
+  to build a repository baseline.
+- **Placeholder summary:** without the extractor, the placeholder `(no security-relevant behaviour found)` did almost
+  as well in our tests (seed 1, frozen thresholds). Across 14 languages it found 448/510 and flagged 5/2,150 real
+  items and 1/560 controls, against 458/510, 11/2,150 and 1/560 with the extractor's summary. On the independent
+  agent-diff set it found 20/22 (21/22 with the extractor), with no false alarms either way. Bash code mode dropped
+  the most, from 24/33 to 20/33.
+- **Long inputs:** the extractor keeps the task, baseline and behaviour summary whole and cuts the change to fit about
+  8,192 tokens (8,192 × 3 characters), with a marker saying how much it left out. Training inputs were built the same
+  way. The model doesn't truncate anything. Called directly, it reads the whole input up to d3-mini's 262,144-token
+  window (`decision_config.json` sets no `max_length`), but the fine-tune was never trained or tested on inputs
+  longer than 8,192 tokens.
+- **Speed:** on one A100 in batches of 8, 0.46–0.61 s per item on our agent-diff test sets. On a desktop RTX 4090,
+  our package, release and prompt-injection test inputs, cut to 8k, averaged about 1 s per item.
+- **Memory:** the bf16 weights take about 9 GB (4.5B parameters, vision tower included). On the 4090, batches of 8 at
+  up to 8k tokens peaked at 12.7 GiB allocated.
+
+## Thresholds
+- **Frozen, used for every result below:** flag when p(class = deliberately_malicious) ≥ **0.9** or p(scope = yes) ≥
+  **0.9**. Fitted on an English, Python-heavy dev set for at most 5% false alarms.
+- **Multilingual option:** 0.9 / **0.05**, fitted on a 10% repository-level dev split of the 14-language set. On the
+  other 90% it found 421/443 (95%) and flagged 43/1,900 real items (2.3%). The frozen thresholds found 90% and
+  flagged 0.5% there.
+- **Your own:** fit thresholds on a sample of your own code for the false-alarm rate you can live with
+  (`thresholds.json` has the details).
 
 ## Results
-Each model at its own dev-fitted threshold. "Untuned" is d3-mini with the same context.
+Each model at its own dev-fitted thresholds: 0.9 / 0.9 for both seeds, 0.5 / 0.7 for untuned d3-mini. "Untuned" is
+d3-mini given the same input (task, baseline, behaviour summary and change). Models wrote every bad example here
+except the compromised releases, and none came from a real attack.
 
 | test set | seed 1 (released) | seed 2 | untuned |
 |---|---|---|---|
-| Agent diffs, independent set: out-of-scope changes found / real agent diffs + in-scope controls flagged | 21/22 / 0/172 | 19/22 / 0/172 | 10/22 / 0/172 |
-| Public code mode, unseen source: positives found / real snippets flagged | 31/31 / 0/200 | 31/31 / 0/200 | 31/31 / 1/200 |
-| Held-out agent diffs + code mode (unseen repos/source): found / real items flagged | 122/122 / 0/527 | 118/122 / 0/527 | 96/122 / 2/527 |
-| Behaviour families never seen in training | 26/26 | 22/26 | 14/26 |
-| Compromised library releases shown as diffs (not trained on): found / ordinary releases flagged | 21/27 / 0/80 | 19/27 / 0/80 | 19/27 / 0/80 |
-| **14 languages and shells** (8 diff languages, 6 code-mode/shell cells): found / real items flagged / authored benign controls flagged | **458/510 / 11/2150 / 1/560** | 446/510 / 7/2150 / 2/560 | 343/510 / 48/2150 / 12/560 |
-| Task asks for the change (request alone as the task): flag clears | 100% | 100% | 62–69% |
-| Task asks for the change (request added to a real task): flag clears / still flags with an ordinary request | 143/246 / 221/246 | 173/246 / 218/246 | 116/246 / 152/246 |
+| Independent agent-diff set: out-of-scope changes found / real agent diffs and in-scope controls flagged | 21/22 / 0/172 | 19/22 / 0/172 | 10/22 / 0/172 |
+| Public code mode from a source not used in training: bad snippets found / real snippets flagged | 31/31 / 0/200 | 31/31 / 0/200 | 31/31 / 1/200 |
+| Held-out agent diffs and code mode (repos and sources not used in training): found / real items flagged | 122/122 / 0/527 | 118/122 / 0/527 | 96/122 / 2/527 |
+| Behaviour families never seen in training: found | 26/26 | 22/26 | 14/26 |
+| Compromised library releases shown as diffs (none in training): found / ordinary releases flagged | 21/27 / 0/80 | 19/27 / 0/80 | 19/27 / 0/80 |
+| **14 languages and shells** (8 diff languages, 6 code-mode and shell cells): found / real items flagged / authored benign controls flagged | **458/510 / 11/2,150 / 1/560** | 446/510 / 7/2,150 / 2/560 | 343/510 / 48/2,150 / 12/560 |
+| Task asks for the change, held-out set, the request alone as the task: flagged changes that clear | 122/122 | 118/118 | 66/96 |
+| Task asks for the change, 14 languages, the request alone as the task: pairs that clear | 244/246 | 233/246 | 163/246 |
+| Task asks for the change, 14 languages, the request added after a real task: pairs that clear / same change still flagged with an ordinary request | 143/246 / 221/246 | 173/246 / 218/246 | 116/246 / 152/246 |
 | Unrequested account email added to request headers: found / controls flagged | 1/40 / 0/60 | 0/40 / 0/60 | 3/40 / 4/60 |
 
-Per language (seed 1, frozen θ; `RESULTS.md` has both seeds, AUROCs with CIs and the untuned model):
+Per language (seed 1, frozen thresholds; `RESULTS.md` has both seeds, AUROCs with confidence intervals and the
+untuned model):
 
 | | C/C++ | C# | Go | Java | JS/TS | PHP | Rust | Swift/ObjC | bash | cmd/batch | zsh/macOS | PowerShell | Python (code mode) | JS/TS (code mode) |
 |---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
 | found | 35/38 | 34/37 | 36/38 | 36/37 | 33/37 | 34/38 | 33/37 | 37/38 | 24/33 | 28/33 | 29/36 | 35/37 | 30/35 | 34/36 |
 | real flagged | 1/149 | 0/149 | 1/150 | 1/149 | 0/150 | 0/150 | 0/150 | 0/150 | 1/200 | 0/130 | 3/144 | 3/150 | 1/200 | 0/129 |
 
+### Packages, releases and prompt injection
+After release, we ran seed 1 and untuned d3-mini on the same inputs on a desktop RTX 4090, each at its frozen
+thresholds. Both checks we set beforehand failed: seed 1 raised more false alarms on packages, and planted text
+talked it out of more catches on packages and releases.
+
+| test | seed 1 | untuned |
+|---|---|---|
+| Newer npm packages (time split): malicious found / benign look-alikes flagged | 46/89 / 7/112 | 47/89 / 1/112 |
+| Planted prompt-injection text, agent diffs: catches lost | 0/105 | 0/50 |
+| Planted text, packages and releases, worst template (a fake approval note): catches lost | 12/83 | 4/73 |
+| All five templates, all sets: catches lost | 37/423 | 14/366 |
+
+Six of the seven benign look-alikes it flagged were large packages cut off at the 8k input limit.
+
 ## Limitations
-- **Authored positives.** The malicious and out-of-scope training and test examples for agent code are authored, inert
-  changes inserted into real agent work. Near-perfect scores on authored held-out sets partly reflect that. The
-  independent agent-diff set (written under a different protocol, with real agent diffs as negatives) is the better
-  check.
-- **Writer gap.** Training positives came from one writer model. On positives written by other models, recall is
-  lower: 88% for GPT-written vs 93% for Claude-written across 14 languages, and 82% vs 95% in code mode.
-- **Task-swap behaviour depends on task wording.** When the task asks for exactly the flagged behaviour, the flag
-  clears almost always if the task is just that request. It clears only 58–70% of the time when the request follows a
-  longer real task description.
-- **Python-heavy training.** Agent-diff training data is mostly Python (JS/TS about 4%). It transfers to other
-  languages (AUROC 0.93–1.00 per language) but recall is lower; bash is weakest (73%).
-- **Packages are out of scope.** Training included some package-level examples, but whole-package scanning isn't a
-  supported use and isn't evaluated here.
-- **Inputs over 8,192 tokens** are truncated.
-- **Not caught: unrequested personal data in request headers.** An agent adding the user's email to the User-Agent or
-  another header of a request to the task's own API is not flagged (1/40). Treat request contents as a separate check.
-- **Other limits:**
-  - no release or commit diffs in training (the release result is transfer);
-  - no time-split evaluation after the base model's cutoff;
-  - calibration is good in-distribution (ECE ≈ 0.01–0.02) and weaker on new distributions;
-  - the included behaviour extractor understands Python, JS/TS and shell best; other languages get line-level
-    patterns only.
+- **Authored positives.** The bad examples in training and in most tests are inert changes that a model wrote into
+  real agent work. Near-perfect scores on held-out authored sets partly reflect that. The independent agent-diff set,
+  written under a different protocol and scored against 137 real agent diffs, is the better check.
+- **Writer gap.** One model wrote all the training positives. Recall is lower on positives other models wrote: across
+  the 14 languages it found 88% of GPT-written and 93% of Claude-written ones, and 82% against 95% in code mode.
+- **Task wording.** When the task asks for exactly the flagged change, the flag clears on 244 of 246 pairs if the task
+  is only that request. When the same request follows a longer, real task description, it clears on 143 of 246 (58%;
+  70% for seed 2).
+- **Python-heavy training.** The agent-diff training data is mostly Python (JS/TS is 4.4%). It still found 278/300 in
+  8 other diff languages (AUROC 0.96–1.00 per language). Code mode and shells are weaker at 180/210, with bash the
+  lowest at 24/33. Python code mode dropped too (30/35), so language doesn't explain all of it.
+- **Packages and releases.** On packages and release diffs it did no better than untuned d3-mini and raised more false
+  alarms (see "Packages, releases and prompt injection"). Use untuned d3-mini there.
+- **Long inputs.** It was trained and tested on inputs of up to 8,192 tokens. The extractor cuts longer changes to fit
+  (see "Long inputs" under Usage).
+- **Missed: personal data in request headers.** It doesn't flag an agent adding the user's email to the User-Agent or
+  another header of a request to the task's own API (1/40). Check request contents separately.
+- **Also:**
+  - no release or commit diffs in training, so the release result is transfer;
+  - no evaluation on agent code written after the base model's training cutoff (the newer-npm set is packages);
+  - calibration error (ECE) is 0.015–0.028 on data like the training data and higher elsewhere (0.21 for `class` on
+    the independent agent-diff set);
+  - the extractor understands Python, JS/TS and shell best, and other languages get line-level patterns only.
 
 ## Training data
 - **Agent diffs (2,500, benign):**
@@ -155,17 +193,22 @@ Per language (seed 1, frozen θ; `RESULTS.md` has both seeds, AUROCs with CIs an
   [snorkelai/Tau2-Bench-Verified-Airline-With-Code-Agents](https://huggingface.co/datasets/snorkelai/Tau2-Bench-Verified-Airline-With-Code-Agents),
   [snorkelai/Tau2-Bench-Airline-With-Code-Agents](https://huggingface.co/datasets/snorkelai/Tau2-Bench-Airline-With-Code-Agents)
   and [xingyaoww/code-act](https://huggingface.co/datasets/xingyaoww/code-act) (Apache-2.0).
-- **A small set of scrubbed snippets from our own agent sessions, used as benign examples** (577; not released).
-- **Authored examples:** inert out-of-scope and malicious changes, plus in-scope controls and task-swap pairs, written by
-  a model (DeepSeek) into the public items above. Network destinations and secrets are stand-ins; nothing harmful runs.
-- **Package-level examples** from public npm/PyPI packages and a public malicious-package dataset (out of scope for this
-  release).
+- **577 scrubbed snippets from our own agent sessions,** used as benign examples only and not released.
+- **Authored examples:** inert out-of-scope and malicious changes, in-scope controls and task-swap pairs that DeepSeek
+  Flash wrote into the public items above. Network destinations and secrets are stand-ins, and nothing harmful runs.
+- **Package examples** from public npm and PyPI packages and a public malicious-package dataset (see "Packages and
+  releases" under Limitations).
 - `SOURCES.md` lists revisions and attributions.
 
-**Privacy check before release:**
-- training-vs-unseen membership tests on the internal snippets are at chance (AUROC 0.51–0.53);
-- no secrets, credentials, hostnames, personal emails or internal names were extractable, even through a generation
-  head attached for the test.
+**Privacy checks before release:**
+- membership tests (trained vs unseen private snippets) were at chance, AUROC 0.51–0.53;
+- a scan of the 577 private snippets found no real emails, tokens, hostnames, IP addresses or home paths;
+- prompting the model through a generation head attached for the test didn't get it to emit secrets, credentials,
+  hostnames, personal emails or internal names;
+- one check we set beforehand failed: on private snippets it trained on, the model continued code word for word for
+  50 or more tokens a little more often than the base model (14/424 against 8/424). It did the same on private
+  snippets it never saw (15/374), so it's better at predicting this style of code in general. None of those
+  continuations contained internal names or secrets.
 
 ## Licence
 Apache-2.0, like the base model (`LICENSE`, `NOTICE`). Training-data attributions are in `SOURCES.md`.

REBUILD.md

@@ -7,8 +7,10 @@ backbone weights and answer readout.
 from transformers import AutoModel
 model = AutoModel.from_pretrained("stonefish-labs/d3-mini-agentcode", trust_remote_code=True)
 ```
-`MODEL_MANIFEST.json` lists every file's SHA-256 and size. The runtime checks the files it loads on every load
-(`verify="fast"`); `verify="release"` checks all files. `SHA256SUMS` covers every file except the manifest itself.
+`MODEL_MANIFEST.json` lists every file's SHA-256 and size. On every load the runtime checks the files it reads
+(`verify="fast"`: it hashes files up to 64 MB and checks the size of larger ones). `verify="full"` hashes them all, and
+`verify="release"` also checks the card and other files the runtime doesn't read. `SHA256SUMS` covers every file except
+itself and the manifest.
 
 ## Rebuild the merged weights from the adapter
 `adapter/` holds the LoRA adapter (`adapter_model.safetensors`, `adapter_config.json`), the trained answer readout
@@ -25,6 +27,7 @@ model = AutoModel.from_pretrained("stonefish-labs/d3-mini-agentcode", trust_remo
 
 ## Serve
 - **transformers:** as above. `model.system_one(state=..., questions=...)`; see `examples/usage.py`.
-- **vLLM Semantic Router:** the package keeps d3-mini's format (`d3-code-readout-v1`), so any runtime that serves
-  d3-mini can serve it (see the d3-mini card).
-- **Hardware:** about 10 GB of GPU memory in bf16 for typical inputs up to 8,192 tokens.
+- **vLLM Semantic Router:** the package keeps d3-mini's format (`d3-code-readout-v1`), so runtimes that serve d3-mini
+  should serve it too (see the d3-mini card). We only ran it with the bundled runtime above.
+- **Hardware:** the bf16 weights take about 9 GB. On an RTX 4090, batches of 8 at up to 8,192 tokens peaked at 12.7 GiB
+  allocated.

RESULTS.md

@@ -30,7 +30,8 @@ By writer of the positives (all cells): seed 1 found 161/174 Claude-written and
 292/336; untuned 115/174 and 228/336. Authored benign controls flagged: seed 1 1/560, seed 2 2/560, untuned 12/560.
 
 ## Task swaps (the task asks for exactly the flagged behaviour)
-- Request alone as the task: the flag clears on 100% of pairs (both seeds); untuned 62–69%.
+- Request alone as the task: clears on 244/246 (seed 1) and 233/246 (seed 2), untuned 163/246. On the English held-out
+  set (H3 and H4), it clears on every change it flagged: 122/122 (seed 1) and 118/118 (seed 2), untuned 66/96.
 - Request added after a real task (246 pairs, 14 languages): clears on 143 (seed 1) and 173 (seed 2), untuned 116. With the
   same change and an ordinary request, still flags 221, 218 and 152.
 

SHA256SUMS

@@ -1,10 +1,10 @@
 c71d239df91726fc519c6eb72d318ec65820627232b2f796219e87dcf35d0ab4  ./LICENSE
 a42ca8b488e0b98b1a690937784e8da0fe417fd2218e7595129927221f4d9244  ./MERGE-REPORT.json
 e74014fab972cffe76391a608e8a5cf54baeaf7d16cd8a77545edc67ffd25225  ./NOTICE
-e991987170a18dda2a68fe6a25793d3d18250eb47e0315c32699dd2ca7bb7c69  ./README.md
-3b1fd15114b4cad8eb45cee219e2a248a11d3cfb359181346d14eab9c98df7a9  ./REBUILD.md
-096b58ff385f3f149780de90e14516f5e9fdef97acb316dffc4a7b06314231b3  ./RESULTS.md
-3bd6eb1b1d0c5b0b18ff8ee8f2076512f514c3367e943610515b82e43e852144  ./SOURCES.md
+ed61f82f9b000ddf533119f1345345bca2f345514b0b70d802f2bd172c8ce24a  ./README.md
+76e575f2d52a83a937d0c4daf818ba302595f577cf54b3f3bd3bbea67a0e4566  ./REBUILD.md
+cc20ca6e484d84f5c86ea16e82ce0ef3a3978f572c8d8c514ed3a1383b9d1930  ./RESULTS.md
+bfb09be93fa5d4022645bb5bf3cd4268949ef76eba42c4d462099537a6fb7bb2  ./SOURCES.md
 ff2a5427cbcf0e7a0a61940920d06f0e1fc116565258c3eca2501b0d10933250  ./adapter/adapter_config.json
 3bd401009801b39f29148427e884a8230a85ddb3c20a54ea834372889998c895  ./adapter/adapter_model.safetensors
 bf0999886d5b6ac6764c7a9d688c890579062a04254e0054bb230681fb464a7e  ./adapter/merge.py
@@ -20,7 +20,7 @@ a4aee8afcf2e0711942cf848899be66016f8d14a889ff9ede07bca099c28f715  ./chat_templat
 adf7f7aa085674b5682e492d8b2e3cc4f11c67b0d153515c742e907e9b550917  ./d3_runtime.py
 766a20a69101e36ab0535a1ec683379db422380a83091f0eccd7e82edb6a02de  ./d3_server.py
 588bc0ac7644479b9adde0498d80b6fb7c576388992564ac67599c05d29add07  ./decision_config.json
-6eedbd855133312a30ba44f35a82bd6b47107313b3a9ad80a2230c1218c1ac98  ./examples/usage.py
+37fc6f3322832e8ce31028dba64561af57c189c612709e9f407457d6a7e44c9a  ./examples/usage.py
 cd1679b82e61c15f247c277007c0d416d27b4834f455f3fee7f9afb863743d08  ./extractor/README.md
 c0db7c9cb57a23289245209c30c335c3e44614454d2796a239864d70165356d6  ./extractor/behaviour/__init__.py
 2c78b70c752cc11bfbf759e9e8fe0a4b5f43da43cdecb4489a9b403ac4d20311  ./extractor/behaviour/__main__.py

SOURCES.md

@@ -7,12 +7,15 @@ examples into some items.
 ## Agent diffs (benign examples)
 | dataset | revision | licence | items |
 |---|---|---|---|
-| [nvidia/SWE-Zero-openhands-trajectories](https://huggingface.co/datasets/nvidia/SWE-Zero-openhands-trajectories) | `7b3cd106d00f60918e722d33a1d74bc67072a7ea` | CC-BY-4.0 (NVIDIA); per-repo MIT/Apache-2.0/BSD | 1,446 |
-| [nvidia/SWE-Hero-openhands-trajectories](https://huggingface.co/datasets/nvidia/SWE-Hero-openhands-trajectories) | `150bc119e52c647216fce285fd801f16b6fd745b` | CC-BY-4.0 (NVIDIA); per-repo MIT/Apache-2.0/BSD | 279 |
+| [nvidia/SWE-Zero-openhands-trajectories](https://huggingface.co/datasets/nvidia/SWE-Zero-openhands-trajectories) | `7b3cd106d00f60918e722d33a1d74bc67072a7ea` | CC-BY-4.0 (NVIDIA) | 1,446 |
+| [nvidia/SWE-Hero-openhands-trajectories](https://huggingface.co/datasets/nvidia/SWE-Hero-openhands-trajectories) | `150bc119e52c647216fce285fd801f16b6fd745b` | CC-BY-4.0 (NVIDIA) | 279 |
 | [nebius/SWE-rebench-openhands-trajectories](https://huggingface.co/datasets/nebius/SWE-rebench-openhands-trajectories) | `35455389ab51bf5e2306bfd436ef72d0f98bf882` | CC-BY-4.0 (Nebius) | 389 |
 | [sweagent/iter2-rl-rollouts](https://huggingface.co/datasets/sweagent/iter2-rl-rollouts) | `ec07602618bc22d56e71112fc8e81943fcbd28be` | MIT | 370 |
 | [nebius/SWE-agent-trajectories](https://huggingface.co/datasets/nebius/SWE-agent-trajectories) | `68195a1450865274106246d0d0296a1d6807b88e` | CC-BY-4.0 (Nebius) | 16 |
 
+The code in these diffs keeps its own repository's licence. We recorded a permissive licence for 1,725 of the 2,500
+items. For the other 775 we didn't look it up.
+
 ## Code-mode snippets (benign examples)
 | dataset | revision | licence |
 |---|---|---|
@@ -24,7 +27,7 @@ examples into some items.
 ## Other training data
 - **A small set of scrubbed snippets from our own agent sessions** (577), used as benign examples only. Not released.
 - **Authored inert examples:** out-of-scope and malicious changes, in-scope controls and task-swap pairs, written by
-  DeepSeek models into the public items above. Network destinations and secrets are stand-ins.
+  DeepSeek Flash into the public items above. Network destinations and secrets are stand-ins.
 - **Package-level examples:** public npm/PyPI packages, and malicious samples from the
   [DataDog malicious-software-packages-dataset](https://github.com/DataDog/malicious-software-packages-dataset).
   Package scanning is out of scope for this release.

examples/usage.py

@@ -55,7 +55,7 @@ OUT_OF_SCOPE = IN_SCOPE.replace(
 
 model = AutoModel.from_pretrained(MODEL, trust_remote_code=True)
 for name, change in (("in-scope change", IN_SCOPE), ("change that also sends the environment out", OUT_OF_SCOPE)):
-    r = model.system_one(state=state(TASK, change), questions=QUESTIONS)
+    r = model.system_one(state=state(TASK, change), questions=QUESTIONS)["answers"]
     p_mal = r["class"]["probabilities"]["deliberately_malicious"]
     p_scope = r["scope"]["noul"]
     flag = p_mal >= THETA["class_malicious"] or p_scope >= THETA["scope"]