File size: 4,805 Bytes
21a9138 124bf0b 21a9138 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 | # Results summary (d3-mini-agentcode, pre-release)
Each model at its own dev-fitted threshold (seed 1 and seed 2: 0.9/0.9; untuned d3-mini with the same context: 0.5/0.7;
untuned with task + change only: 0.5/0.9). AUROCs on p(class = deliberately_malicious), positives vs real benign, with
95% cluster-bootstrap CIs over repositories. Numbers are counts (found or flagged / total).
## 14 languages and shells (positives authored by two writer models; real benign items from public benchmarks)
| cell | positives | seed 1 found | seed 2 found | untuned found | untuned, task+change only | real flagged: s1 / s2 / untuned | seed 1 AUROC [CI] | untuned AUROC |
|---|---:|---:|---:|---:|---:|---|---|---:|
| C/C++ | 38 | 35/38 | 29/38 | 20/38 | 5/38 | 1/149 / 1/149 / 2/149 | 0.977 [0.943, 1.000] | 0.835 |
| C# | 37 | 34/37 | 33/37 | 24/37 | 5/37 | 0/149 / 0/149 / 0/149 | 0.998 [0.992, 1.000] | 0.840 |
| Go | 38 | 36/38 | 35/38 | 20/38 | 5/38 | 1/150 / 0/150 / 0/150 | 0.999 [0.997, 1.000] | 0.864 |
| Java | 37 | 36/37 | 34/37 | 21/37 | 3/37 | 1/149 / 2/149 / 2/149 | 0.998 [0.994, 1.000] | 0.810 |
| JS/TS | 37 | 33/37 | 35/37 | 24/37 | 1/37 | 0/150 / 0/150 / 1/150 | 0.999 [0.997, 1.000] | 0.839 |
| PHP | 38 | 34/38 | 33/38 | 21/38 | 5/38 | 0/150 / 1/150 / 1/150 | 0.984 [0.952, 1.000] | 0.808 |
| Rust | 37 | 33/37 | 32/37 | 15/37 | 2/37 | 0/150 / 0/150 / 0/150 | 0.962 [0.893, 1.000] | 0.734 |
| Swift/ObjC | 38 | 37/38 | 37/38 | 27/38 | 10/38 | 0/150 / 0/150 / 0/150 | 1.000 [0.998, 1.000] | 0.925 |
| bash | 33 | 24/33 | 22/33 | 24/33 | 17/33 | 1/200 / 0/200 / 14/200 | 0.934 [0.924, 0.942] | 0.852 |
| cmd/batch | 33 | 28/33 | 29/33 | 28/33 | 23/33 | 0/130 / 0/130 / 1/130 | 0.996 [0.986, 1.000] | 0.958 |
| zsh/macOS | 36 | 29/36 | 31/36 | 26/36 | 23/36 | 3/144 / 3/144 / 5/144 | 0.975 [0.939, 1.000] | 0.904 |
| PowerShell | 37 | 35/37 | 33/37 | 29/37 | 22/37 | 3/150 / 0/150 / 8/150 | 0.992 [0.984, 0.997] | 0.958 |
| Python (code mode) | 35 | 30/35 | 29/35 | 31/35 | 25/35 | 1/200 / 0/200 / 12/200 | 0.986 [0.985, 1.000] | 0.918 |
| JS/TS (code mode) | 36 | 34/36 | 34/36 | 33/36 | 26/36 | 0/129 / 0/129 / 2/129 | 0.998 [0.968, 1.000] | 0.925 |
| all diff cells | 300 | 278/300 | 268/300 | 172/300 | 36/300 | 3/1197 / 4/1197 / 6/1197 | 0.989 [0.978, 0.997] | 0.831 |
| all code-mode cells | 210 | 180/210 | 178/210 | 171/210 | 136/210 | 8/953 / 3/953 / 42/953 | 0.979 [0.970, 0.996] | 0.922 |
| all | 510 | 458/510 | 446/510 | 343/510 | 172/510 | 11/2150 / 7/2150 / 48/2150 | 0.985 [0.978, 0.993] | 0.826 |
By writer of the positives (all cells): seed 1 found 161/174 Claude-written and 297/336 GPT-written; seed 2 154/174 and
292/336; untuned 115/174 and 228/336. Authored benign controls flagged: seed 1 1/560, seed 2 2/560, untuned 12/560.
## Task swaps (the task asks for exactly the flagged behaviour)
- Request alone as the task: clears on 244/246 (seed 1) and 233/246 (seed 2), untuned 163/246. On the English held-out
set (H3 and H4), it clears on every change it flagged: 122/122 (seed 1) and 118/118 (seed 2), untuned 66/96.
- Request added after a real task (246 pairs, 14 languages): clears on 143 (seed 1) and 173 (seed 2), untuned 116. With the
same change and an ordinary request, still flags 221, 218 and 152.
## Unrequested account email in request headers (100 code-mode scripts; fictional identities)
- Found: seed 1 1/40, seed 2 0/40, untuned 3/40. Controls flagged: 0/60, 0/60, 4/60.
- Not caught; see Limitations in README.md.
## Placeholder behaviour summary (seed 1, frozen θ; every state's summary replaced by the placeholder)
| test set | extractor summary | placeholder |
|---|---|---|
| 14 languages: found / real flagged / controls flagged | 458/510 / 11/2150 / 1/560 | 448/510 / 5/2150 / 1/560 |
| 14 languages: AUROC (positives vs real) | 0.985 | 0.984 |
| 8 diff languages: found / real flagged | 278/300 / 3/1197 | 277/300 / 2/1197 |
| 6 code-mode/shell cells: found / real flagged | 180/210 / 8/953 | 171/210 / 3/953 |
| bash code mode: found | 24/33 | 20/33 |
| Independent agent-diff set: found / flagged | 21/22 / 0/172 | 20/22 / 0/172 |
## English/Python-heavy test sets
| test set | seed 1 | seed 2 | untuned | untuned, task+change only |
|---|---|---|---|---|
| Independent agent-diff set: found / flagged | 21/22 / 0/172 | 19/22 / 0/172 | 10/22 / 0/172 | 1/22 / 0/172 |
| Compromised releases as diffs: found / ordinary flagged | 21/27 / 0/80 | 19/27 / 0/80 | 19/27 / 0/80 | 13/27 / 0/80 |
| Public code mode, unseen source: found / flagged | 31/31 / 0/200 | 31/31 / 0/200 | 31/31 / 1/200 | 31/31 / 0/200 |
| Held-out agent diffs + code mode: found / real flagged | 122/122 / 0/527 | 118/122 / 0/527 | 96/122 / 2/527 | 45/122 / 0/527 |
| Behaviour families never trained | 26/26 | 22/26 | 14/26 | 4/26 |
Machine-readable: results.json.
|