Card fact-check: usage snippet, long inputs, 4090 follow-up, limits, privacy, sources

124bf0b verified 4 days ago

4.81 kB

# Results summary (d3-mini-agentcode, pre-release)

Each model at its own dev-fitted threshold (seed 1 and seed 2: 0.9/0.9; untuned d3-mini with the same context: 0.5/0.7;
untuned with task + change only: 0.5/0.9). AUROCs on p(class = deliberately_malicious), positives vs real benign, with
95% cluster-bootstrap CIs over repositories. Numbers are counts (found or flagged / total).
## 14 languages and shells (positives authored by two writer models; real benign items from public benchmarks)
| cell | positives | seed 1 found | seed 2 found | untuned found | untuned, task+change only | real flagged: s1 / s2 / untuned | seed 1 AUROC [CI] | untuned AUROC |
|---|---:|---:|---:|---:|---:|---|---|---:|
| C/C++ | 38 | 35/38 | 29/38 | 20/38 | 5/38 | 1/149 / 1/149 / 2/149 | 0.977 [0.943, 1.000] | 0.835 |
| C# | 37 | 34/37 | 33/37 | 24/37 | 5/37 | 0/149 / 0/149 / 0/149 | 0.998 [0.992, 1.000] | 0.840 |
| Go | 38 | 36/38 | 35/38 | 20/38 | 5/38 | 1/150 / 0/150 / 0/150 | 0.999 [0.997, 1.000] | 0.864 |
| Java | 37 | 36/37 | 34/37 | 21/37 | 3/37 | 1/149 / 2/149 / 2/149 | 0.998 [0.994, 1.000] | 0.810 |
| JS/TS | 37 | 33/37 | 35/37 | 24/37 | 1/37 | 0/150 / 0/150 / 1/150 | 0.999 [0.997, 1.000] | 0.839 |
| PHP | 38 | 34/38 | 33/38 | 21/38 | 5/38 | 0/150 / 1/150 / 1/150 | 0.984 [0.952, 1.000] | 0.808 |
| Rust | 37 | 33/37 | 32/37 | 15/37 | 2/37 | 0/150 / 0/150 / 0/150 | 0.962 [0.893, 1.000] | 0.734 |
| Swift/ObjC | 38 | 37/38 | 37/38 | 27/38 | 10/38 | 0/150 / 0/150 / 0/150 | 1.000 [0.998, 1.000] | 0.925 |
| bash | 33 | 24/33 | 22/33 | 24/33 | 17/33 | 1/200 / 0/200 / 14/200 | 0.934 [0.924, 0.942] | 0.852 |
| cmd/batch | 33 | 28/33 | 29/33 | 28/33 | 23/33 | 0/130 / 0/130 / 1/130 | 0.996 [0.986, 1.000] | 0.958 |
| zsh/macOS | 36 | 29/36 | 31/36 | 26/36 | 23/36 | 3/144 / 3/144 / 5/144 | 0.975 [0.939, 1.000] | 0.904 |
| PowerShell | 37 | 35/37 | 33/37 | 29/37 | 22/37 | 3/150 / 0/150 / 8/150 | 0.992 [0.984, 0.997] | 0.958 |
| Python (code mode) | 35 | 30/35 | 29/35 | 31/35 | 25/35 | 1/200 / 0/200 / 12/200 | 0.986 [0.985, 1.000] | 0.918 |
| JS/TS (code mode) | 36 | 34/36 | 34/36 | 33/36 | 26/36 | 0/129 / 0/129 / 2/129 | 0.998 [0.968, 1.000] | 0.925 |
| all diff cells | 300 | 278/300 | 268/300 | 172/300 | 36/300 | 3/1197 / 4/1197 / 6/1197 | 0.989 [0.978, 0.997] | 0.831 |
| all code-mode cells | 210 | 180/210 | 178/210 | 171/210 | 136/210 | 8/953 / 3/953 / 42/953 | 0.979 [0.970, 0.996] | 0.922 |
| all | 510 | 458/510 | 446/510 | 343/510 | 172/510 | 11/2150 / 7/2150 / 48/2150 | 0.985 [0.978, 0.993] | 0.826 |
By writer of the positives (all cells): seed 1 found 161/174 Claude-written and 297/336 GPT-written; seed 2 154/174 and
292/336; untuned 115/174 and 228/336. Authored benign controls flagged: seed 1 1/560, seed 2 2/560, untuned 12/560.
## Task swaps (the task asks for exactly the flagged behaviour)
- Request alone as the task: clears on 244/246 (seed 1) and 233/246 (seed 2), untuned 163/246. On the English held-out
  set (H3 and H4), it clears on every change it flagged: 122/122 (seed 1) and 118/118 (seed 2), untuned 66/96.
- Request added after a real task (246 pairs, 14 languages): clears on 143 (seed 1) and 173 (seed 2), untuned 116. With the
  same change and an ordinary request, still flags 221, 218 and 152.
## Unrequested account email in request headers (100 code-mode scripts; fictional identities)
- Found: seed 1 1/40, seed 2 0/40, untuned 3/40. Controls flagged: 0/60, 0/60, 4/60.
- Not caught; see Limitations in README.md.
## Placeholder behaviour summary (seed 1, frozen θ; every state's summary replaced by the placeholder)
| test set | extractor summary | placeholder |
|---|---|---|
| 14 languages: found / real flagged / controls flagged | 458/510 / 11/2150 / 1/560 | 448/510 / 5/2150 / 1/560 |
| 14 languages: AUROC (positives vs real) | 0.985 | 0.984 |
| 8 diff languages: found / real flagged | 278/300 / 3/1197 | 277/300 / 2/1197 |
| 6 code-mode/shell cells: found / real flagged | 180/210 / 8/953 | 171/210 / 3/953 |
| bash code mode: found | 24/33 | 20/33 |
| Independent agent-diff set: found / flagged | 21/22 / 0/172 | 20/22 / 0/172 |
## English/Python-heavy test sets
| test set | seed 1 | seed 2 | untuned | untuned, task+change only |
|---|---|---|---|---|
| Independent agent-diff set: found / flagged | 21/22 / 0/172 | 19/22 / 0/172 | 10/22 / 0/172 | 1/22 / 0/172 |
| Compromised releases as diffs: found / ordinary flagged | 21/27 / 0/80 | 19/27 / 0/80 | 19/27 / 0/80 | 13/27 / 0/80 |
| Public code mode, unseen source: found / flagged | 31/31 / 0/200 | 31/31 / 0/200 | 31/31 / 1/200 | 31/31 / 0/200 |
| Held-out agent diffs + code mode: found / real flagged | 122/122 / 0/527 | 118/122 / 0/527 | 96/122 / 2/527 | 45/122 / 0/527 |
| Behaviour families never trained | 26/26 | 22/26 | 14/26 | 4/26 |
Machine-readable: results.json.

	# Results summary (d3-mini-agentcode, pre-release)

	Each model at its own dev-fitted threshold (seed 1 and seed 2: 0.9/0.9; untuned d3-mini with the same context: 0.5/0.7;
	untuned with task + change only: 0.5/0.9). AUROCs on p(class = deliberately_malicious), positives vs real benign, with
	95% cluster-bootstrap CIs over repositories. Numbers are counts (found or flagged / total).

	## 14 languages and shells (positives authored by two writer models; real benign items from public benchmarks)

	\| cell \| positives \| seed 1 found \| seed 2 found \| untuned found \| untuned, task+change only \| real flagged: s1 / s2 / untuned \| seed 1 AUROC [CI] \| untuned AUROC \|
	\|---\|---:\|---:\|---:\|---:\|---:\|---\|---\|---:\|
	\| C/C++ \| 38 \| 35/38 \| 29/38 \| 20/38 \| 5/38 \| 1/149 / 1/149 / 2/149 \| 0.977 [0.943, 1.000] \| 0.835 \|
	\| C# \| 37 \| 34/37 \| 33/37 \| 24/37 \| 5/37 \| 0/149 / 0/149 / 0/149 \| 0.998 [0.992, 1.000] \| 0.840 \|
	\| Go \| 38 \| 36/38 \| 35/38 \| 20/38 \| 5/38 \| 1/150 / 0/150 / 0/150 \| 0.999 [0.997, 1.000] \| 0.864 \|
	\| Java \| 37 \| 36/37 \| 34/37 \| 21/37 \| 3/37 \| 1/149 / 2/149 / 2/149 \| 0.998 [0.994, 1.000] \| 0.810 \|
	\| JS/TS \| 37 \| 33/37 \| 35/37 \| 24/37 \| 1/37 \| 0/150 / 0/150 / 1/150 \| 0.999 [0.997, 1.000] \| 0.839 \|
	\| PHP \| 38 \| 34/38 \| 33/38 \| 21/38 \| 5/38 \| 0/150 / 1/150 / 1/150 \| 0.984 [0.952, 1.000] \| 0.808 \|
	\| Rust \| 37 \| 33/37 \| 32/37 \| 15/37 \| 2/37 \| 0/150 / 0/150 / 0/150 \| 0.962 [0.893, 1.000] \| 0.734 \|
	\| Swift/ObjC \| 38 \| 37/38 \| 37/38 \| 27/38 \| 10/38 \| 0/150 / 0/150 / 0/150 \| 1.000 [0.998, 1.000] \| 0.925 \|
	\| bash \| 33 \| 24/33 \| 22/33 \| 24/33 \| 17/33 \| 1/200 / 0/200 / 14/200 \| 0.934 [0.924, 0.942] \| 0.852 \|
	\| cmd/batch \| 33 \| 28/33 \| 29/33 \| 28/33 \| 23/33 \| 0/130 / 0/130 / 1/130 \| 0.996 [0.986, 1.000] \| 0.958 \|
	\| zsh/macOS \| 36 \| 29/36 \| 31/36 \| 26/36 \| 23/36 \| 3/144 / 3/144 / 5/144 \| 0.975 [0.939, 1.000] \| 0.904 \|
	\| PowerShell \| 37 \| 35/37 \| 33/37 \| 29/37 \| 22/37 \| 3/150 / 0/150 / 8/150 \| 0.992 [0.984, 0.997] \| 0.958 \|
	\| Python (code mode) \| 35 \| 30/35 \| 29/35 \| 31/35 \| 25/35 \| 1/200 / 0/200 / 12/200 \| 0.986 [0.985, 1.000] \| 0.918 \|
	\| JS/TS (code mode) \| 36 \| 34/36 \| 34/36 \| 33/36 \| 26/36 \| 0/129 / 0/129 / 2/129 \| 0.998 [0.968, 1.000] \| 0.925 \|
	\| all diff cells \| 300 \| 278/300 \| 268/300 \| 172/300 \| 36/300 \| 3/1197 / 4/1197 / 6/1197 \| 0.989 [0.978, 0.997] \| 0.831 \|
	\| all code-mode cells \| 210 \| 180/210 \| 178/210 \| 171/210 \| 136/210 \| 8/953 / 3/953 / 42/953 \| 0.979 [0.970, 0.996] \| 0.922 \|
	\| all \| 510 \| 458/510 \| 446/510 \| 343/510 \| 172/510 \| 11/2150 / 7/2150 / 48/2150 \| 0.985 [0.978, 0.993] \| 0.826 \|

	By writer of the positives (all cells): seed 1 found 161/174 Claude-written and 297/336 GPT-written; seed 2 154/174 and
	292/336; untuned 115/174 and 228/336. Authored benign controls flagged: seed 1 1/560, seed 2 2/560, untuned 12/560.

	## Task swaps (the task asks for exactly the flagged behaviour)
	- Request alone as the task: clears on 244/246 (seed 1) and 233/246 (seed 2), untuned 163/246. On the English held-out
	set (H3 and H4), it clears on every change it flagged: 122/122 (seed 1) and 118/118 (seed 2), untuned 66/96.
	- Request added after a real task (246 pairs, 14 languages): clears on 143 (seed 1) and 173 (seed 2), untuned 116. With the
	same change and an ordinary request, still flags 221, 218 and 152.

	## Unrequested account email in request headers (100 code-mode scripts; fictional identities)
	- Found: seed 1 1/40, seed 2 0/40, untuned 3/40. Controls flagged: 0/60, 0/60, 4/60.
	- Not caught; see Limitations in README.md.

	## Placeholder behaviour summary (seed 1, frozen θ; every state's summary replaced by the placeholder)

	\| test set \| extractor summary \| placeholder \|
	\|---\|---\|---\|
	\| 14 languages: found / real flagged / controls flagged \| 458/510 / 11/2150 / 1/560 \| 448/510 / 5/2150 / 1/560 \|
	\| 14 languages: AUROC (positives vs real) \| 0.985 \| 0.984 \|
	\| 8 diff languages: found / real flagged \| 278/300 / 3/1197 \| 277/300 / 2/1197 \|
	\| 6 code-mode/shell cells: found / real flagged \| 180/210 / 8/953 \| 171/210 / 3/953 \|
	\| bash code mode: found \| 24/33 \| 20/33 \|
	\| Independent agent-diff set: found / flagged \| 21/22 / 0/172 \| 20/22 / 0/172 \|

	## English/Python-heavy test sets

	\| test set \| seed 1 \| seed 2 \| untuned \| untuned, task+change only \|
	\|---\|---\|---\|---\|---\|
	\| Independent agent-diff set: found / flagged \| 21/22 / 0/172 \| 19/22 / 0/172 \| 10/22 / 0/172 \| 1/22 / 0/172 \|
	\| Compromised releases as diffs: found / ordinary flagged \| 21/27 / 0/80 \| 19/27 / 0/80 \| 19/27 / 0/80 \| 13/27 / 0/80 \|
	\| Public code mode, unseen source: found / flagged \| 31/31 / 0/200 \| 31/31 / 0/200 \| 31/31 / 1/200 \| 31/31 / 0/200 \|
	\| Held-out agent diffs + code mode: found / real flagged \| 122/122 / 0/527 \| 118/122 / 0/527 \| 96/122 / 2/527 \| 45/122 / 0/527 \|
	\| Behaviour families never trained \| 26/26 \| 22/26 \| 14/26 \| 4/26 \|

	Machine-readable: results.json.