RESULTS.md

Results summary (d3-mini-agentcode, pre-release)

Each model at its own dev-fitted threshold (seed 1 and seed 2: 0.9/0.9; untuned d3-mini with the same context: 0.5/0.7; untuned with task + change only: 0.5/0.9). AUROCs on p(class = deliberately_malicious), positives vs real benign, with 95% cluster-bootstrap CIs over repositories. Numbers are counts (found or flagged / total).

14 languages and shells (positives authored by two writer models; real benign items from public benchmarks)

cell	positives	seed 1 found	seed 2 found	untuned found	untuned, task+change only	real flagged: s1 / s2 / untuned	seed 1 AUROC [CI]	untuned AUROC
C/C++	38	35/38	29/38	20/38	5/38	1/149 / 1/149 / 2/149	0.977 [0.943, 1.000]	0.835
C#	37	34/37	33/37	24/37	5/37	0/149 / 0/149 / 0/149	0.998 [0.992, 1.000]	0.840
Go	38	36/38	35/38	20/38	5/38	1/150 / 0/150 / 0/150	0.999 [0.997, 1.000]	0.864
Java	37	36/37	34/37	21/37	3/37	1/149 / 2/149 / 2/149	0.998 [0.994, 1.000]	0.810
JS/TS	37	33/37	35/37	24/37	1/37	0/150 / 0/150 / 1/150	0.999 [0.997, 1.000]	0.839
PHP	38	34/38	33/38	21/38	5/38	0/150 / 1/150 / 1/150	0.984 [0.952, 1.000]	0.808
Rust	37	33/37	32/37	15/37	2/37	0/150 / 0/150 / 0/150	0.962 [0.893, 1.000]	0.734
Swift/ObjC	38	37/38	37/38	27/38	10/38	0/150 / 0/150 / 0/150	1.000 [0.998, 1.000]	0.925
bash	33	24/33	22/33	24/33	17/33	1/200 / 0/200 / 14/200	0.934 [0.924, 0.942]	0.852
cmd/batch	33	28/33	29/33	28/33	23/33	0/130 / 0/130 / 1/130	0.996 [0.986, 1.000]	0.958
zsh/macOS	36	29/36	31/36	26/36	23/36	3/144 / 3/144 / 5/144	0.975 [0.939, 1.000]	0.904
PowerShell	37	35/37	33/37	29/37	22/37	3/150 / 0/150 / 8/150	0.992 [0.984, 0.997]	0.958
Python (code mode)	35	30/35	29/35	31/35	25/35	1/200 / 0/200 / 12/200	0.986 [0.985, 1.000]	0.918
JS/TS (code mode)	36	34/36	34/36	33/36	26/36	0/129 / 0/129 / 2/129	0.998 [0.968, 1.000]	0.925
all diff cells	300	278/300	268/300	172/300	36/300	3/1197 / 4/1197 / 6/1197	0.989 [0.978, 0.997]	0.831
all code-mode cells	210	180/210	178/210	171/210	136/210	8/953 / 3/953 / 42/953	0.979 [0.970, 0.996]	0.922
all	510	458/510	446/510	343/510	172/510	11/2150 / 7/2150 / 48/2150	0.985 [0.978, 0.993]	0.826

By writer of the positives (all cells): seed 1 found 161/174 Claude-written and 297/336 GPT-written; seed 2 154/174 and 292/336; untuned 115/174 and 228/336. Authored benign controls flagged: seed 1 1/560, seed 2 2/560, untuned 12/560.

Task swaps (the task asks for exactly the flagged behaviour)

• Request alone as the task: clears on 244/246 (seed 1) and 233/246 (seed 2), untuned 163/246. On the English held-out set (H3 and H4), it clears on every change it flagged: 122/122 (seed 1) and 118/118 (seed 2), untuned 66/96.
• Request added after a real task (246 pairs, 14 languages): clears on 143 (seed 1) and 173 (seed 2), untuned 116. With the same change and an ordinary request, still flags 221, 218 and 152.

Unrequested account email in request headers (100 code-mode scripts; fictional identities)

• Found: seed 1 1/40, seed 2 0/40, untuned 3/40. Controls flagged: 0/60, 0/60, 4/60.
• Not caught; see Limitations in README.md.

Placeholder behaviour summary (seed 1, frozen θ; every state's summary replaced by the placeholder)

test set	extractor summary	placeholder
14 languages: found / real flagged / controls flagged	458/510 / 11/2150 / 1/560	448/510 / 5/2150 / 1/560
14 languages: AUROC (positives vs real)	0.985	0.984
8 diff languages: found / real flagged	278/300 / 3/1197	277/300 / 2/1197
6 code-mode/shell cells: found / real flagged	180/210 / 8/953	171/210 / 3/953
bash code mode: found	24/33	20/33
Independent agent-diff set: found / flagged	21/22 / 0/172	20/22 / 0/172

English/Python-heavy test sets

test set	seed 1	seed 2	untuned	untuned, task+change only
Independent agent-diff set: found / flagged	21/22 / 0/172	19/22 / 0/172	10/22 / 0/172	1/22 / 0/172
Compromised releases as diffs: found / ordinary flagged	21/27 / 0/80	19/27 / 0/80	19/27 / 0/80	13/27 / 0/80
Public code mode, unseen source: found / flagged	31/31 / 0/200	31/31 / 0/200	31/31 / 1/200	31/31 / 0/200
Held-out agent diffs + code mode: found / real flagged	122/122 / 0/527	118/122 / 0/527	96/122 / 2/527	45/122 / 0/527
Behaviour families never trained	26/26	22/26	14/26	4/26

Machine-readable: results.json.