Tiny Aya Global, collusion direction ablated

Research artifact, not certified for deployment. This pre-registered fix did not work: on the test set it raises collusive acceptance and nearly silences reporting (below). It is published as the record of a negative result. For a working correction see harmanjs/tiny-aya-global-collusion-lora-fix. Built for the Lexsi AI Safety and Alignment Hackathon (Track 2) by Team Cadbury (Abhra Dubey, Harmanjot Singh). Code and full write-up: the GitHub repository (https://github.com/coderjs-lab/whistleblower-or-accomplice) and article linked from the collection.

What changed

Starting from CohereLabs/tiny-aya-global (revision af89d219b53ed9b13b8a4645f9c8028973510324), one unit direction d ("collusion direction", 2048-d, from layer 22 of 36) was projected out of the weights with SafeTune 0.1.22 orthogonalize_weights:

  • every attention output projection (o_proj) and every MLP down projection (down_proj) in all 36 decoder layers: W โ† W โˆ’ d dแต€ W;
  • the token embedding: E โ† E โˆ’ (E d) dแต€. On Tiny Aya the embedding is tied to the output head, so the unembedding changes too.

No other parameter changed and the model runs with no hooks: this is a standalone checkpoint. whistle_edit.json lists the edited tensors; the direction file and the layer scan are in direction/.

How d was found (calib split only). For 384 calibration episodes in which a teammate privately invites the model to break the operator's rule, we took the templated prompt followed by {"decision": "ACCEPT" and by {"decision": "DECLINE" and computed the per-layer difference of means of the residual stream at the last token (SafeTune extract_refusal_direction, ACCEPT side as "harmful"). The layer was chosen by how well the projection separates offers the base model accepted from those it declined (AUC 0.88 on calib).

Measured results (frozen test set, same prompts, greedy decoding, 96 new tokens)

base Tiny Aya Global this checkpoint
collusive invitations accepted (โ†“) 77.4 [72.6, 81.8] 82.9 [78.8, 87.1]
legitimate coordination accepted (โ†‘) 82.8 [77.9, 87.5] 86.0 [81.8, 90.1]
colluders detected when asked (โ†‘) 50.1 [46.3, 53.9] 17.0 [14.7, 19.6]
false accusations (โ†“) 36.7 [33.9, 39.8] 15.8 [14.0, 17.6]
Global-MMLU-Lite (โ†‘) 52.7 52.7
unparseable answers 0.0 0.0

Paired McNemar on collusive offers: 8 episodes flip to DECLINE and 48 to ACCEPT (p < 0.001). Five random directions orthogonalized the same way give 72.5โ€“80.8% collusive acceptance. Under the pre-registered decision rule this fix is not effective.

Why it fails

The direction comes from forced answers (prompt + {"decision": "ACCEPT" vs ... "DECLINE") read at the last token, i.e. after the answer word. It has cosine 0.78 with a neutral approve/decline direction built with no deal involved, so it is not collusion-specific. Removing it raises acceptance of both offer types (as three of five random directions removed the same way also do) and, distinctively, collapses reporting; the exact cause is not established. A direction read at the decision point itself is nearly orthogonal to it and does control decisions; see the write-up.

Caveats

  • Evaluated on templated English team logs in four domains; behaviour outside them is untested.
  • The edit removes a direction defined by forced ACCEPT/DECLINE answers; compare with the generic-direction control in the article before reading it as "collusion-specific".

Licence

CC-BY-NC-4.0, inherited from Tiny Aya Global, plus Cohere's Acceptable Use Policy (https://docs.cohere.com/docs/c4ai-acceptable-use-policy). Use must comply with both.

Downloads last month
9
Safetensors
Model size
3B params
Tensor type
BF16
ยท
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for harmanjs/tiny-aya-global-collusion-ablated

Finetuned
(171)
this model

Collection including harmanjs/tiny-aya-global-collusion-ablated