lexsi-tiny-aya-earth-rmu

This repository contains a full bf16 RMU-unlearned checkpoint of CohereLabs/tiny-aya-earth after fact injection (LoRA) and Representation Misdirection for Unlearning (RMU). It is a complete saved model (not a LoRA or adapter). The gated base weights are not included.

  • Base model: CohereLabs/tiny-aya-earth (gated, CC-BY-NC-4.0)
  • Architecture: Cohere2ForCausalLM, 3.35B, 36 layers, hidden size 2048, tied embeddings
  • Dtype: bfloat16
  • Run id: 20261113T074830Z-inject+verify+unlearn+eval+probes+interp+relearn--d55bdb88
  • Seed: one seed (this run)
  • RMU: steers layer 18, updates layers 16โ€“18, alpha=100, steering_coeff=20, 80 steps, content forget mask
  • Variant: Earth

What this does and does not claim

These weights are released so others can inspect suppression, utility, multilingual transfer, probes, and relearning. They do not establish that the forgotten facts were erased. A probe or a short relearning pass can still recover the facts. Logit lens, activation patching, head ablation, and CircuitKIT circuit analysis are not analyzed in this version and are left to future work.

How to load

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "mackenzietechdocs/lexsi-tiny-aya-earth-rmu"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype="bfloat16", device_map="auto")

You need access to the gated base tokenizer/config family if you also want to compare against CohereLabs/tiny-aya-earth itself.

License

CC-BY-NC-4.0, inherited from Tiny Aya. Non-commercial use only. Please also respect the Tiny Aya license and acceptable-use terms.

Citation / companion artifacts

Paper PDF, scrubbed traces, and translations are in the dataset mackenzietechdocs/lexsi-unlearning-eval. The demo Space (mackenzietechdocs/lexsi-unlearning-demo) browses those precomputed outputs and does not run live inference.

Downloads last month
14
Safetensors
Model size
3B params
Tensor type
BF16
ยท
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for mackenzietechdocs/lexsi-tiny-aya-earth-rmu

Finetuned
(29)
this model

Collection including mackenzietechdocs/lexsi-tiny-aya-earth-rmu