aisa-group/training_against_probes

Training Against Probes: LoRA adapters and detectors

LoRA adapters and the internal detectors they were trained against, from the paper Alignment via Training Against Probes Without Losing Monitorability.

We study probe-guided fine-tuning: instead of rewarding outputs that look aligned, we use probes that detect undesired properties in the model's activations as a direct training signal. We train against a single linear probe and against a non-linear polytope of K=16 probes, either frozen after an initial fit or continuously updated as the model's representations change during training. We target harmlessness with Mistral 7B Instruct v0.1 and the abliterated Llama 3 8B Instruct, and honesty with Qwen3-14B.

For every run we release the LoRA adapters at all steps on the run's Pareto front (the same fronts as in the paper's figures) and the detector the model was trained against: the single frozen detector, or the full sequence of refit detectors for the continuously updated regime.

Runs

Harmfulness

Model	Detector	Regime	Run	Pareto-front steps	Selected step
Mistral 7B Instruct v0.1	Linear probe	Frozen	harmfulness/mistral-7b-instruct-v0.1/probe/frozen	7, 8, 9, 29, 49	9 (StrongREJECT 0.33, utility 1.03)
Mistral 7B Instruct v0.1	Linear probe	Continuously updated	harmfulness/mistral-7b-instruct-v0.1/probe/continuously_updated	32, 34, 35, 47, 48	47 (StrongREJECT 0.03, utility 0.96)
Mistral 7B Instruct v0.1	Non-linear polytope (K=16)	Frozen	harmfulness/mistral-7b-instruct-v0.1/polytope/frozen	8, 10, 34, 41	8 (StrongREJECT 0.40, utility 1.01)
Mistral 7B Instruct v0.1	Non-linear polytope (K=16)	Continuously updated	harmfulness/mistral-7b-instruct-v0.1/polytope/continuously_updated	28, 29, 31, 38, 44, 45, 46, 47, 48, 49	44 (StrongREJECT 0.05, utility 0.96)
Llama 3 8B Instruct (abliterated)	Linear probe	Frozen	harmfulness/llama-3-8b-instruct-abliterated/probe/frozen	1, 6, 8	8 (StrongREJECT 0.46, utility 0.99)
Llama 3 8B Instruct (abliterated)	Linear probe	Continuously updated	harmfulness/llama-3-8b-instruct-abliterated/probe/continuously_updated	2, 5, 6, 11, 19, 20, 21, 23, 24, 25, 27, 28, 29	19 (StrongREJECT 0.29, utility 0.97)
Llama 3 8B Instruct (abliterated)	Non-linear polytope (K=16)	Frozen	harmfulness/llama-3-8b-instruct-abliterated/polytope/frozen	1, 11, 12	12 (StrongREJECT 0.50, utility 0.98)
Llama 3 8B Instruct (abliterated)	Non-linear polytope (K=16)	Continuously updated	harmfulness/llama-3-8b-instruct-abliterated/polytope/continuously_updated	31, 32	31 (StrongREJECT 0.08, utility 1.00)

Base models (step 0): Mistral 7B Instruct v0.1 StrongREJECT 0.72; Llama 3 8B Instruct (abliterated) StrongREJECT 0.71.

Honesty

Model	Detector	Regime	Run	Pareto-front steps	Selected step
Qwen3-14B	Linear probe	Continuously updated	honesty/qwen3-14b/probe/continuously_updated	13, 17, 30	17 (MASK honesty 63.0, utility 1.00)
Qwen3-14B	Non-linear polytope (K=16)	Continuously updated	honesty/qwen3-14b/polytope/continuously_updated	1, 4, 18	18 (MASK honesty 70.0, utility 1.00)

Base models (step 0): Qwen3-14B MASK honesty 47.0.

Repository layout

<concept>/<model>/<detector>/<regime>/
    config.yaml            full training configuration of the run
    metrics.json           Pareto steps, selected step and the per-step trajectory
    adapters/step_NN/      LoRA adapter after training step NN (Pareto-front steps only)
    detector/              frozen regime: the detector used for every step
    detectors/step_NN/     continuously updated regime: the detector used during step NN
manifest.json              one entry per run

A detector directory holds one layer_L/ folder per supervised layer, each with config.json (architecture) and model.pt (weights). detectors/step_NN is the detector that scored the generations of training step NN. It was refit on the checkpoint after step NN-1 (step_01 is the initial fit on the base model). The detector refit on the checkpoint in adapters/step_NN is therefore detectors/step_{NN+1}.

Usage

from huggingface_hub import snapshot_download
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

run = "harmfulness/mistral-7b-instruct-v0.1/polytope/continuously_updated"
base = AutoModelForCausalLM.from_pretrained("mistralai/Mistral-7B-Instruct-v0.1",
                                            torch_dtype="auto", device_map="auto")
tok = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-Instruct-v0.1")
model = PeftModel.from_pretrained(base, "aisa-group/training_against_probes", subfolder=f"{run}/adapters/step_44")

The detectors load with the code repository (src/probes on the path):

local = snapshot_download("aisa-group/training_against_probes", allow_patterns=[f"{run}/detectors/step_44/*"])
from train import load_model
detector = load_model(f"{local}/{run}/detectors/step_44")   # {layer: probe module}

The abliterated Llama runs use lenalibon/Meta-Llama-3-8B-Instruct-heretic-mlabonne (Llama 3 8B Instruct with its refusal direction removed) as the base model, with the tokenizer of meta-llama/Meta-Llama-3-8B-Instruct. Qwen3-14B was trained and evaluated with thinking enabled.

Citation

@article{libon2026alignment,
  title={Alignment via Training Against Probes Without Losing Monitorability},
  author={Libon, Lena and Panfilov, Alexander and Rank, Ben and Chen, Xin and Geiping, Jonas and Andriushchenko, Maksym},
  journal={arXiv preprint arXiv:2609.38645},
  year={2026}
}